Skip to content

fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#663

Open
BingqingLyu wants to merge 3 commits into
mainfrom
fork-pr-45712-fix-health-monitor-reconnect-grace
Open

fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#663
BingqingLyu wants to merge 3 commits into
mainfrom
fork-pr-45712-fix-health-monitor-reconnect-grace

Conversation

@BingqingLyu

@BingqingLyu BingqingLyu commented Apr 27, 2026

Copy link
Copy Markdown
Owner

Problem

When a Discord WebSocket drops (code 1006), the gateway lifecycle has built-in reconnection logic (resume/identify) that typically recovers within seconds. However, the health monitor runs every 5 minutes and immediately triggers a full provider teardown + restart when it sees connected: false, racing with and destroying the in-progress reconnection.

Each full restart creates a new Discord.js Client, allocating significant heap memory. The old client's resources may not be fully GC'd before the next restart cycle. Over time this causes steady memory growth leading to OOM kills.

Observed impact (real deployment)

  • 505 health-monitor restarts in 7 days (~72/day, every ~20 minutes)
  • 56 "disconnected" + 18 "stale-socket" restarts in last 24 hours
  • Gateway OOM-killed twice in one evening (at 18:00 and 21:18)
  • Memory peak: 1.2GB before first kill
  • Network was stable: 0% packet loss, ~7ms latency to Discord gateway
  • Single guild, 17 channels — not unusual load

Root cause

In channel-health-policy.ts, when snapshot.connected === false, the health evaluation immediately returns { healthy: false, reason: "disconnected" } with no grace period for reconnection. The existing channelConnectGraceMs (120s) only applies to fresh starts (lastStartAt), not to mid-lifecycle reconnections.

The gateway lifecycle already has a RECONNECT_STALL_TIMEOUT_MS (5 min) watchdog that force-stops the provider if reconnection truly stalls. The health monitor should not race with this internal logic.

Solution

Add a configurable reconnect grace period (default: 5 minutes) that lets running channels recover from transient disconnects before the health monitor intervenes:

  1. Added reconnectGraceMs to ChannelHealthPolicy and ChannelHealthTimingPolicy
  2. Added lastDisconnectAt to ChannelHealthSnapshot
  3. In the health monitor, extract lastDisconnect.at from the runtime snapshot
  4. When connected === false but the channel is still running, check if the disconnect happened within the grace period before marking unhealthy
  5. Fall back to lastEventAt as a proxy when lastDisconnect timestamp is unavailable
  6. Non-running channels bypass grace entirely (handled by the existing not-running path)

Configuration

The grace period can be tuned via the timing config:

startChannelHealthMonitor({
  channelManager,
  timing: {
    reconnectGraceMs: 10 * 60_000, // 10 minutes
  },
});

Testing

  • 7 new test cases covering reconnect grace behavior
  • All 34 existing health monitor tests pass
  • Updated existing "stuck channel" test to account for grace period

Files changed

  • src/gateway/channel-health-policy.ts — reconnect grace logic + constant
  • src/gateway/channel-health-monitor.ts — extract lastDisconnectAt, pass to policy
  • src/gateway/channel-health-monitor.test.ts — new test suite + updated existing test

Fixes openclaw#41354
Related: openclaw#42178, openclaw#39096, openclaw#30212, openclaw#39288

Cass and others added 3 commits March 13, 2026 21:58
…Discord restarts

When a Discord WebSocket drops (code 1006), the gateway lifecycle's
built-in reconnection logic (resume/identify) typically recovers within
seconds. However, the health monitor checks every 5 minutes and
immediately restarts the entire provider when it sees connected=false,
racing with and destroying the in-progress reconnection.

Each full restart tears down the Discord.js client and creates a new
one, allocating significant memory that may not be fully GC'd. Over
time this causes memory growth leading to OOM kills — observed as
~72 restarts/day (505 in 7 days) on a stable network with 0% packet
loss.

This change adds a configurable reconnect grace period (default: 5min)
that lets running channels recover from transient disconnects before
the health monitor intervenes with a full teardown/restart cycle:

- Add reconnectGraceMs to ChannelHealthPolicy and timing config
- Add lastDisconnectAt to ChannelHealthSnapshot
- Extract lastDisconnect.at from runtime snapshot in health monitor
- When connected=false but channel is still running, check if
  disconnect is recent (within grace period) before restarting
- Fall back to lastEventAt when lastDisconnect timestamp is missing
- Non-running channels bypass grace (handled by not-running path)

Fixes openclaw#41354
Related: openclaw#42178, openclaw#39096, openclaw#30212
…-grace to startup log

- Replace unsafe `as { lastDisconnect?: unknown }` cast with direct
  property access via the typed ChannelAccountSnapshot.lastDisconnect field
- Add reconnect-grace to the health monitor startup log so the configured
  value is visible in operational logs alongside interval, startup-grace,
  and channel-connect-grace
- Fix oxfmt formatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discord plugin disconnects every ~10-15 minutes (health-monitor restart loop)

1 participant