Skip to content

fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#45712

Closed
cass-clearly wants to merge 3 commits into
openclaw:mainfrom
cass-clearly:fix/health-monitor-reconnect-grace
Closed

fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#45712
cass-clearly wants to merge 3 commits into
openclaw:mainfrom
cass-clearly:fix/health-monitor-reconnect-grace

Conversation

@cass-clearly

Copy link
Copy Markdown

Problem

When a Discord WebSocket drops (code 1006), the gateway lifecycle has built-in reconnection logic (resume/identify) that typically recovers within seconds. However, the health monitor runs every 5 minutes and immediately triggers a full provider teardown + restart when it sees connected: false, racing with and destroying the in-progress reconnection.

Each full restart creates a new Discord.js Client, allocating significant heap memory. The old client's resources may not be fully GC'd before the next restart cycle. Over time this causes steady memory growth leading to OOM kills.

Observed impact (real deployment)

  • 505 health-monitor restarts in 7 days (~72/day, every ~20 minutes)
  • 56 "disconnected" + 18 "stale-socket" restarts in last 24 hours
  • Gateway OOM-killed twice in one evening (at 18:00 and 21:18)
  • Memory peak: 1.2GB before first kill
  • Network was stable: 0% packet loss, ~7ms latency to Discord gateway
  • Single guild, 17 channels — not unusual load

Root cause

In channel-health-policy.ts, when snapshot.connected === false, the health evaluation immediately returns { healthy: false, reason: "disconnected" } with no grace period for reconnection. The existing channelConnectGraceMs (120s) only applies to fresh starts (lastStartAt), not to mid-lifecycle reconnections.

The gateway lifecycle already has a RECONNECT_STALL_TIMEOUT_MS (5 min) watchdog that force-stops the provider if reconnection truly stalls. The health monitor should not race with this internal logic.

Solution

Add a configurable reconnect grace period (default: 5 minutes) that lets running channels recover from transient disconnects before the health monitor intervenes:

  1. Added reconnectGraceMs to ChannelHealthPolicy and ChannelHealthTimingPolicy
  2. Added lastDisconnectAt to ChannelHealthSnapshot
  3. In the health monitor, extract lastDisconnect.at from the runtime snapshot
  4. When connected === false but the channel is still running, check if the disconnect happened within the grace period before marking unhealthy
  5. Fall back to lastEventAt as a proxy when lastDisconnect timestamp is unavailable
  6. Non-running channels bypass grace entirely (handled by the existing not-running path)

Configuration

The grace period can be tuned via the timing config:

startChannelHealthMonitor({
  channelManager,
  timing: {
    reconnectGraceMs: 10 * 60_000, // 10 minutes
  },
});

Testing

  • 7 new test cases covering reconnect grace behavior
  • All 34 existing health monitor tests pass
  • Updated existing "stuck channel" test to account for grace period

Files changed

  • src/gateway/channel-health-policy.ts — reconnect grace logic + constant
  • src/gateway/channel-health-monitor.ts — extract lastDisconnectAt, pass to policy
  • src/gateway/channel-health-monitor.test.ts — new test suite + updated existing test

Fixes #41354
Related: #42178, #39096, #30212, #39288

…Discord restarts

When a Discord WebSocket drops (code 1006), the gateway lifecycle's
built-in reconnection logic (resume/identify) typically recovers within
seconds. However, the health monitor checks every 5 minutes and
immediately restarts the entire provider when it sees connected=false,
racing with and destroying the in-progress reconnection.

Each full restart tears down the Discord.js client and creates a new
one, allocating significant memory that may not be fully GC'd. Over
time this causes memory growth leading to OOM kills — observed as
~72 restarts/day (505 in 7 days) on a stable network with 0% packet
loss.

This change adds a configurable reconnect grace period (default: 5min)
that lets running channels recover from transient disconnects before
the health monitor intervenes with a full teardown/restart cycle:

- Add reconnectGraceMs to ChannelHealthPolicy and timing config
- Add lastDisconnectAt to ChannelHealthSnapshot
- Extract lastDisconnect.at from runtime snapshot in health monitor
- When connected=false but channel is still running, check if
  disconnect is recent (within grace period) before restarting
- Fall back to lastEventAt when lastDisconnect timestamp is missing
- Non-running channels bypass grace (handled by not-running path)

Fixes openclaw#41354
Related: openclaw#42178, openclaw#39096, openclaw#30212
@greptile-apps

greptile-apps Bot commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a configurable reconnect grace period (default 5 minutes) to the channel health monitor, preventing it from tearing down and recreating Discord providers during the brief window when the gateway's own reconnect logic (resume/identify) is still in progress. The root cause — health checks racing with built-in reconnect cycles — is well-diagnosed and the fix is correctly scoped.

Key changes:

  • DEFAULT_RECONNECT_GRACE_MS (5 min) added alongside the existing startup and stale-event constants.
  • evaluateChannelHealth now returns healthy: true for a running-but-disconnected channel if the disconnect is newer than reconnectGraceMs, falling back to lastEventAt as a proxy when lastDisconnect is unavailable.
  • channel-health-monitor.ts extracts lastDisconnect.at from the runtime snapshot and maps it to the typed lastDisconnectAt field on ChannelHealthSnapshot.
  • 7 new tests thoroughly cover the grace-period paths; the updated "stuck channel" test correctly uses explicit timestamps to assert past-grace behavior.

Issues found:

  • lastDisconnect is not declared in ChannelAccountSnapshot so it is accessed via an unsafe as { lastDisconnect?: unknown } cast with manual runtime shape checks, rather than through the type system. Adding the field to ChannelAccountSnapshot would eliminate this bypass and simplify the extraction to a single optional-chain.
  • The startup log (started (interval: …s, startup-grace: …s, …)) does not include reconnectGraceMs, making the configured value invisible in operational logs — worth adding alongside the other timing parameters.

Confidence Score: 4/5

  • Safe to merge; the logic is sound and well-tested, with two minor polish issues.
  • The fix correctly addresses the documented race condition, is gated on the running flag so it cannot suppress restarts for stopped channels, and is well-covered by tests. The two issues (unsafe lastDisconnect type cast and missing reconnectGraceMs in the startup log) are style/observability concerns that do not affect correctness.
  • src/gateway/channel-health-monitor.ts — the lastDisconnect type cast on lines 137–144 should be replaced by properly typing the field in ChannelAccountSnapshot.

Comments Outside Diff (1)

  1. src/gateway/channel-health-monitor.ts, line 211-212 (link)

    Startup log omits reconnectGraceMs

    The startup log.info records interval, startup-grace, and channel-connect-grace, but does not include the new reconnectGraceMs value. Given that this parameter is the core of the fix (and misconfiguring it would silently change how quickly the monitor intervenes after a disconnect), it should appear in the startup log alongside the other timing values.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: src/gateway/channel-health-monitor.ts
    Line: 211-212
    
    Comment:
    **Startup log omits `reconnectGraceMs`**
    
    The startup `log.info` records `interval`, `startup-grace`, and `channel-connect-grace`, but does not include the new `reconnectGraceMs` value. Given that this parameter is the core of the fix (and misconfiguring it would silently change how quickly the monitor intervenes after a disconnect), it should appear in the startup log alongside the other timing values.
    
    
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/gateway/channel-health-monitor.ts
Line: 137-144

Comment:
**Unsafe type cast for `lastDisconnect`**

`status` is typed as `ChannelAccountSnapshot`, which does not appear to declare a `lastDisconnect` field, so it is accessed by casting `status as { lastDisconnect?: unknown }`. This bypasses TypeScript's type system and will silently stop working correctly if the runtime shape of the field changes.

The correct fix is to add `lastDisconnect?: { at?: number } | null` to `ChannelAccountSnapshot` (in `src/channels/plugins/types.ts` or wherever it is declared). Once typed, the entire extraction block can be simplified to a one-liner:

```ts
const lastDisconnectAt = typeof status.lastDisconnect?.at === "number"
  ? status.lastDisconnect.at
  : null;
```

As written, the cast and the manual shape-checks work at runtime, but they are invisible to the type checker and will not catch future structural regressions.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/gateway/channel-health-monitor.ts
Line: 211-212

Comment:
**Startup log omits `reconnectGraceMs`**

The startup `log.info` records `interval`, `startup-grace`, and `channel-connect-grace`, but does not include the new `reconnectGraceMs` value. Given that this parameter is the core of the fix (and misconfiguring it would silently change how quickly the monitor intervenes after a disconnect), it should appear in the startup log alongside the other timing values.

```suggestion
    log.info?.(
      `started (interval: ${Math.round(checkIntervalMs / 1000)}s, startup-grace: ${Math.round(timing.monitorStartupGraceMs / 1000)}s, channel-connect-grace: ${Math.round(timing.channelConnectGraceMs / 1000)}s, reconnect-grace: ${Math.round(timing.reconnectGraceMs / 1000)}s)`,
    );
```

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: 14dee8d

Comment thread src/gateway/channel-health-monitor.ts Outdated
Comment on lines +137 to +144
const lastDisconnect = (status as { lastDisconnect?: unknown }).lastDisconnect;
const lastDisconnectAt =
lastDisconnect != null &&
typeof lastDisconnect === "object" &&
"at" in lastDisconnect &&
typeof (lastDisconnect as { at?: unknown }).at === "number"
? (lastDisconnect as { at: number }).at
: null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsafe type cast for lastDisconnect

status is typed as ChannelAccountSnapshot, which does not appear to declare a lastDisconnect field, so it is accessed by casting status as { lastDisconnect?: unknown }. This bypasses TypeScript's type system and will silently stop working correctly if the runtime shape of the field changes.

The correct fix is to add lastDisconnect?: { at?: number } | null to ChannelAccountSnapshot (in src/channels/plugins/types.ts or wherever it is declared). Once typed, the entire extraction block can be simplified to a one-liner:

const lastDisconnectAt = typeof status.lastDisconnect?.at === "number"
  ? status.lastDisconnect.at
  : null;

As written, the cast and the manual shape-checks work at runtime, but they are invisible to the type checker and will not catch future structural regressions.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/channel-health-monitor.ts
Line: 137-144

Comment:
**Unsafe type cast for `lastDisconnect`**

`status` is typed as `ChannelAccountSnapshot`, which does not appear to declare a `lastDisconnect` field, so it is accessed by casting `status as { lastDisconnect?: unknown }`. This bypasses TypeScript's type system and will silently stop working correctly if the runtime shape of the field changes.

The correct fix is to add `lastDisconnect?: { at?: number } | null` to `ChannelAccountSnapshot` (in `src/channels/plugins/types.ts` or wherever it is declared). Once typed, the entire extraction block can be simplified to a one-liner:

```ts
const lastDisconnectAt = typeof status.lastDisconnect?.at === "number"
  ? status.lastDisconnect.at
  : null;
```

As written, the cast and the manual shape-checks work at runtime, but they are invisible to the type checker and will not catch future structural regressions.

How can I resolve this? If you propose a fix, please make it concise.

Cass and others added 2 commits March 13, 2026 22:07
…-grace to startup log

- Replace unsafe `as { lastDisconnect?: unknown }` cast with direct
  property access via the typed ChannelAccountSnapshot.lastDisconnect field
- Add reconnect-grace to the health monitor startup log so the configured
  value is visible in operational logs alongside interval, startup-grace,
  and channel-connect-grace
- Fix oxfmt formatting
@clawsweeper

clawsweeper Bot commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve.

Keep this PR open. Current main still lacks the PR's central reconnect-grace behavior: after startup connect grace, a running channel with connected: false is still unhealthy immediately and the health monitor can restart it. Main now has related hardening such as cooldown/hour caps and typed Discord lastDisconnect data, but those do not supersede the delayed-intervention behavior proposed here.

Best possible solution:

Keep this PR open as a useful implementation candidate. Rebase it onto current main, add a generic reconnectGraceMs/lastDisconnectAt path in the shared health policy, consume typed lastDisconnect.at from runtime snapshots with a clear fallback policy, update both health-monitor and readiness call sites, and add focused tests for recent disconnects, expired disconnects, stopped channels, stale sockets, active runs, and restart caps.

What I checked:

  • Current health policy has no reconnect grace contract: ChannelHealthSnapshot has no lastDisconnectAt input and ChannelHealthPolicy only carries channelId, now, staleEventThresholdMs, and channelConnectGraceMs; there is no reconnectGraceMs field on current main. (src/gateway/channel-health-policy.ts:3, bdfb408ce6ee)
  • Disconnected running channels still fail immediately: After startup connect grace, evaluateChannelHealth returns { healthy: false, reason: "disconnected" } as soon as snapshot.connected === false, with no check against disconnect age or lastDisconnect.at. (src/gateway/channel-health-policy.ts:112, bdfb408ce6ee)
  • Monitor does not plumb reconnect timing: ChannelHealthTimingPolicy includes startup grace, channel connect grace, and stale-event threshold only. The monitor builds the policy without reconnect grace and passes the raw account status to evaluateChannelHealth. (src/gateway/channel-health-monitor.ts:26, bdfb408ce6ee)
  • Existing tests still assert restart for running disconnected channels: The health monitor test for a running but not connected channel with lastStartAt five minutes ago expects stopChannel, resetRestartAttempts, and startChannel, which is the behavior this PR is trying to delay during reconnect grace. (src/gateway/channel-health-monitor.test.ts:277, bdfb408ce6ee)
  • Readiness shares the immediate-disconnect policy: Readiness constructs the same ChannelHealthPolicy shape without reconnect grace, and its test still reports a disconnected managed Discord channel as not ready after startup grace. A rebased implementation needs to update this surface deliberately. (src/gateway/server/readiness.ts:72, bdfb408ce6ee)
  • Discord disconnect timestamp is available to consume: Current main already types ChannelAccountSnapshot.lastDisconnect with an at field, and Discord lifecycle code publishes that field on gateway close/runtime-not-ready events, so the remaining gap is policy plumbing rather than a missing status API. (src/channels/plugins/types.core.ts:200, bdfb408ce6ee)

Remaining risk / open question:

  • Merging the PR as-is against current main may miss the newer readiness call site, which also constructs ChannelHealthPolicy and currently treats post-startup disconnects as failures.
  • The rebased change needs to preserve current busy-run handling, stale-socket transport-activity behavior, cooldown/hourly restart caps, and non-running/gave-up recovery paths.
  • The reconnect grace window must be bounded so it does not hide genuinely dead channels indefinitely.

Codex review notes: model gpt-5.5, reasoning high; reviewed against bdfb408ce6ee.

@openclaw-clownfish

Copy link
Copy Markdown
Contributor

ProjectClownfish could not safely update this branch, so it opened a narrow replacement PR instead.

Replacement PR: #73989
Source PR: #45712
Contributor credit is preserved in the replacement PR body and changelog plan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discord plugin disconnects every ~10-15 minutes (health-monitor restart loop)

1 participant