fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#45712
fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#45712cass-clearly wants to merge 3 commits into
Conversation
…Discord restarts When a Discord WebSocket drops (code 1006), the gateway lifecycle's built-in reconnection logic (resume/identify) typically recovers within seconds. However, the health monitor checks every 5 minutes and immediately restarts the entire provider when it sees connected=false, racing with and destroying the in-progress reconnection. Each full restart tears down the Discord.js client and creates a new one, allocating significant memory that may not be fully GC'd. Over time this causes memory growth leading to OOM kills — observed as ~72 restarts/day (505 in 7 days) on a stable network with 0% packet loss. This change adds a configurable reconnect grace period (default: 5min) that lets running channels recover from transient disconnects before the health monitor intervenes with a full teardown/restart cycle: - Add reconnectGraceMs to ChannelHealthPolicy and timing config - Add lastDisconnectAt to ChannelHealthSnapshot - Extract lastDisconnect.at from runtime snapshot in health monitor - When connected=false but channel is still running, check if disconnect is recent (within grace period) before restarting - Fall back to lastEventAt when lastDisconnect timestamp is missing - Non-running channels bypass grace (handled by not-running path) Fixes openclaw#41354 Related: openclaw#42178, openclaw#39096, openclaw#30212
Greptile SummaryThis PR adds a configurable reconnect grace period (default 5 minutes) to the channel health monitor, preventing it from tearing down and recreating Discord providers during the brief window when the gateway's own reconnect logic (resume/identify) is still in progress. The root cause — health checks racing with built-in reconnect cycles — is well-diagnosed and the fix is correctly scoped. Key changes:
Issues found:
Confidence Score: 4/5
|
| const lastDisconnect = (status as { lastDisconnect?: unknown }).lastDisconnect; | ||
| const lastDisconnectAt = | ||
| lastDisconnect != null && | ||
| typeof lastDisconnect === "object" && | ||
| "at" in lastDisconnect && | ||
| typeof (lastDisconnect as { at?: unknown }).at === "number" | ||
| ? (lastDisconnect as { at: number }).at | ||
| : null; |
There was a problem hiding this comment.
Unsafe type cast for lastDisconnect
status is typed as ChannelAccountSnapshot, which does not appear to declare a lastDisconnect field, so it is accessed by casting status as { lastDisconnect?: unknown }. This bypasses TypeScript's type system and will silently stop working correctly if the runtime shape of the field changes.
The correct fix is to add lastDisconnect?: { at?: number } | null to ChannelAccountSnapshot (in src/channels/plugins/types.ts or wherever it is declared). Once typed, the entire extraction block can be simplified to a one-liner:
const lastDisconnectAt = typeof status.lastDisconnect?.at === "number"
? status.lastDisconnect.at
: null;As written, the cast and the manual shape-checks work at runtime, but they are invisible to the type checker and will not catch future structural regressions.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/channel-health-monitor.ts
Line: 137-144
Comment:
**Unsafe type cast for `lastDisconnect`**
`status` is typed as `ChannelAccountSnapshot`, which does not appear to declare a `lastDisconnect` field, so it is accessed by casting `status as { lastDisconnect?: unknown }`. This bypasses TypeScript's type system and will silently stop working correctly if the runtime shape of the field changes.
The correct fix is to add `lastDisconnect?: { at?: number } | null` to `ChannelAccountSnapshot` (in `src/channels/plugins/types.ts` or wherever it is declared). Once typed, the entire extraction block can be simplified to a one-liner:
```ts
const lastDisconnectAt = typeof status.lastDisconnect?.at === "number"
? status.lastDisconnect.at
: null;
```
As written, the cast and the manual shape-checks work at runtime, but they are invisible to the type checker and will not catch future structural regressions.
How can I resolve this? If you propose a fix, please make it concise.…-grace to startup log
- Replace unsafe `as { lastDisconnect?: unknown }` cast with direct
property access via the typed ChannelAccountSnapshot.lastDisconnect field
- Add reconnect-grace to the health monitor startup log so the configured
value is visible in operational logs alongside interval, startup-grace,
and channel-connect-grace
- Fix oxfmt formatting
|
Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Keep this PR open. Current main still lacks the PR's central reconnect-grace behavior: after startup connect grace, a running channel with Best possible solution: Keep this PR open as a useful implementation candidate. Rebase it onto current main, add a generic What I checked:
Remaining risk / open question:
Codex review notes: model gpt-5.5, reasoning high; reviewed against bdfb408ce6ee. |
Problem
When a Discord WebSocket drops (code 1006), the gateway lifecycle has built-in reconnection logic (resume/identify) that typically recovers within seconds. However, the health monitor runs every 5 minutes and immediately triggers a full provider teardown + restart when it sees
connected: false, racing with and destroying the in-progress reconnection.Each full restart creates a new Discord.js
Client, allocating significant heap memory. The old client's resources may not be fully GC'd before the next restart cycle. Over time this causes steady memory growth leading to OOM kills.Observed impact (real deployment)
Root cause
In
channel-health-policy.ts, whensnapshot.connected === false, the health evaluation immediately returns{ healthy: false, reason: "disconnected" }with no grace period for reconnection. The existingchannelConnectGraceMs(120s) only applies to fresh starts (lastStartAt), not to mid-lifecycle reconnections.The gateway lifecycle already has a
RECONNECT_STALL_TIMEOUT_MS(5 min) watchdog that force-stops the provider if reconnection truly stalls. The health monitor should not race with this internal logic.Solution
Add a configurable reconnect grace period (default: 5 minutes) that lets running channels recover from transient disconnects before the health monitor intervenes:
reconnectGraceMstoChannelHealthPolicyandChannelHealthTimingPolicylastDisconnectAttoChannelHealthSnapshotlastDisconnect.atfrom the runtime snapshotconnected === falsebut the channel is stillrunning, check if the disconnect happened within the grace period before marking unhealthylastEventAtas a proxy whenlastDisconnecttimestamp is unavailablenot-runningpath)Configuration
The grace period can be tuned via the timing config:
Testing
Files changed
src/gateway/channel-health-policy.ts— reconnect grace logic + constantsrc/gateway/channel-health-monitor.ts— extractlastDisconnectAt, pass to policysrc/gateway/channel-health-monitor.test.ts— new test suite + updated existing testFixes #41354
Related: #42178, #39096, #30212, #39288