fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#663
Open
BingqingLyu wants to merge 3 commits into
Open
fix(health-monitor): add reconnect grace period to prevent excessive Discord restarts#663BingqingLyu wants to merge 3 commits into
BingqingLyu wants to merge 3 commits into
Conversation
…Discord restarts When a Discord WebSocket drops (code 1006), the gateway lifecycle's built-in reconnection logic (resume/identify) typically recovers within seconds. However, the health monitor checks every 5 minutes and immediately restarts the entire provider when it sees connected=false, racing with and destroying the in-progress reconnection. Each full restart tears down the Discord.js client and creates a new one, allocating significant memory that may not be fully GC'd. Over time this causes memory growth leading to OOM kills — observed as ~72 restarts/day (505 in 7 days) on a stable network with 0% packet loss. This change adds a configurable reconnect grace period (default: 5min) that lets running channels recover from transient disconnects before the health monitor intervenes with a full teardown/restart cycle: - Add reconnectGraceMs to ChannelHealthPolicy and timing config - Add lastDisconnectAt to ChannelHealthSnapshot - Extract lastDisconnect.at from runtime snapshot in health monitor - When connected=false but channel is still running, check if disconnect is recent (within grace period) before restarting - Fall back to lastEventAt when lastDisconnect timestamp is missing - Non-running channels bypass grace (handled by not-running path) Fixes openclaw#41354 Related: openclaw#42178, openclaw#39096, openclaw#30212
…-grace to startup log
- Replace unsafe `as { lastDisconnect?: unknown }` cast with direct
property access via the typed ChannelAccountSnapshot.lastDisconnect field
- Add reconnect-grace to the health monitor startup log so the configured
value is visible in operational logs alongside interval, startup-grace,
and channel-connect-grace
- Fix oxfmt formatting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a Discord WebSocket drops (code 1006), the gateway lifecycle has built-in reconnection logic (resume/identify) that typically recovers within seconds. However, the health monitor runs every 5 minutes and immediately triggers a full provider teardown + restart when it sees
connected: false, racing with and destroying the in-progress reconnection.Each full restart creates a new Discord.js
Client, allocating significant heap memory. The old client's resources may not be fully GC'd before the next restart cycle. Over time this causes steady memory growth leading to OOM kills.Observed impact (real deployment)
Root cause
In
channel-health-policy.ts, whensnapshot.connected === false, the health evaluation immediately returns{ healthy: false, reason: "disconnected" }with no grace period for reconnection. The existingchannelConnectGraceMs(120s) only applies to fresh starts (lastStartAt), not to mid-lifecycle reconnections.The gateway lifecycle already has a
RECONNECT_STALL_TIMEOUT_MS(5 min) watchdog that force-stops the provider if reconnection truly stalls. The health monitor should not race with this internal logic.Solution
Add a configurable reconnect grace period (default: 5 minutes) that lets running channels recover from transient disconnects before the health monitor intervenes:
reconnectGraceMstoChannelHealthPolicyandChannelHealthTimingPolicylastDisconnectAttoChannelHealthSnapshotlastDisconnect.atfrom the runtime snapshotconnected === falsebut the channel is stillrunning, check if the disconnect happened within the grace period before marking unhealthylastEventAtas a proxy whenlastDisconnecttimestamp is unavailablenot-runningpath)Configuration
The grace period can be tuned via the timing config:
Testing
Files changed
src/gateway/channel-health-policy.ts— reconnect grace logic + constantsrc/gateway/channel-health-monitor.ts— extractlastDisconnectAt, pass to policysrc/gateway/channel-health-monitor.test.ts— new test suite + updated existing testFixes openclaw#41354
Related: openclaw#42178, openclaw#39096, openclaw#30212, openclaw#39288