Skip to content

[Feature]: Health monitor stale-socket threshold should be configurable (gateway.staleEventThresholdMinutes) #35532

Description

@shejianmin

Summary

Expose the hardcoded DEFAULT_STALE_EVENT_THRESHOLD_MS (30 minutes) in the channel health monitor as a user-configurable option gateway.staleEventThresholdMinutes, with hot-reload support. This allows operators to tune or disable the stale-socket detection for low-traffic channels that are falsely flagged as dead.

Problem to solve

The channel health monitor treats any connected channel that receives no application-level events for 30 minutes as a stale socket and forces a WebSocket restart. This is a reasonable heuristic for high-traffic channels, but creates a destructive loop on low-traffic ones:

  1. Channel is idle for 30 min (no user messages — perfectly normal for off-peak hours)
  2. Health monitor marks the healthy WebSocket as stale-socket
  3. WebSocket is forcibly restarted (stop → reconnect)
  4. Messages arriving during the ~1s restart window are silently dropped
  5. Users see a permanent "⏳ Thinking..." indicator because the bot never responds
  6. 35 minutes later, the cycle repeats

The underlying WebSocket transport is fully alive — the Lark SDK's own ping/pong (120s server-negotiated interval) continues to succeed. The health monitor conflates "no user traffic" with "dead connection".

Currently there is no way to adjust this threshold without patching the compiled dist files. The only built-in workaround is gateway.channelHealthCheckMinutes: 0, which disables the entire health monitor — losing the ability to detect genuinely dead connections.

Proposed solution

Add gateway.staleEventThresholdMinutes to the config schema, wired through the existing staleEventThresholdMs parameter that resolveTimingPolicy() already supports but no caller currently passes.

Changes required

1. Config schema (Zod definition alongside channelHealthCheckMinutes):

staleEventThresholdMinutes: z.number().int().min(0).optional(),

2. Gateway startup — read and forward:

const staleThresholdMinutes = cfg.gateway?.staleEventThresholdMinutes;
startChannelHealthMonitor({
    channelManager,
    checkIntervalMs: (healthCheckMinutes ?? 5) * 6e4,
    staleEventThresholdMs: staleThresholdMinutes != null ? staleThresholdMinutes * 6e4 : undefined
});

3. Hot-reload factory — accept and forward:

createHealthMonitor: (checkIntervalMs, staleEventThresholdMs) => startChannelHealthMonitor({
    channelManager,
    checkIntervalMs,
    staleEventThresholdMs
});

4. Hot-reload execution — read from updated config:

const staleMinutes = nextConfig.gateway?.staleEventThresholdMinutes;
params.createHealthMonitor(
    (minutes ?? 5) * 6e4,
    staleMinutes != null ? staleMinutes * 6e4 : undefined
);

5. Config reload registry — register as hot-reloadable:

{ prefix: "gateway.staleEventThresholdMinutes", kind: "hot", actions: ["restart-health-monitor"] }

6. Config description:

"gateway.staleEventThresholdMinutes": "How long (in minutes) a connected channel can go without
receiving any event before the health monitor treats it as stale and triggers a restart.
Default is 30. Increase for low-traffic channels to avoid unnecessary reconnects."

Behavior

Config value Effect
Not set Uses existing 30-minute default (fully backward compatible)
240 4 hours — moderate-traffic channels
1440 24 hours — low-traffic channels
0 Effectively disables stale-socket detection

Alternatives considered

Alternative Why not
gateway.channelHealthCheckMinutes: 0 Disables the entire health monitor, losing detection of genuinely dead connections
Increase the hardcoded default to a larger value Would reduce the issue but doesn't let operators tune per their traffic pattern; may mask real dead-connection scenarios for high-traffic channels
Use WebSocket-level ping/pong as the liveness signal instead of application events More correct but requires deeper refactoring of the health monitor to integrate with each channel SDK's transport layer
Per-channel stale threshold override More granular but adds config complexity; a single global knob covers 90% of use cases

Impact

  • Backward compatible: No behavior change when the config key is absent
  • Minimal code change: The staleEventThresholdMs parameter path already exists in resolveTimingPolicy() — this just wires it to user config
  • Hot-reloadable: Operators can adjust at runtime without gateway restart
  • Affects all WebSocket-based channels: Feishu, Slack, Discord, etc. — any channel that may have idle periods

Evidence/examples

Production instance*: OpenClaw 2026.3.2 (85377a2), Feishu channel in WebSocket mode, @larksuiteoapi/node-sdk 1.59.0.

7 stale-socket restarts in 4 hours on an otherwise healthy connection:

02:27:04 [health-monitor] [feishu:main] restarting (reason: stale-socket)
03:02:04 [health-monitor] [feishu:main] restarting (reason: stale-socket)  # +35min
03:37:04 [health-monitor] [feishu:main] restarting (reason: stale-socket)  # +35min
04:21:04 [health-monitor] [feishu:main] restarting (reason: stale-socket)  # +44min
04:56:04 [health-monitor] [feishu:main] restarting (reason: stale-socket)  # +35min

After applying a dist-level patch with staleEventThresholdMinutes: 1440: zero spurious restarts, Feishu WebSocket stable, all messages delivered.

Additional information

  • The Lark SDK's built-in WebSocket ping/pong (120s interval, server-negotiated via ClientConfig.PingInterval) keeps the TCP connection alive. The health monitor's stale detection operates at a higher layer (application events only) and does not account for transport-level liveness.
  • We validated the proposed change by patching compiled dist files on a production instance. A dist-level patch description is available at: patches/gateway-stale-event-threshold-config.patch (referenced for context, not a formal PR).
  • Happy to submit a source-level PR if the team agrees with the approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions