Bug Description

The Mattermost WebSocket channel can silently disconnect and remain unresponsive for 12+ hours while the gateway process itself stays alive. The channel health monitor fails to detect and recover from this state.

Environment

• OpenClaw version: 2026.3.13
• OS: macOS 15 (arm64)
• Channel: Mattermost (WebSocket mode)
• Node.js: v25.8.1

Steps to Reproduce

1. Run OpenClaw gateway with Mattermost channel connected via WebSocket
2. Wait for a network interruption or WebSocket disconnect event
3. Observe that the gateway process remains alive (confirmed via macOS system log processID tracking), but the Mattermost channel stops processing messages
4. No error or warning is logged after the disconnect

Expected Behavior

The channel health monitor should detect the silent disconnection within a few minutes and automatically restart the Mattermost channel.

Actual Behavior

• Gateway process (PID 85440) stayed alive from 21:04 to 09:27 next day (12+ hours)
• macOS system log confirmed the process was running the entire time (network path checks every minute showed satisfied)
• Last OpenClaw application-level log entry was at 21:05:10 (delivered reply to user:...)
• Zero log entries from 21:05 to 09:27 — no errors, no warnings, no health monitor output
• The health monitor (configured at default 300s interval) did not detect or restart the channel

Root Cause Analysis

Looking at the source code in gateway-cli-Ol-vpIk7.js, the evaluateChannelHealth() function has a gap in its detection logic:

// Line ~1596: stale-socket detection is gated on connected === true
if (policy.channelId !== "telegram" && snapshot.mode !== "webhook" 
    && snapshot.connected === true && snapshot.lastEventAt != null) {
    if (now - snapshot.lastEventAt > staleEventThresholdMs) 
        return { healthy: false, reason: "stale-socket" };
}

If the Mattermost WebSocket disconnects but the internal snapshot.connected state becomes false, the stale-socket detection path is skipped entirely. The monitor then falls through to the disconnected branch, which only triggers a restart if reconnectAttempts >= 10. If the plugin's internal reconnection logic stalls (e.g., exponential backoff grows too large, or reconnect count never reaches 10), the health monitor considers the channel "still trying" and never intervenes.

Additionally, there appears to be a scenario where the channel enters a zombie state — the process is alive, the channel object exists, but no events are processed and no logs are written. This suggests the event loop or message processing pipeline may be blocked/deadlocked, which the health monitor cannot detect because it only checks metadata snapshots, not actual liveness.

Suggested Fixes

1. Add active liveness probing: The health monitor should periodically send a ping/probe through the actual WebSocket connection, not just check metadata snapshots
2. Treat prolonged silence as unhealthy: If a channel has connected === true but lastEventAt has not updated in staleEventThresholdMs (default 30 min), it should be considered unhealthy regardless — the current code does this, but the zombie state where connected is neither clearly true nor false may bypass all checks
3. Log health monitor evaluations: Even when healthy, periodically log the health status so that silent failures are diagnosable (currently zero log output for 12 hours makes debugging very difficult)
4. Add a watchdog for application-level log silence: If the gateway writes no application log entries for N minutes while the process is alive, something is wrong

Workaround

Reduce the health check interval to catch issues faster:

openclaw config set gateway.channelHealthCheckMinutes 1
openclaw gateway restart

This does not fix the root cause but reduces the window of undetected failure.