Problem
The Mattermost channel plugin uses a plain WebSocket connection (monitor-websocket.ts → createMattermostConnectOnce) that does not send WebSocket-level ping frames. This means half-dead connections (TCP ESTABLISHED but no application-layer data flow) go undetected until the staleEventThresholdMs timer fires (default 30 minutes).
Observed behavior
- Gateway process is running,
/health returns {"ok":true}
- TCP connections to the Mattermost server show
ESTABLISHED in ss
- Bot stops responding to messages on Mattermost
sessions_list shows zero Mattermost sessions — only webchat
- The channel health monitor does not restart the channel because
connected === true and lastEventAt hasn't crossed the 30-minute stale threshold yet
- Workaround: trigger a config reload by touching
channels.mattermost in openclaw.json
Root cause
In monitor-websocket.ts, the connectOnce function only listens for open, message, close, and error events. There is no periodic ws.ping() call, so the client cannot distinguish between "nobody is talking" and "the connection is dead".
The staleEventThresholdMs in channel-health-policy.ts conflates silence with failure — on a quiet channel, it will either:
- Miss dead connections for 30 minutes, or
- If lowered aggressively (e.g. 5 min), cause unnecessary reconnect storms on idle channels
Proposed solution
1. Add WebSocket ping/pong keepalive (primary fix)
Use the ws library's native ping()/on("pong") support:
ws.on("open", () => {
// ... existing auth code ...
let pongReceived = true;
const pingTimer = setInterval(() => {
if (!pongReceived) {
clearInterval(pingTimer);
ws.terminate(); // force close → triggers reconnect
return;
}
pongReceived = false;
if (ws.readyState === WebSocket.OPEN) {
ws.ping();
}
}, 30_000); // every 30s
ws.on("pong", () => { pongReceived = true; });
ws.on("close", () => { clearInterval(pingTimer); });
});
This detects dead connections within ~60 seconds (2 missed pong cycles) regardless of channel activity.
2. Expose staleEventThresholdMs as a config option (secondary)
Currently hardcoded as 18e5 (30 min) in channel-health-policy.ts. Exposing it as e.g. gateway.channelHealthStaleThresholdMinutes would let operators tune the fallback detection independently.
Environment
- OpenClaw version: 2026.3.13 (61d171a)
- Mattermost server: self-hosted
- OS: Debian Linux 6.1
- Channel config: 3 Mattermost bot accounts,
channelHealthCheckMinutes: 1
Workaround
Trigger a manual channel reload by adding/removing a field in channels.mattermost in openclaw.json. The chokidar file watcher detects the change and restarts the Mattermost channel accounts.
Problem
The Mattermost channel plugin uses a plain WebSocket connection (
monitor-websocket.ts→createMattermostConnectOnce) that does not send WebSocket-level ping frames. This means half-dead connections (TCP ESTABLISHED but no application-layer data flow) go undetected until thestaleEventThresholdMstimer fires (default 30 minutes).Observed behavior
/healthreturns{"ok":true}ESTABLISHEDinsssessions_listshows zero Mattermost sessions — only webchatconnected === trueandlastEventAthasn't crossed the 30-minute stale threshold yetchannels.mattermostinopenclaw.jsonRoot cause
In
monitor-websocket.ts, theconnectOncefunction only listens foropen,message,close, anderrorevents. There is no periodicws.ping()call, so the client cannot distinguish between "nobody is talking" and "the connection is dead".The
staleEventThresholdMsinchannel-health-policy.tsconflates silence with failure — on a quiet channel, it will either:Proposed solution
1. Add WebSocket ping/pong keepalive (primary fix)
Use the
wslibrary's nativeping()/on("pong")support:This detects dead connections within ~60 seconds (2 missed pong cycles) regardless of channel activity.
2. Expose
staleEventThresholdMsas a config option (secondary)Currently hardcoded as
18e5(30 min) inchannel-health-policy.ts. Exposing it as e.g.gateway.channelHealthStaleThresholdMinuteswould let operators tune the fallback detection independently.Environment
channelHealthCheckMinutes: 1Workaround
Trigger a manual channel reload by adding/removing a field in
channels.mattermostinopenclaw.json. The chokidar file watcher detects the change and restarts the Mattermost channel accounts.