Skip to content

[Bug]: Mattermost WebSocket add ping/pong keepalive to detect half-dead connections #51104

Description

@meredith620

Problem

The Mattermost channel plugin uses a plain WebSocket connection (monitor-websocket.tscreateMattermostConnectOnce) that does not send WebSocket-level ping frames. This means half-dead connections (TCP ESTABLISHED but no application-layer data flow) go undetected until the staleEventThresholdMs timer fires (default 30 minutes).

Observed behavior

  1. Gateway process is running, /health returns {"ok":true}
  2. TCP connections to the Mattermost server show ESTABLISHED in ss
  3. Bot stops responding to messages on Mattermost
  4. sessions_list shows zero Mattermost sessions — only webchat
  5. The channel health monitor does not restart the channel because connected === true and lastEventAt hasn't crossed the 30-minute stale threshold yet
  6. Workaround: trigger a config reload by touching channels.mattermost in openclaw.json

Root cause

In monitor-websocket.ts, the connectOnce function only listens for open, message, close, and error events. There is no periodic ws.ping() call, so the client cannot distinguish between "nobody is talking" and "the connection is dead".

The staleEventThresholdMs in channel-health-policy.ts conflates silence with failure — on a quiet channel, it will either:

  • Miss dead connections for 30 minutes, or
  • If lowered aggressively (e.g. 5 min), cause unnecessary reconnect storms on idle channels

Proposed solution

1. Add WebSocket ping/pong keepalive (primary fix)

Use the ws library's native ping()/on("pong") support:

ws.on("open", () => {
  // ... existing auth code ...

  let pongReceived = true;
  const pingTimer = setInterval(() => {
    if (!pongReceived) {
      clearInterval(pingTimer);
      ws.terminate(); // force close → triggers reconnect
      return;
    }
    pongReceived = false;
    if (ws.readyState === WebSocket.OPEN) {
      ws.ping();
    }
  }, 30_000); // every 30s

  ws.on("pong", () => { pongReceived = true; });
  ws.on("close", () => { clearInterval(pingTimer); });
});

This detects dead connections within ~60 seconds (2 missed pong cycles) regardless of channel activity.

2. Expose staleEventThresholdMs as a config option (secondary)

Currently hardcoded as 18e5 (30 min) in channel-health-policy.ts. Exposing it as e.g. gateway.channelHealthStaleThresholdMinutes would let operators tune the fallback detection independently.

Environment

  • OpenClaw version: 2026.3.13 (61d171a)
  • Mattermost server: self-hosted
  • OS: Debian Linux 6.1
  • Channel config: 3 Mattermost bot accounts, channelHealthCheckMinutes: 1

Workaround

Trigger a manual channel reload by adding/removing a field in channels.mattermost in openclaw.json. The chokidar file watcher detects the change and restarts the Mattermost channel accounts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions