Summary

The Mattermost channel health monitor fails to detect a "deaf websocket" condition where the websocket connection appears healthy (connected, no errors) but is not actually delivering inbound messages to the agent.

Observed Behavior

• Date: 2026-03-09
• Version: 2026.3.3
• Mattermost websocket disconnected at ~07:47 PDT with ETIMEDOUT error
• Health monitor correctly detected the failure and restarted the connection at 07:48
• After reconnect, the websocket appeared connected (connected as @bullwinkle logged)
• Outbound messages worked fine — sends to Mattermost DMs were delivered successfully
• Inbound messages were silently dropped — no messages received for ~13 hours until a full gateway restart at ~21:18 PDT
• The health monitor did NOT flag this as unhealthy during that entire period

Expected Behavior

The health monitor should detect that a channel is connected but not receiving messages and trigger a reconnect. This could be implemented via:

1. Application-level ping/pong — periodically send a message to self or check for server-side heartbeat frames
2. Message flow monitoring — track "last inbound message received" timestamp and flag if it exceeds a threshold (e.g., no inbound for 30+ minutes on an active channel)
3. Echo test — periodically fetch recent messages via REST API and compare against what was received via websocket

Comparison with Slack

The Slack health monitor correctly detected stale-socket conditions on the same day and automatically restarted the connection multiple times (logged as [health-monitor] [slack:default] health-monitor: restarting (reason: stale-socket)). The Mattermost health monitor lacks this stale-socket detection.

Relevant Logs

2026-03-09T07:47:41.327-07:00 [mattermost] websocket error: Error: connect ETIMEDOUT 146.190.134.110:443
2026-03-09T07:47:41.333-07:00 [mattermost] connection failed: WebSocketClosedBeforeOpenError: websocket closed before open (code 1006)
2026-03-09T07:48:53.588-07:00 [health-monitor] [mattermost:default] health-monitor: restarting (reason: disconnected)
2026-03-09T07:48:53.599-07:00 [mattermost] [default] starting channel
2026-03-09T07:48:53.793-07:00 [mattermost] connected as @bullwinkle
# ^^^ After this point: outbound works, inbound silently dead for 13 hours
# No further health-monitor entries for mattermost until gateway restart

Environment

• OpenClaw 2026.3.3
• macOS (arm64), Node v25.8.0
• Mattermost server: self-hosted, accessed via websocket over HTTPS
• Gateway mode: local (loopback)

Workaround

Full gateway restart resolves the issue.