fix(mattermost): add WebSocket ping/pong keepalive to detect silent drops#46345
fix(mattermost): add WebSocket ping/pong keepalive to detect silent drops#46345Br1an67 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b7996e83f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| pongTimeout = setTimeout(() => { | ||
| opts.runtime.error?.("mattermost websocket pong timeout, terminating connection"); | ||
| ws.terminate(); | ||
| }, PONG_TIMEOUT_MS); | ||
| ws.ping(); |
There was a problem hiding this comment.
Handle ping failures in keepalive timer
The keepalive interval calls ws.ping() without guarding for sockets that have already transitioned out of OPEN, and clearKeepalive() in close/error cannot prevent a timer callback that was already queued from running. In that race (for example, close and interval firing in the same loop tick), ws.ping() can throw synchronously and bubble as an uncaught exception, which can terminate the monitor process instead of triggering normal reconnect handling.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR adds a standard WebSocket keepalive mechanism (30s ping / 10s pong-timeout) to the Mattermost monitor to detect silent TCP connection drops caused by NAT expiry or proxy resets — a correct and well-scoped fix.
Confidence Score: 4/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: extensions/mattermost/src/mattermost/monitor-websocket.ts
Line: 163-169
Comment:
**Stale `pongTimeout` reference if interval fires before clearing**
On each interval tick, `pongTimeout` is overwritten without first calling `clearTimeout` on the previous value. In the current setup `PING_INTERVAL_MS (30s) > PONG_TIMEOUT_MS (10s)`, so the previous timeout always fires (or gets cleared by the "pong" handler) before the next interval tick — meaning the overwrite is safe today. However, this is a fragile invariant: if the constants were ever adjusted so that `PING_INTERVAL_MS ≤ PONG_TIMEOUT_MS`, the previous active timeout would be silently leaked, and two `ws.terminate()` calls could race. Additionally, after the timeout *fires* it doesn't reset `pongTimeout = undefined`, so `clearKeepalive()` ends up calling `clearTimeout` on an already-dead timer (harmless, but the `!== undefined` guard becomes unreliable).
A more robust pattern that is immune to constant changes:
```suggestion
pingInterval = setInterval(() => {
clearTimeout(pongTimeout);
pongTimeout = setTimeout(() => {
opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
ws.terminate();
pongTimeout = undefined;
}, PONG_TIMEOUT_MS);
ws.ping();
}, PING_INTERVAL_MS);
```
How can I resolve this? If you propose a fix, please make it concise.Last reviewed commit: 0b7996e |
| pingInterval = setInterval(() => { | ||
| pongTimeout = setTimeout(() => { | ||
| opts.runtime.error?.("mattermost websocket pong timeout, terminating connection"); | ||
| ws.terminate(); | ||
| }, PONG_TIMEOUT_MS); | ||
| ws.ping(); | ||
| }, PING_INTERVAL_MS); |
There was a problem hiding this comment.
Stale pongTimeout reference if interval fires before clearing
On each interval tick, pongTimeout is overwritten without first calling clearTimeout on the previous value. In the current setup PING_INTERVAL_MS (30s) > PONG_TIMEOUT_MS (10s), so the previous timeout always fires (or gets cleared by the "pong" handler) before the next interval tick — meaning the overwrite is safe today. However, this is a fragile invariant: if the constants were ever adjusted so that PING_INTERVAL_MS ≤ PONG_TIMEOUT_MS, the previous active timeout would be silently leaked, and two ws.terminate() calls could race. Additionally, after the timeout fires it doesn't reset pongTimeout = undefined, so clearKeepalive() ends up calling clearTimeout on an already-dead timer (harmless, but the !== undefined guard becomes unreliable).
A more robust pattern that is immune to constant changes:
| pingInterval = setInterval(() => { | |
| pongTimeout = setTimeout(() => { | |
| opts.runtime.error?.("mattermost websocket pong timeout, terminating connection"); | |
| ws.terminate(); | |
| }, PONG_TIMEOUT_MS); | |
| ws.ping(); | |
| }, PING_INTERVAL_MS); | |
| pingInterval = setInterval(() => { | |
| clearTimeout(pongTimeout); | |
| pongTimeout = setTimeout(() => { | |
| opts.runtime.error?.("mattermost websocket pong timeout, terminating connection"); | |
| ws.terminate(); | |
| pongTimeout = undefined; | |
| }, PONG_TIMEOUT_MS); | |
| ws.ping(); | |
| }, PING_INTERVAL_MS); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/mattermost/src/mattermost/monitor-websocket.ts
Line: 163-169
Comment:
**Stale `pongTimeout` reference if interval fires before clearing**
On each interval tick, `pongTimeout` is overwritten without first calling `clearTimeout` on the previous value. In the current setup `PING_INTERVAL_MS (30s) > PONG_TIMEOUT_MS (10s)`, so the previous timeout always fires (or gets cleared by the "pong" handler) before the next interval tick — meaning the overwrite is safe today. However, this is a fragile invariant: if the constants were ever adjusted so that `PING_INTERVAL_MS ≤ PONG_TIMEOUT_MS`, the previous active timeout would be silently leaked, and two `ws.terminate()` calls could race. Additionally, after the timeout *fires* it doesn't reset `pongTimeout = undefined`, so `clearKeepalive()` ends up calling `clearTimeout` on an already-dead timer (harmless, but the `!== undefined` guard becomes unreliable).
A more robust pattern that is immune to constant changes:
```suggestion
pingInterval = setInterval(() => {
clearTimeout(pongTimeout);
pongTimeout = setTimeout(() => {
opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
ws.terminate();
pongTimeout = undefined;
}, PONG_TIMEOUT_MS);
ws.ping();
}, PING_INTERVAL_MS);
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f850b71eff
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ws.once("pong", onPong); | ||
| ws.ping(); |
There was a problem hiding this comment.
Avoid assuming keepalive APIs on injected sockets
createMattermostConnectOnce still accepts any MattermostWebSocketLike from webSocketFactory, but the new keepalive path now unconditionally calls readyState, OPEN, once, ping, and removeListener, which are not part of that interface (extensions/mattermost/src/mattermost/monitor-websocket.ts defines only on/send/close/terminate). This introduces a runtime crash path for valid injected sockets (including the existing test fake in extensions/mattermost/src/mattermost/monitor-websocket.test.ts) once the 30s keepalive interval runs, because ws.once/ws.ping can be undefined and throw a TypeError instead of reconnecting cleanly.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fca9e27051
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ws.once("pong", onPong); | ||
| ws.ping(); |
There was a problem hiding this comment.
Gate keepalive calls to APIs outside socket contract
createMattermostConnectOnce still accepts any MattermostWebSocketLike from webSocketFactory, but the new keepalive path calls once()/ping() (and later removeListener()/readyState) that are not part of that contract. Any injected socket that previously worked by implementing only on/send/close/terminate will now throw a TypeError once the 30s interval fires, which aborts the monitor loop instead of reconnecting cleanly.
Useful? React with 👍 / 👎.
93fc4aa to
0ed2aeb
Compare
…rops (closes openclaw#44160) Add periodic WebSocket ping (30s interval) with pong timeout (10s). When a pong is not received, the connection is terminated, triggering the existing reconnect logic in runWithReconnect. This detects silent connection drops caused by NAT table expiry or intermediate proxy timeouts. Co-authored-by: Copilot <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d9d4fb8cc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| id: string, | ||
| factory: ContextEngineFactory, | ||
| ): ContextEngineRegistrationResult { | ||
| return registerContextEngineForOwner(id, factory, PUBLIC_CONTEXT_ENGINE_OWNER); |
There was a problem hiding this comment.
Preserve core registration of default context engine
registerContextEngine() now always delegates to registerContextEngineForOwner(..., "public-sdk"), but the new owner guard rejects non-core registrations for the default context-engine slot (legacy). Core initialization still calls registerContextEngine("legacy", ...) via registerLegacyContextEngine, so that registration now returns { ok: false } and is silently ignored, leaving the default engine unregistered; subsequent resolveContextEngine() calls with default config fail with "Context engine "legacy" is not registered" during normal agent startup paths.
Useful? React with 👍 / 👎.
|
Closing to manage active PR count. Will reopen when slot is available. |
…rops When the TCP connection silently dies (NAT expiry, proxy timeout, network switch), no close or error event fires. The bot appears connected but stops receiving messages — potentially for hours before detection. This adds a standard WebSocket ping/pong keepalive alongside the existing health check mechanism: - Sends ws.ping() every 30s (configurable via pingIntervalMs) - If no pong received before the next ping, terminates the connection - runWithReconnect then establishes a fresh connection automatically - All timers cleaned up reliably via the existing clearTimers() function The ping/pong mechanism complements the existing getBotUpdateAt health check: ping/pong detects dead TCP connections, while getBotUpdateAt detects silenced-but-alive connections after bot account modifications. Refs: openclaw#51104, openclaw#44160, openclaw#41837, openclaw#50138 Supersedes: openclaw#46345 (closed by author for PR count management)
Summary
Add periodic WebSocket ping (30s) with pong timeout (10s) to Mattermost monitor. When TCP dies silently (NAT expiry, proxy drops), no close/error fires — the keepalive detects this and triggers ws.terminate() so runWithReconnect kicks in.
Change Type
Scope
extensions/)Linked Issue/PR
Closes #44160
Security Impact
Repro + Verification
Compatibility / Migration
Backward compatible — adds keepalive to existing WebSocket connection.
Risks and Mitigations
Low risk — standard WebSocket keepalive pattern.