Skip to content

fix(mattermost): add WebSocket ping/pong keepalive to detect silent drops#46345

Closed
Br1an67 wants to merge 1 commit into
openclaw:mainfrom
Br1an67:fix/44160
Closed

fix(mattermost): add WebSocket ping/pong keepalive to detect silent drops#46345
Br1an67 wants to merge 1 commit into
openclaw:mainfrom
Br1an67:fix/44160

Conversation

@Br1an67

@Br1an67 Br1an67 commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Add periodic WebSocket ping (30s) with pong timeout (10s) to Mattermost monitor. When TCP dies silently (NAT expiry, proxy drops), no close/error fires — the keepalive detects this and triggers ws.terminate() so runWithReconnect kicks in.

Change Type

  • Bug fix

Scope

  • Extensions (extensions/)

Linked Issue/PR

Closes #44160

Security Impact

  • No security implications

Repro + Verification

  • Build passes

Compatibility / Migration

Backward compatible — adds keepalive to existing WebSocket connection.

Risks and Mitigations

Low risk — standard WebSocket keepalive pattern.

@openclaw-barnacle openclaw-barnacle Bot added channel: mattermost Channel integration: mattermost size: XS labels Mar 14, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b7996e83f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +164 to +168
pongTimeout = setTimeout(() => {
opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
ws.terminate();
}, PONG_TIMEOUT_MS);
ws.ping();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Handle ping failures in keepalive timer

The keepalive interval calls ws.ping() without guarding for sockets that have already transitioned out of OPEN, and clearKeepalive() in close/error cannot prevent a timer callback that was already queued from running. In that race (for example, close and interval firing in the same loop tick), ws.ping() can throw synchronously and bubble as an uncaught exception, which can terminate the monitor process instead of triggering normal reconnect handling.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a standard WebSocket keepalive mechanism (30s ping / 10s pong-timeout) to the Mattermost monitor to detect silent TCP connection drops caused by NAT expiry or proxy resets — a correct and well-scoped fix. clearKeepalive() is consistently called in both the "close" and "error" event handlers, and the promise settlement path (resolveOnce/rejectOnce) is always preceded by clearKeepalive(), so there are no timer leaks in the normal flow.

  • Keepalive intervals and timeouts are correctly cleaned up on connection close and error.
  • One minor robustness concern: inside the setInterval callback, pongTimeout is overwritten without first calling clearTimeout on the previous value, and the pong-timeout callback itself doesn't reset pongTimeout = undefined after it fires. With the current constants (PING_INTERVAL_MS = 30s > PONG_TIMEOUT_MS = 10s) this is harmless — the previous timeout always fires or is cleared before the next interval tick — but it makes the code fragile if the constants are ever changed, and it causes clearKeepalive() to call clearTimeout on a dead timer unnecessarily.
  • MattermostWebSocketLike interface is correctly extended with ping(): void and the "pong" event listener signature.

Confidence Score: 4/5

  • Safe to merge — the keepalive logic is correct for all expected runtime scenarios with the current timer constants.
  • The core implementation is correct: ping interval starts on "open", pongTimeout triggers ws.terminate() on silence, and clearKeepalive() is reliably called in both "close" and "error" handlers before the promise settles. The one concern (pongTimeout overwritten without clearing the previous) is not a real bug with 30s > 10s, but is worth hardening. No security or data-loss risk.
  • No files require special attention beyond the minor robustness note on monitor-websocket.ts.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/mattermost/src/mattermost/monitor-websocket.ts
Line: 163-169

Comment:
**Stale `pongTimeout` reference if interval fires before clearing**

On each interval tick, `pongTimeout` is overwritten without first calling `clearTimeout` on the previous value. In the current setup `PING_INTERVAL_MS (30s) > PONG_TIMEOUT_MS (10s)`, so the previous timeout always fires (or gets cleared by the "pong" handler) before the next interval tick — meaning the overwrite is safe today. However, this is a fragile invariant: if the constants were ever adjusted so that `PING_INTERVAL_MS ≤ PONG_TIMEOUT_MS`, the previous active timeout would be silently leaked, and two `ws.terminate()` calls could race. Additionally, after the timeout *fires* it doesn't reset `pongTimeout = undefined`, so `clearKeepalive()` ends up calling `clearTimeout` on an already-dead timer (harmless, but the `!== undefined` guard becomes unreliable).

A more robust pattern that is immune to constant changes:

```suggestion
          pingInterval = setInterval(() => {
            clearTimeout(pongTimeout);
            pongTimeout = setTimeout(() => {
              opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
              ws.terminate();
              pongTimeout = undefined;
            }, PONG_TIMEOUT_MS);
            ws.ping();
          }, PING_INTERVAL_MS);
```

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: 0b7996e

Comment on lines +163 to +169
pingInterval = setInterval(() => {
pongTimeout = setTimeout(() => {
opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
ws.terminate();
}, PONG_TIMEOUT_MS);
ws.ping();
}, PING_INTERVAL_MS);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale pongTimeout reference if interval fires before clearing

On each interval tick, pongTimeout is overwritten without first calling clearTimeout on the previous value. In the current setup PING_INTERVAL_MS (30s) > PONG_TIMEOUT_MS (10s), so the previous timeout always fires (or gets cleared by the "pong" handler) before the next interval tick — meaning the overwrite is safe today. However, this is a fragile invariant: if the constants were ever adjusted so that PING_INTERVAL_MS ≤ PONG_TIMEOUT_MS, the previous active timeout would be silently leaked, and two ws.terminate() calls could race. Additionally, after the timeout fires it doesn't reset pongTimeout = undefined, so clearKeepalive() ends up calling clearTimeout on an already-dead timer (harmless, but the !== undefined guard becomes unreliable).

A more robust pattern that is immune to constant changes:

Suggested change
pingInterval = setInterval(() => {
pongTimeout = setTimeout(() => {
opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
ws.terminate();
}, PONG_TIMEOUT_MS);
ws.ping();
}, PING_INTERVAL_MS);
pingInterval = setInterval(() => {
clearTimeout(pongTimeout);
pongTimeout = setTimeout(() => {
opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
ws.terminate();
pongTimeout = undefined;
}, PONG_TIMEOUT_MS);
ws.ping();
}, PING_INTERVAL_MS);
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/mattermost/src/mattermost/monitor-websocket.ts
Line: 163-169

Comment:
**Stale `pongTimeout` reference if interval fires before clearing**

On each interval tick, `pongTimeout` is overwritten without first calling `clearTimeout` on the previous value. In the current setup `PING_INTERVAL_MS (30s) > PONG_TIMEOUT_MS (10s)`, so the previous timeout always fires (or gets cleared by the "pong" handler) before the next interval tick — meaning the overwrite is safe today. However, this is a fragile invariant: if the constants were ever adjusted so that `PING_INTERVAL_MS ≤ PONG_TIMEOUT_MS`, the previous active timeout would be silently leaked, and two `ws.terminate()` calls could race. Additionally, after the timeout *fires* it doesn't reset `pongTimeout = undefined`, so `clearKeepalive()` ends up calling `clearTimeout` on an already-dead timer (harmless, but the `!== undefined` guard becomes unreliable).

A more robust pattern that is immune to constant changes:

```suggestion
          pingInterval = setInterval(() => {
            clearTimeout(pongTimeout);
            pongTimeout = setTimeout(() => {
              opts.runtime.error?.("mattermost websocket pong timeout, terminating connection");
              ws.terminate();
              pongTimeout = undefined;
            }, PONG_TIMEOUT_MS);
            ws.ping();
          }, PING_INTERVAL_MS);
```

How can I resolve this? If you propose a fix, please make it concise.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f850b71eff

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +123 to +124
ws.once("pong", onPong);
ws.ping();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid assuming keepalive APIs on injected sockets

createMattermostConnectOnce still accepts any MattermostWebSocketLike from webSocketFactory, but the new keepalive path now unconditionally calls readyState, OPEN, once, ping, and removeListener, which are not part of that interface (extensions/mattermost/src/mattermost/monitor-websocket.ts defines only on/send/close/terminate). This introduces a runtime crash path for valid injected sockets (including the existing test fake in extensions/mattermost/src/mattermost/monitor-websocket.test.ts) once the 30s keepalive interval runs, because ws.once/ws.ping can be undefined and throw a TypeError instead of reconnecting cleanly.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fca9e27051

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +123 to +124
ws.once("pong", onPong);
ws.ping();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate keepalive calls to APIs outside socket contract

createMattermostConnectOnce still accepts any MattermostWebSocketLike from webSocketFactory, but the new keepalive path calls once()/ping() (and later removeListener()/readyState) that are not part of that contract. Any injected socket that previously worked by implementing only on/send/close/terminate will now throw a TypeError once the 30s interval fires, which aborts the monitor loop instead of reconnecting cleanly.

Useful? React with 👍 / 👎.

@Br1an67
Br1an67 force-pushed the fix/44160 branch 2 times, most recently from 93fc4aa to 0ed2aeb Compare March 15, 2026 04:31
…rops (closes openclaw#44160)

Add periodic WebSocket ping (30s interval) with pong timeout (10s). When a pong
is not received, the connection is terminated, triggering the existing reconnect
logic in runWithReconnect. This detects silent connection drops caused by NAT
table expiry or intermediate proxy timeouts.

Co-authored-by: Copilot <[email protected]>
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Mar 15, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7d9d4fb8cc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

id: string,
factory: ContextEngineFactory,
): ContextEngineRegistrationResult {
return registerContextEngineForOwner(id, factory, PUBLIC_CONTEXT_ENGINE_OWNER);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve core registration of default context engine

registerContextEngine() now always delegates to registerContextEngineForOwner(..., "public-sdk"), but the new owner guard rejects non-core registrations for the default context-engine slot (legacy). Core initialization still calls registerContextEngine("legacy", ...) via registerLegacyContextEngine, so that registration now returns { ok: false } and is silently ignored, leaving the default engine unregistered; subsequent resolveContextEngine() calls with default config fail with "Context engine "legacy" is not registered" during normal agent startup paths.

Useful? React with 👍 / 👎.

@Br1an67

Br1an67 commented Mar 17, 2026

Copy link
Copy Markdown
Contributor Author

Closing to manage active PR count. Will reopen when slot is available.

@Br1an67 Br1an67 closed this Mar 17, 2026
JasonWang1124 pushed a commit to JasonWang1124/openclaw that referenced this pull request Mar 30, 2026
…rops

When the TCP connection silently dies (NAT expiry, proxy timeout, network
switch), no close or error event fires. The bot appears connected but
stops receiving messages — potentially for hours before detection.

This adds a standard WebSocket ping/pong keepalive alongside the existing
health check mechanism:

- Sends ws.ping() every 30s (configurable via pingIntervalMs)
- If no pong received before the next ping, terminates the connection
- runWithReconnect then establishes a fresh connection automatically
- All timers cleaned up reliably via the existing clearTimers() function

The ping/pong mechanism complements the existing getBotUpdateAt health
check: ping/pong detects dead TCP connections, while getBotUpdateAt
detects silenced-but-alive connections after bot account modifications.

Refs: openclaw#51104, openclaw#44160, openclaw#41837, openclaw#50138
Supersedes: openclaw#46345 (closed by author for PR count management)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: mattermost Channel integration: mattermost docs Improvements or additions to documentation size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mattermost WebSocket: add ping/pong keepalive to detect silent connection drops

1 participant