fix(gateway): stop stale-socket restarts before first event by Takhoffman · Pull Request #38643 · openclaw/openclaw

Takhoffman · 2026-03-07T06:33:54Z

Summary

Describe the problem and fix in 2–5 bullets:

Problem: gateway stale-socket health checks restarted connected channels that had never published event-liveness timestamps, which caused Telegram reconnect storms and duplicate delivery loops.
Why it matters: quiet or freshly reconnected channels could be recycled indefinitely before their first real event, destabilizing sessions and creating duplicate downstream work.
What changed: stale-socket recovery now requires explicit event-liveness tracking, and connect-time liveness seeding is centralized in a shared helper used by Slack, Discord, and web channel startup paths.
What did NOT change (scope boundary): auth, transport retry policy, message parsing, and non-stale health outcomes are unchanged.

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

User-visible / Behavior Changes

Telegram-shaped channels no longer get restarted as stale-socket before they have published any event-liveness timestamp. Slack, Discord, and web channels now seed connect-time liveness consistently so stale-socket recovery still works after a real connection is established.

Security Impact (required)

New permissions/capabilities? (Yes/No) No
Secrets/tokens handling changed? (Yes/No) No
New/changed network calls? (Yes/No) No
Command/tool execution surface changed? (Yes/No) No
Data access scope changed? (Yes/No) No
If any Yes, explain risk + mitigation:

Repro + Verification

Environment

OS: macOS
Runtime/container: Node.js via pnpm
Model/provider: N/A
Integration/channel (if any): Gateway health monitor; Telegram/Slack/Discord/Web providers
Relevant config (redacted): default local test config; no secret/config changes required

Steps

Start a channel provider that reports connected: true before receiving its first inbound event.
Let the channel health monitor evaluate a long-lived idle connection.
Observe whether the provider is restarted as stale-socket without any recorded event-liveness.

Expected

Channels without event-liveness tracking stay healthy, while channels that do track liveness can still be restarted when they go stale.

Actual

Before the fix, connected channels with lastEventAt: null could be restarted repeatedly, causing reconnect churn and duplicate delivery loops.

Evidence

Attach at least one:

Failing test/log before + passing after
Trace/log snippets
Screenshot/recording
Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

Verified scenarios: focused stale-socket policy and monitor regressions, shared connect-time liveness helper coverage, Slack connect-time status seeding, full repo test suite.
Edge cases checked: no-event channels remain healthy; Slack/Discord/web connected paths still seed stale-socket liveness; lastError clearing remains intact for Slack connect.
What you did not verify: live production channel sessions outside automated tests.

Compatibility / Migration

Backward compatible? (Yes/No) Yes
Config/env changes? (Yes/No) No
Migration needed? (Yes/No) No
If yes, exact upgrade steps:

Failure Recovery (if this breaks)

How to disable/revert this change quickly: revert commits 0d6110e08eac92a42124800b90a9138637f2c367 and e5af30283abf051964a1b5f0eb8a4a734e617d08 from the branch.
Files/config to restore: src/gateway/channel-health-policy.ts, provider status publishers, and src/gateway/channel-status-patches.ts.
Known bad symptoms reviewers should watch for: stale-socket restarts stop firing for connected channels that should recover, or channels begin reconnecting before their first inbound event again.

Risks and Mitigations

List only real risks for this PR. Add/remove entries as needed. If none, write None.

Risk: a provider path that should participate in stale-socket recovery might bypass the shared connect-time liveness helper.
- Mitigation: the helper is covered by a focused test and adopted in the currently touched Slack, Discord, and web connected paths.

greptile-apps · 2026-03-07T06:38:19Z

Greptile Summary

This PR fixes a Telegram (and Telegram-shaped channel) restart storm caused by the stale-socket health check mishandling channels that never publish lastEventAt timestamps. The core fix is in evaluateChannelHealth (channel-health-policy.ts): the old logic entered the stale-socket branch whenever lastStartAt != null, then fell back to lastEventAt ?? 0 — epoch-0 — as the "last event", causing any idle connected channel to appear stale. The new logic gates the entire stale-socket path on lastEventAt != null, so channels that opt out of event-liveness tracking are never restarted as stale.

To keep stale-socket recovery working for channels that do track liveness (Slack, Discord, Web/WhatsApp), a shared createConnectedChannelStatusPatch helper is introduced that seeds both lastConnectedAt and lastEventAt to the same connect-time timestamp. All three provider connect paths now use this helper, ensuring the stale timer starts from a known good baseline rather than epoch-0.

Key observations:

The removal of the upDuration > staleEventThresholdMs guard is a subtle behavioral change: a channel with an old inherited lastEventAt (patch-merged from a previous lifecycle) that has just exited the startup-connect grace window could now be flagged stale-socket before its first createConnectedChannelStatusPatch call. In practice this is guarded by the connected: false → disconnected fast-path that all three providers set on disconnect, but it is worth being aware of.
src/web/auto-reply/monitor.ts assigns the connected status patch field-by-field rather than using object spread like every other callsite; this is fragile if ConnectedChannelStatusPatch ever gains new fields (see inline comment).
Test coverage is solid: policy-level unit tests, monitor-level integration tests for the Telegram opt-out case, and a dedicated test for the new helper and the Slack connect-status publisher.

Confidence Score: 4/5

Safe to merge; the core fix is correct and well-tested, with one minor style inconsistency in the web monitor.
The fix correctly addresses the false-stale-socket root cause, all three provider connect paths are covered by the shared helper, and tests exercise both the opt-out (Telegram) and opt-in (Slack) paths. The one point deduction is for the verbose field-by-field assignment in web/auto-reply/monitor.ts which diverges from every other callsite and could silently miss future fields, and for the undocumented behavioral change around removal of the upDuration guard (unlikely to cause issues in practice given connected: false disconnect signals, but worth tracking).
src/web/auto-reply/monitor.ts — verbose field-by-field patch assignment should match other callsites; src/gateway/channel-health-policy.ts — removal of upDuration guard is correct but reviewers should be aware of the edge case with inherited stale lastEventAt across lifecycle restarts.

_{Last reviewed commit: 0d6110e}

src/web/auto-reply/monitor.ts

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d6110e08e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/gateway/channel-health-policy.ts

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d5a5dc64e3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/gateway/channel-health-policy.ts

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b47b76116

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-03-07T07:00:24Z

src/gateway/channel-health-policy.ts

+    if (lastStartAt != null && snapshot.lastEventAt < lastStartAt) {
+      return { healthy: true, reason: "healthy" };


Bound inherited-event bypass by lifecycle timeout

The new early return for lastEventAt < lastStartAt makes stale-socket checks permanently skip when a restarted lifecycle inherits old status and then never publishes a fresh event timestamp. This can happen with patch-merged runtime state (src/gateway/server-channels.ts merges patches without clearing connected/lastEventAt) plus startup paths that can block before status publication (for example Slack waits on app.start() before calling publishSlackConnectedStatus in src/slack/monitor/provider.ts), leaving connected: true and an old lastEventAt forever and preventing health monitor recovery for a hung start.

Useful? React with 👍 / 👎.

* main: (45 commits) chore: update dependencies except carbon test(memory): make mcporter EINVAL retry test deterministic refactor(extensions): reuse shared helper primitives refactor(core): extract shared dedup helpers fix(ci): re-enable detect-secrets on main docs: reorder 2026.3.7 changelog highlights chore: bump version to 2026.3.7 fix(android): align run command with app id docs: add changelog entry for Android package rename (openclaw#38712) fix(android): rename app package to ai.openclaw.app fix(gateway): stop stale-socket restarts before first event (openclaw#38643) fix(gateway): skip stale-socket restarts for Telegram polling (openclaw#38405) fix(gateway): invalidate bootstrap cache on session rollover (openclaw#38535) docs: update changelog for reply media delivery (openclaw#38572) fix: contain final reply media normalization failures fix: contain block reply media failures fix: normalize reply media paths Fix owner-only auth and overlapping skill env regressions (openclaw#38548) fix(feishu): disable block streaming to prevent silent reply drops (openclaw#38422) fix: suppress ACP NO_REPLY fragments in console output (openclaw#38436) ...

…#38643) * fix(gateway): guard stale-socket restarts by event liveness * fix(gateway): centralize connect-time liveness tracking * fix(web): apply connected status patch atomically * fix(gateway): require active socket for stale checks * fix(gateway): ignore inherited stale event timestamps

* fix(gateway): guard stale-socket restarts by event liveness * fix(gateway): centralize connect-time liveness tracking * fix(web): apply connected status patch atomically * fix(gateway): require active socket for stale checks * fix(gateway): ignore inherited stale event timestamps

…#38643) * fix(gateway): guard stale-socket restarts by event liveness * fix(gateway): centralize connect-time liveness tracking * fix(web): apply connected status patch atomically * fix(gateway): require active socket for stale checks * fix(gateway): ignore inherited stale event timestamps

…#38643) * fix(gateway): guard stale-socket restarts by event liveness * fix(gateway): centralize connect-time liveness tracking * fix(web): apply connected status patch atomically * fix(gateway): require active socket for stale checks * fix(gateway): ignore inherited stale event timestamps (cherry picked from commit 8873e13)

openclaw-barnacle bot added channel: discord Channel integration: discord channel: slack Channel integration: slack channel: whatsapp-web Channel integration: whatsapp-web gateway Gateway runtime size: S maintainer Maintainer-authored PR labels Mar 7, 2026

Takhoffman added 2 commits March 7, 2026 00:38

fix(gateway): guard stale-socket restarts by event liveness

c677d07

fix(gateway): centralize connect-time liveness tracking

78f6f98

greptile-apps bot reviewed Mar 7, 2026

View reviewed changes

src/web/auto-reply/monitor.ts Outdated Show resolved Hide resolved

Takhoffman force-pushed the issue-38464-guided branch from 0d6110e to 78f6f98 Compare March 7, 2026 06:39

fix(web): apply connected status patch atomically

e768a30

chatgpt-codex-connector bot reviewed Mar 7, 2026

View reviewed changes

src/gateway/channel-health-policy.ts Outdated Show resolved Hide resolved

fix(gateway): require active socket for stale checks

d5a5dc6

chatgpt-codex-connector bot reviewed Mar 7, 2026

View reviewed changes

src/gateway/channel-health-policy.ts Show resolved Hide resolved

fix(gateway): ignore inherited stale event timestamps

2b47b76

Takhoffman merged commit 8873e13 into main Mar 7, 2026
29 of 30 checks passed

Takhoffman deleted the issue-38464-guided branch March 7, 2026 06:58

chatgpt-codex-connector bot reviewed Mar 7, 2026

View reviewed changes

alexyyyander mentioned this pull request Mar 7, 2026

fix/gateway token mismatch 38617 #38676

Closed

Takhoffman mentioned this pull request Mar 7, 2026

fix(gateway): clear stale Slack socket state after disconnect #39083

Merged

18 tasks

github-actions bot mentioned this pull request Mar 8, 2026

上游更新: v2026.3.7 — 19 P0 + 122 P1 待合并 jiulingyun/openclaw-cn#486

Open

drmmai mentioned this pull request Mar 8, 2026

Discord health-monitor false positive: stale-socket restart kills inbound before first event (2026.3.7) #40073

Open

alexey-pelykh mentioned this pull request Mar 10, 2026

Cherry-pick: gateway readiness probes, stale routes, HEIC media, Discord dedup remoteclaw/remoteclaw#845

Closed

fanyangCS mentioned this pull request Mar 31, 2026

fix(gateway): skip stale-socket detection for Discord channels #58404

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(gateway): stop stale-socket restarts before first event#38643

fix(gateway): stop stale-socket restarts before first event#38643
Takhoffman merged 5 commits intomainfrom
issue-38464-guided

Takhoffman commented Mar 7, 2026

Uh oh!

greptile-apps bot commented Mar 7, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if (lastStartAt != null && snapshot.lastEventAt < lastStartAt) {
		return { healthy: true, reason: "healthy" };

Uh oh!

Conversation

Takhoffman commented Mar 7, 2026

Summary

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

User-visible / Behavior Changes

Security Impact (required)

Repro + Verification

Environment

Steps

Expected

Actual

Evidence

Human Verification (required)

Compatibility / Migration

Failure Recovery (if this breaks)

Risks and Mitigations

Uh oh!

greptile-apps bot commented Mar 7, 2026

Greptile Summary

Confidence Score: 4/5

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant