Skip to content

fix(health-monitor): add reconnect grace for gateway reconnects#73989

Closed
openclaw-clownfish[bot] wants to merge 2 commits into
mainfrom
clownfish/ghcrawl-157023-autonomous-smoke
Closed

fix(health-monitor): add reconnect grace for gateway reconnects#73989
openclaw-clownfish[bot] wants to merge 2 commits into
mainfrom
clownfish/ghcrawl-157023-autonomous-smoke

Conversation

@openclaw-clownfish

@openclaw-clownfish openclaw-clownfish Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Repairs the existing #45712 branch rather than replacing it.

Summary:

  • add shared reconnectGraceMs/lastDisconnectAt handling so running-but-recently-disconnected channels are not restarted before provider reconnect grace expires
  • consume typed lastDisconnect.at from runtime snapshots without unsafe casts
  • update health monitor and readiness call sites plus focused regression coverage for recent disconnects, expired disconnects, stopped channels, stale sockets, active runs, and restart caps

Validation:

  • pnpm check:changed

Credit: based on #45712 by @cass-clearly / Cass.

ProjectClownfish replacement details:

@openclaw-clownfish openclaw-clownfish Bot mentioned this pull request Apr 29, 2026
Closed
@openclaw-clownfish openclaw-clownfish Bot added the clawsweeper Tracked by ClawSweeper automation label Apr 29, 2026
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: M r: too-many-prs Auto-close: author has more than twenty active PRs. labels Apr 29, 2026
@openclaw-barnacle

openclaw-barnacle Bot commented Apr 29, 2026

Copy link
Copy Markdown

Closing this PR because the author has more than 10 active PRs in this repo. Please reduce the active PR queue and reopen or resubmit once it is back under the limit. You can close your own PRs to get back under the limit.

@greptile-apps

greptile-apps Bot commented Apr 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a shared reconnect grace window to the channel health monitor and readiness checker so that running-but-recently-disconnected channels (e.g., mid-provider reconnect loop) are not restarted prematurely. It extracts lastDisconnect.at safely via createChannelHealthSnapshot / resolveChannelLastDisconnectAt, threads reconnectGraceMs through the policy and timing types, and adds focused regression tests for all key cases (recent disconnect, expired grace, pre-lifecycle timestamps, string disconnect errors).

Confidence Score: 5/5

Safe to merge; the implementation is correct, well-bounded, and covered by focused regression tests.

No logic bugs found. The lifecycle guard (lastDisconnectAt >= lastStartAt) correctly ignores stale disconnect timestamps from previous runs, disconnectAge is time-bounded so grace cannot last indefinitely, and the Math.max(0, ...) clamp handles zero/negative configurations cleanly. The two open style-level notes (missing @deprecated on the flat reconnectGraceMs field, omitted grace value in the startup log line) were already flagged in earlier review rounds and do not affect correctness.

No files require special attention.

Reviews (2): Last reviewed commit: "fix(health-monitor): add reconnect grace..." | Re-trigger Greptile

greptile-apps[bot]
greptile-apps Bot reviewed Apr 29, 2026
View reviewed changes
Comment thread src/gateway/channel-health-monitor.ts
@vincentkoc vincentkoc reopened this Apr 29, 2026
@openclaw-barnacle openclaw-barnacle Bot added r: too-many-prs Auto-close: author has more than twenty active PRs. and removed r: too-many-prs Auto-close: author has more than twenty active PRs. labels Apr 29, 2026
@clawsweeper

clawsweeper Bot commented Apr 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Keep open: current main still marks a running managed channel with connected === false as disconnected after startup grace, so the reconnect-grace fix remains useful, but this exact PR is conflicting, misses a current health-policy caller, edits release-owned changelog content, and lacks real behavior proof.

Canonical path: Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review.

So I’m closing this here because the remaining work is already tracked in the canonical issue.

Review details

Best possible solution:

Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection: current main marks connected === false running channels as disconnected after startup grace, which is the restart path the PR changes. I did not run tests because this review is read-only.

Is this the best way to solve the issue?

No, not as submitted: reconnect grace is a plausible fix, but this branch is not the best merge shape until it is refreshed, all current callers are updated, and the default restart-delay tradeoff is owner-approved.

Security review:

Security review cleared: The diff touches gateway TypeScript, tests, and changelog text only; I found no dependency, workflow, credential, install, publish, or supply-chain change.

AGENTS.md: found and applied where relevant.

What I checked:

  • stale F-rated PR: PR was opened 2026-04-29T02:34:33Z, is older than 60 days, and the latest review rated it F.
  • proof blocker: real behavior proof is mock_only and proof tier is F, so this branch is not merge-ready without contributor follow-up.
  • no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

  • Peter Lee: Current shallow blame attributes the central gateway health-policy and monitor code in the checked-out main snapshot to the grafted current-main commit. (role: recent area contributor; confidence: low; commits: 85f7834852d3; files: src/gateway/channel-health-policy.ts, src/gateway/channel-health-monitor.ts, src/gateway/server/readiness.ts)
  • vincentkoc: Authored the merged Discord health-monitor restart escalation PR and also made the review-fix commit on this branch, making them a strong routing candidate for the gateway restart tradeoff. (role: recent adjacent contributor; confidence: medium; commits: 24b797e6672b, b4cdd55f6273, 60a6452d4b8b; files: extensions/discord/src/monitor/provider.lifecycle.test.ts, src/gateway/channel-health-monitor.test.ts, src/gateway/server.reload.test.ts)
  • cass-clearly: Authored the closed source reconnect-grace PR that this branch explicitly repairs and linked production restart/OOM evidence in the related issue discussion. (role: source behavior contributor; confidence: medium; commits: 14dee8d60ad4, c9052ba6a2a2, fcb1d2ea1234; files: src/gateway/channel-health-policy.ts, src/gateway/channel-health-monitor.ts, src/gateway/channel-health-monitor.test.ts)
  • lml2468: Authored an earlier closed reconnect-grace PR for the same health-monitor false restart pattern, useful context for maintainers comparing prior attempts. (role: earlier adjacent contributor; confidence: low; commits: 92f12418e415, c582322d361d; files: src/gateway/channel-health-policy.ts, src/gateway/channel-health-monitor.ts, src/gateway/channel-health-monitor.test.ts)

Codex review notes: model internal, reasoning high; reviewed against eb417fa206e6.

@vincentkoc

vincentkoc commented Apr 29, 2026

Copy link
Copy Markdown
Member

/clownfish automerge

@openclaw-clownfish openclaw-clownfish Bot added the clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge label Apr 29, 2026
@openclaw-clownfish

openclaw-clownfish Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

Clownfish is on the reef for this PR. 🐠

I tagged clownfish:automerge and sent ClawSweeper over this exact head. If the sweep finds rough coral, failing checks, or needs-human, I will take another bounded repair lap and ask for a fresh review.

A maintainer can call /clownfish stop any time and I will drift this back to human review.

@clawsweeper clawsweeper Bot added the clawsweeper:human-review Needs maintainer review before ClawSweeper can continue label Apr 30, 2026
@clawsweeper

clawsweeper Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🦞🦞
ClawSweeper is pausing this repair loop for human review.

Source: clawsweeper[bot]
Reason: structured ClawSweeper verdict: needs-human (sha=b62721f04f5213b41bf749dbb3a539ab169fe24a)

I added clawsweeper:human-review and left the final call with a maintainer.

@clawsweeper clawsweeper Bot removed the clawsweeper:human-review Needs maintainer review before ClawSweeper can continue label May 1, 2026
@clawsweeper

clawsweeper Bot commented May 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🦞
ClawSweeper applied a repair to this PR branch.

Repair: rebased this branch deterministically; Codex fix/edit was not needed.
Validation: GitHub checks and exact-head ClawSweeper review gate this rebase
Updated head: 60a6452d4b8b
Run: https://github.com/openclaw/clawsweeper/actions/runs/25246653411

Current state: exact-head review queued immediately; GitHub checks and the review verdict gate final merge.

Automerge progress:

  • 2026-05-02 05:57:16 UTC review requested repair [`b62721f04f52`](https://github.com/openclaw/openclaw/commit/b62721f04f5213b41bf749dbb3a539ab169fe24a) (structured ClawSweeper marker: fix-required (finding=review-feedback sha=b62721...)
  • 2026-05-01 15:40:59 UTC repair queued `b62721f04f52` (autonomous) Run: https://github.com/openclaw/clawsweeper/actions/runs/25220842888
  • 2026-05-01 15:47:48 UTC repair completed (no branch change) in 4m 2s Run: https://github.com/openclaw/clawsweeper/actions/runs/25220842888 validation command failed (pnpm check:changed): [check:changed] lanes=core, coreTests, docs [check:changed] src/gateway/channel-health-monitor.test.ts: core te...
  • 2026-05-02 06:06:56 UTC repair queued [`b62721f04f52`](https://github.com/openclaw/openclaw/commit/b62721f04f5213b41bf749dbb3a539ab169fe24a) (autonomous) Run: https://github.com/openclaw/clawsweeper/actions/runs/25245445297
  • 2026-05-02 07:04:32 UTC repair completed [`d9659eb9c8c2`](https://github.com/openclaw/openclaw/commit/d9659eb9c8c2999a75435f4e8d9cf642e1f33b75) (deterministic rebase) in 1m 21s Run: https://github.com/openclaw/clawsweeper/actions/runs/25246368121 mechanically resolved deterministic rebase
  • 2026-05-02 07:04:31 UTC review queued [`d9659eb9c8c2`](https://github.com/openclaw/openclaw/commit/d9659eb9c8c2999a75435f4e8d9cf642e1f33b75) (after repair)
  • 2026-05-02 07:08:29 UTC review requested repair [`d9659eb9c8c2`](https://github.com/openclaw/openclaw/commit/d9659eb9c8c2999a75435f4e8d9cf642e1f33b75) (structured ClawSweeper marker: fix-required (finding=review-feedback sha=d9659e...)
  • 2026-05-02 07:08:38 UTC repair queued [`d9659eb9c8c2`](https://github.com/openclaw/openclaw/commit/d9659eb9c8c2999a75435f4e8d9cf642e1f33b75) (autonomous) Run: https://github.com/openclaw/clawsweeper/actions/runs/25246519561
  • 2026-05-02 07:12:44 UTC repair completed [`8007e25a5088`](https://github.com/openclaw/openclaw/commit/8007e25a508866297be131eaecb71bdf13485285) (deterministic rebase) in 1m 28s Run: https://github.com/openclaw/clawsweeper/actions/runs/25246519561 clean deterministic rebase
  • 2026-05-02 07:12:43 UTC review queued [`8007e25a5088`](https://github.com/openclaw/openclaw/commit/8007e25a508866297be131eaecb71bdf13485285) (after repair)
  • 2026-05-02 07:16:03 UTC review requested repair [`8007e25a5088`](https://github.com/openclaw/openclaw/commit/8007e25a508866297be131eaecb71bdf13485285) (structured ClawSweeper marker: fix-required (finding=review-feedback sha=8007e2...)
  • 2026-05-02 07:16:13 UTC repair queued [`8007e25a5088`](https://github.com/openclaw/openclaw/commit/8007e25a508866297be131eaecb71bdf13485285) (autonomous) Run: https://github.com/openclaw/clawsweeper/actions/runs/25246653411
  • 2026-05-02 07:20:34 UTC repair completed [`60a6452d4b8b`](https://github.com/openclaw/openclaw/commit/60a6452d4b8ba35891f1728053d13c8ab6eaa6c1) (deterministic rebase) in 1m 34s Run: https://github.com/openclaw/clawsweeper/actions/runs/25246653411 clean deterministic rebase
  • 2026-05-02 07:20:33 UTC review queued [`60a6452d4b8b`](https://github.com/openclaw/openclaw/commit/60a6452d4b8ba35891f1728053d13c8ab6eaa6c1) (after repair)

@clawsweeper
clawsweeper Bot force-pushed the clownfish/ghcrawl-157023-autonomous-smoke branch 2 times, most recently from d9659eb to 8007e25 Compare May 2, 2026 07:12
openclaw-clownfish Bot and others added 2 commits May 2, 2026 07:20
@openclaw-clownfish @clawsweeper
fix(health-monitor): add reconnect grace for gateway reconnects
1737997
@vincentkoc @greptile-apps @clawsweeper
Update src/gateway/channel-health-monitor.ts
60a6452 
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
@clawsweeper
clawsweeper Bot force-pushed the clownfish/ghcrawl-157023-autonomous-smoke branch from 8007e25 to 60a6452 Compare May 2, 2026 07:20
@clawsweeper clawsweeper Bot added the clawsweeper:human-review Needs maintainer review before ClawSweeper can continue label May 2, 2026
@clawsweeper

clawsweeper Bot commented May 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞✅
ClawSweeper is pausing this repair loop for human review.

Source: clawsweeper[bot]
Reason: - Review did not complete, so no work-lane recommendation was made. (sha=60a6452d4b8ba35891f1728053d13c8ab6eaa6c1)

Why human review is needed:
ClawSweeper found a blocker that should be resolved or accepted by a maintainer before the repair or automerge loop continues.

What the maintainer can do as a next step:
If the maintainer accepts the current risk and wants ClawSweeper to continue merge gates, comment @clawsweeper approve. If more work is needed, resolve the blocker first, then comment @clawsweeper automerge to re-review and continue. If automation should stay paused, leave clawsweeper:human-review in place or comment @clawsweeper stop.

I added clawsweeper:human-review and left the final call with a maintainer.

@openclaw-barnacle

openclaw-barnacle Bot commented May 30, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label May 30, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. status: 🚀 automerge armed This PR is in ClawSweeper's automerge lane. labels May 30, 2026
@barnacle-openclaw barnacle-openclaw Bot removed the stale Marked as stale due to inactivity label May 30, 2026
@openclaw-barnacle

openclaw-barnacle Bot commented Jun 14, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 14, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 15, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jun 15, 2026
@clawsweeper clawsweeper Bot closed this Jul 1, 2026
@clawsweeper

clawsweeper Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🦞✅
ClawSweeper autoclose is complete.

Reason: structured ClawSweeper close marker: close-required (sha=60a6452d4b8ba35891f1728053d13c8ab6eaa6c1)

Closed:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments

Assignees

No one assigned

Labels

clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge clawsweeper:human-review Needs maintainer review before ClawSweeper can continue clawsweeper Tracked by ClawSweeper automation gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 🚀 automerge armed This PR is in ClawSweeper's automerge lane.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vincentkoc