Skip to content

fix(gateway): add max reconnect limit with opt-in maxReconnectAttempts#77961

Open
stellamariesays wants to merge 5 commits into
openclaw:mainfrom
stellamariesays:fix/max-reconnect-limit-v2
Open

fix(gateway): add max reconnect limit with opt-in maxReconnectAttempts#77961
stellamariesays wants to merge 5 commits into
openclaw:mainfrom
stellamariesays:fix/max-reconnect-limit-v2

Conversation

@stellamariesays

@stellamariesays stellamariesays commented May 5, 2026

Copy link
Copy Markdown

Summary

Adds an opt-in reconnect attempt limit for GatewayClient to prevent infinite reconnect loops in short-lived clients (gateway CLI, MCP bridge, etc.).

Resolves #45469

Changes

  • Add maxReconnectAttempts?: number to GatewayClientOptions
  • When set, scheduleReconnect() gives up after N attempts and fires onReconnectPaused with detailCode: null
  • When unset (default), behavior is unchanged — unlimited retries, preserving node-host recovery semantics
  • Reset reconnectAttempts counter on successful hello-ok handshake
  • Two tests: cap exhaustion + counter reset after recovery

Why opt-in

Node-host relies on GatewayClient retrying indefinitely during gateway outages. A global cap would zombie node-host processes. Making it opt-in means node-host is unchanged while CLI/bridge callers can opt in to bounded retries.

Test plan

  • Stops after N reconnect attempts and fires onReconnectPaused
  • Counter resets on successful connect
  • CI green (lint, type-check, security, boundaries)

Real behavior proof

Behavior or issue addressed: GatewayClient reconnects infinitely with no upper bound on attempts. After applying this patch, callers can opt in to a bounded reconnect limit via maxReconnectAttempts while node-host retains unbounded retries.

Real environment tested: ARM64 Linux (aarch64, Ubuntu), Node.js v22.22.2, running vitest against patched src/gateway/client.ts from fix/max-reconnect-limit-v2 branch.

Exact steps or command run after the patch: Ran targeted reconnect tests: npx vitest run src/gateway/client.test.ts -t "reconnect" --reporter=verbose

  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):

Observed result after fix: With maxReconnectAttempts: 30 set, the client fires onReconnectPaused({ code: -1, reason: "max reconnect attempts reached (30)", detailCode: null }) after 30 failures and stops reconnecting. After a successful hello-ok handshake the counter resets, allowing another full cycle. Without the option set, reconnects continue unbounded — no behavior change.

Not tested: Full end-to-end reconnect against a live gateway server (no local gateway instance available on this ARM host).

…45469)

scheduleReconnect() retried indefinitely with exponential backoff capped at
30s but no maximum retry count. If the gateway is unreachable, the client
loops forever accumulating listeners and zombie processes.

Changes:
- Add MAX_RECONNECT_ATTEMPTS (30) as a static class constant
- scheduleReconnect() now gives up after 30 failed attempts, fires
  onReconnectPaused with a descriptive reason, and sets closed = true
- reconnectAttempts resets to 0 on successful hello handshake
- reconnectAttempts resets to 0 on explicit stop()
- Two new tests: max-reconnect-gives-up and counter-resets-on-success

Closes openclaw#45469
TypeScript strict null checks require string | null, not undefined.
Address ClawSweeper review feedback:
- P2: Make reconnect cap opt-in via maxReconnectAttempts option.
  Undefined = unlimited, preserving node-host unbounded retry behavior.
- P3: Fix reset test to send proper type:"res" with payload.type:"hello-ok"
  instead of type:"connect" so handleMessage actually processes it.
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. gateway Gateway runtime size: S and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 5, 2026
@clawsweeper

clawsweeper Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Keep open: this is a useful linked fix candidate, but current main moved the active reconnect scheduler into the gateway-client package, while this branch is conflicting and still patches the stale facade-era implementation with no production caller opt-in.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #45469
Summary: This PR is the open candidate fix for the canonical GatewayClient reconnect retry-ceiling issue; earlier PR attempts are closed and unmerged, and a merged request-timeout PR covered adjacent reliability only.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review.

So I’m closing this here because the remaining work is already tracked in the canonical issue.

Review details

Best possible solution:

Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current package-client close handling reaches scheduleReconnect(), which schedules another start() without a max-attempt counter. I did not run a live long-duration reproduction of the reported process buildup.

Is this the best way to solve the issue?

No as submitted: the opt-in cap is a plausible direction, but this branch patches the stale facade-era implementation and does not wire any caller that should stop retrying. The maintainable fix belongs in the package client with maintainer-approved caller policy.

Security review:

Security review cleared: The diff changes gateway TypeScript and tests only; it adds no dependency, workflow, lockfile, package-script, credential, permission, or artifact-download surface.

AGENTS.md: found and applied where relevant.

What I checked:

  • stale F-rated PR: PR was opened 2026-05-05T17:17:05Z, is older than 60 days, and the latest review rated it F.
  • proof blocker: real behavior proof is mock_only and proof tier is F, so this branch is not merge-ready without contributor follow-up.
  • no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

  • steipete: Commit history ties the original gateway WebSocket control plane and the later packages/gateway-client extraction to this area, with several recent package-client fixes also authored there. (role: introduced behavior and package extraction owner; confidence: high; commits: b2e7fb01a926, b1117d98622f, 984951f55cd5; files: src/gateway/client.ts, packages/gateway-client/src/client.ts)
  • vincentkoc: Recent main history shows multiple June 2026 fixes on the active gateway-client package, including lifecycle callback handling and watchdog timeout behavior. (role: recent area contributor; confidence: high; commits: 09427aa76091, 47cad606f4db, 9d04064e736a; files: packages/gateway-client/src/client.ts, src/gateway/client.ts, src/gateway/client.test.ts)
  • joshavant: Recent gateway work moved caller identity through the gateway path and touched packages/gateway-client/src/client.ts, making this a useful routing candidate for caller-policy review. (role: recent adjacent gateway contributor; confidence: medium; commits: cee2aca40912; files: packages/gateway-client/src/client.ts, src/gateway/client.ts)
  • Takhoffman: The merged unanswered-request timeout fix handled a neighboring GatewayClient reliability issue and explicitly left reconnect retry policy as follow-up work. (role: adjacent reliability contributor; confidence: medium; commits: 5fc43ff0ecd8; files: src/gateway/client.ts, src/gateway/client.watchdog.test.ts, src/gateway/call.ts)

Codex review notes: model internal, reasoning high; reviewed against 010b61746379.

@stellamariesays

Copy link
Copy Markdown
Author

Hi maintainers — the Real behavior proof check is failing because our evidence is vitest output, which the checker classifies as "mock only." This PR is a targeted fix to GatewayClient that adds an opt-in maxReconnectAttempts option (default: unlimited, no behavior change). The change is fully covered by unit tests and reviewed by ClawSweeper (both P2 and P3 feedback addressed). We don't have a live gateway instance on this ARM host to produce runtime logs. Could someone apply the proof: override label? Happy to provide additional evidence if needed. Thanks!

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added stale Marked as stale due to inactivity triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 31, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels May 31, 2026
@barnacle-openclaw barnacle-openclaw Bot removed the stale Marked as stale due to inactivity label May 31, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 30, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 20, 2026
@clawsweeper

clawsweeper Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@stellamariesays thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward.

Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive.

Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it.

@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug P2] scheduleReconnect() has no max retry limit — infinite reconnect loop

2 participants