fix(gateway): cap scheduleReconnect retries to prevent infinite loop (closes #45469)#46487
fix(gateway): cap scheduleReconnect retries to prevent infinite loop (closes #45469)#46487Br1an67 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d37906bba5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -637,6 +640,13 @@ export class GatewayClient { | |||
| if (this.closed) { | |||
| return; | |||
| } | |||
| const maxRetries = this.opts.maxReconnectRetries ?? 10; | |||
There was a problem hiding this comment.
Keep default reconnect policy unbounded for long-lived clients
Defaulting maxReconnectRetries to 10 makes every existing GatewayClient caller stop reconnecting after roughly 3 minutes of outage, even though long-lived services currently instantiate the client without this option and rely on automatic recovery (for example runNodeHost logs "keep retrying" and then blocks forever). Once this branch calls stop(), closed is permanent for that instance, so the process will stay disconnected after transient gateway downtime until an external restart, which is a behavior regression introduced by this change.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR fixes the unbounded reconnect loop in Key findings:
Confidence Score: 3/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-644
Comment:
**`NaN`/`Infinity` bypasses the retry cap entirely**
`this.opts.maxReconnectRetries ?? 10` uses the nullish coalescing operator, which only substitutes `10` for `null`/`undefined`. If a caller passes `maxReconnectRetries: NaN` or `maxReconnectRetries: Infinity`, the nullish check passes through the bad value and the cap is silently broken:
- `NaN ?? 10` → `NaN`; then `++reconnectCount > NaN` → `false` forever → infinite loop (re-introduces the original bug)
- `Infinity ?? 10` → `Infinity`; the count can never exceed `Infinity` → infinite loop
Every other numeric option in this file is validated with `typeof x === "number" && Number.isFinite(x)` (see `requestTimeoutMs`, `connectDelayMs`, `tickWatchMinIntervalMs`). The same pattern should be applied here:
```suggestion
const rawMax = this.opts.maxReconnectRetries;
const maxRetries =
typeof rawMax === "number" && Number.isFinite(rawMax) && rawMax >= 0
? Math.floor(rawMax)
: 10;
if (++this.reconnectCount > maxRetries) {
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-648
Comment:
**No test coverage for the new retry-cap behaviour**
The PR description notes that the existing 20 tests pass, but none of them exercise `maxReconnectRetries`. There are no test cases that verify:
- The client stops after exactly `maxReconnectRetries` attempts
- `onConnectError` fires with the "limit reached" message
- `reconnectCount` resets to `0` on a successful `hello-ok`
- `maxReconnectRetries: 0` stops on the first reconnect attempt
Given that this feature exists specifically to prevent the zombie-process regression from #45469, having automated tests would prevent regressions from future refactors of `scheduleReconnect`.
How can I resolve this? If you propose a fix, please make it concise.Last reviewed commit: d37906b |
| const maxRetries = this.opts.maxReconnectRetries ?? 10; | ||
| if (++this.reconnectCount > maxRetries) { |
There was a problem hiding this comment.
NaN/Infinity bypasses the retry cap entirely
this.opts.maxReconnectRetries ?? 10 uses the nullish coalescing operator, which only substitutes 10 for null/undefined. If a caller passes maxReconnectRetries: NaN or maxReconnectRetries: Infinity, the nullish check passes through the bad value and the cap is silently broken:
NaN ?? 10→NaN; then++reconnectCount > NaN→falseforever → infinite loop (re-introduces the original bug)Infinity ?? 10→Infinity; the count can never exceedInfinity→ infinite loop
Every other numeric option in this file is validated with typeof x === "number" && Number.isFinite(x) (see requestTimeoutMs, connectDelayMs, tickWatchMinIntervalMs). The same pattern should be applied here:
| const maxRetries = this.opts.maxReconnectRetries ?? 10; | |
| if (++this.reconnectCount > maxRetries) { | |
| const rawMax = this.opts.maxReconnectRetries; | |
| const maxRetries = | |
| typeof rawMax === "number" && Number.isFinite(rawMax) && rawMax >= 0 | |
| ? Math.floor(rawMax) | |
| : 10; | |
| if (++this.reconnectCount > maxRetries) { |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-644
Comment:
**`NaN`/`Infinity` bypasses the retry cap entirely**
`this.opts.maxReconnectRetries ?? 10` uses the nullish coalescing operator, which only substitutes `10` for `null`/`undefined`. If a caller passes `maxReconnectRetries: NaN` or `maxReconnectRetries: Infinity`, the nullish check passes through the bad value and the cap is silently broken:
- `NaN ?? 10` → `NaN`; then `++reconnectCount > NaN` → `false` forever → infinite loop (re-introduces the original bug)
- `Infinity ?? 10` → `Infinity`; the count can never exceed `Infinity` → infinite loop
Every other numeric option in this file is validated with `typeof x === "number" && Number.isFinite(x)` (see `requestTimeoutMs`, `connectDelayMs`, `tickWatchMinIntervalMs`). The same pattern should be applied here:
```suggestion
const rawMax = this.opts.maxReconnectRetries;
const maxRetries =
typeof rawMax === "number" && Number.isFinite(rawMax) && rawMax >= 0
? Math.floor(rawMax)
: 10;
if (++this.reconnectCount > maxRetries) {
```
How can I resolve this? If you propose a fix, please make it concise.| const maxRetries = this.opts.maxReconnectRetries ?? 10; | ||
| if (++this.reconnectCount > maxRetries) { | ||
| const err = new Error(`gateway reconnect limit reached (${maxRetries} attempts)`); | ||
| this.opts.onConnectError?.(err); | ||
| this.stop(); | ||
| return; |
There was a problem hiding this comment.
No test coverage for the new retry-cap behaviour
The PR description notes that the existing 20 tests pass, but none of them exercise maxReconnectRetries. There are no test cases that verify:
- The client stops after exactly
maxReconnectRetriesattempts onConnectErrorfires with the "limit reached" messagereconnectCountresets to0on a successfulhello-okmaxReconnectRetries: 0stops on the first reconnect attempt
Given that this feature exists specifically to prevent the zombie-process regression from #45469, having automated tests would prevent regressions from future refactors of scheduleReconnect.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-648
Comment:
**No test coverage for the new retry-cap behaviour**
The PR description notes that the existing 20 tests pass, but none of them exercise `maxReconnectRetries`. There are no test cases that verify:
- The client stops after exactly `maxReconnectRetries` attempts
- `onConnectError` fires with the "limit reached" message
- `reconnectCount` resets to `0` on a successful `hello-ok`
- `maxReconnectRetries: 0` stops on the first reconnect attempt
Given that this feature exists specifically to prevent the zombie-process regression from #45469, having automated tests would prevent regressions from future refactors of `scheduleReconnect`.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6a8ad33384
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const maxRetries = this.opts.maxReconnectRetries ?? 10; | ||
| if (++this.reconnectCount > maxRetries) { | ||
| const err = new Error(`gateway reconnect limit reached (${maxRetries} attempts)`); | ||
| this.opts.onConnectError?.(err); | ||
| this.stop(); |
There was a problem hiding this comment.
Restore unbounded default reconnect policy
Using this.opts.maxReconnectRetries ?? 10 changes every existing GatewayClient caller that does not set this new option from infinite reconnect to a hard stop after 10 retries (~3 minutes with current backoff), and stop() then permanently sets closed. This is a behavior regression for long-lived processes that explicitly depend on automatic recovery (for example runNodeHost documents "keep retrying" in src/node-host/runner.ts:211 and then blocks forever), so transient gateway downtime now leaves those clients disconnected until an external restart.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c7d03f28e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -637,6 +640,16 @@ export class GatewayClient { | |||
| if (this.closed) { | |||
| return; | |||
| } | |||
| const maxRetries = this.opts.maxReconnectRetries ?? 10; | |||
There was a problem hiding this comment.
Keep reconnect retries unbounded by default
Using this.opts.maxReconnectRetries ?? 10 changes existing GatewayClient callers that do not set the new option from infinite recovery to a hard stop after roughly a few minutes of outage, because exceeding the cap calls stop() and permanently flips closed. I checked long-lived callers in src/node-host/runner.ts (the process blocks forever after startup and comments that reconnect is handled by the client) and src/acp/server.ts (onClose logic assumes transient disconnects are followed by auto-reconnect), so transient gateway downtime now leaves these services disconnected until an external restart.
Useful? React with 👍 / 👎.
…loses openclaw#45469) Add reconnectCount with configurable maxReconnectRetries (default 10). When exhausted, fires onConnectError and stops the client cleanly. Counter resets on successful connection (hello-ok). Co-authored-by: Copilot <[email protected]>
|
Closing to manage active PR count. Will reopen when slot is available. |
Summary
scheduleReconnect()retried indefinitely with exponential backoff capped at 30s but no maximum attempt count. When the gateway is permanently unreachable this accumulates WebSocket listeners and zombie processes (272 zombies observed after 266-day uptime per the issue).Added a
reconnectCountthat increments on each retry. When it exceedsmaxReconnectRetries(configurable, default 10), the client firesonConnectErrorand callsstop()to clean up. The counter resets to 0 on successful connection (hello-ok path).Change Type
Bug fix
Scope
src/gateway/client.ts—GatewayClientclass +GatewayClientOptionstypeLinked Issue
Closes #45469
Security Impact
None — no auth or permission changes.
Evidence
pnpm buildpassesnpx vitest run src/gateway/client.test.ts— 20 tests passHuman Verification
maxReconnectRetries: 3in client optionsonConnectErrorCompatibility
Fully backward-compatible — default
maxReconnectRetriesis 10 (generous for typical networks). Existing callers that don't set the option get bounded retries instead of infinite.Failure Recovery
When retries exhaust,
onConnectErrorcallback fires with descriptive error, thenstop()flushes all pending requests and closes the WebSocket.Risks
Callers relying on infinite retries may need to increase
maxReconnectRetriesor implement their own reconnect logic. The default of 10 with exponential backoff (1s→30s cap) covers ~3+ minutes of retrying.This PR was AI-assisted (fully tested with pnpm build/check/test).