Skip to content

fix(gateway): cap scheduleReconnect retries to prevent infinite loop (closes #45469)#46487

Closed
Br1an67 wants to merge 1 commit into
openclaw:mainfrom
Br1an67:fix/45469
Closed

fix(gateway): cap scheduleReconnect retries to prevent infinite loop (closes #45469)#46487
Br1an67 wants to merge 1 commit into
openclaw:mainfrom
Br1an67:fix/45469

Conversation

@Br1an67

@Br1an67 Br1an67 commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Summary

scheduleReconnect() retried indefinitely with exponential backoff capped at 30s but no maximum attempt count. When the gateway is permanently unreachable this accumulates WebSocket listeners and zombie processes (272 zombies observed after 266-day uptime per the issue).

Added a reconnectCount that increments on each retry. When it exceeds maxReconnectRetries (configurable, default 10), the client fires onConnectError and calls stop() to clean up. The counter resets to 0 on successful connection (hello-ok path).

Change Type

Bug fix

Scope

src/gateway/client.tsGatewayClient class + GatewayClientOptions type

Linked Issue

Closes #45469

Security Impact

None — no auth or permission changes.

Evidence

  • pnpm build passes
  • npx vitest run src/gateway/client.test.ts — 20 tests pass

Human Verification

  1. Set maxReconnectRetries: 3 in client options
  2. Point client at unreachable gateway URL
  3. Observe 3 reconnect attempts then clean stop with onConnectError
  4. Verify no zombie processes or listener leaks

Compatibility

Fully backward-compatible — default maxReconnectRetries is 10 (generous for typical networks). Existing callers that don't set the option get bounded retries instead of infinite.

Failure Recovery

When retries exhaust, onConnectError callback fires with descriptive error, then stop() flushes all pending requests and closes the WebSocket.

Risks

Callers relying on infinite retries may need to increase maxReconnectRetries or implement their own reconnect logic. The default of 10 with exponential backoff (1s→30s cap) covers ~3+ minutes of retrying.

This PR was AI-assisted (fully tested with pnpm build/check/test).

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: XS labels Mar 14, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d37906bba5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/client.ts Outdated
@@ -637,6 +640,13 @@ export class GatewayClient {
if (this.closed) {
return;
}
const maxRetries = this.opts.maxReconnectRetries ?? 10;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep default reconnect policy unbounded for long-lived clients

Defaulting maxReconnectRetries to 10 makes every existing GatewayClient caller stop reconnecting after roughly 3 minutes of outage, even though long-lived services currently instantiate the client without this option and rely on automatic recovery (for example runNodeHost logs "keep retrying" and then blocks forever). Once this branch calls stop(), closed is permanent for that instance, so the process will stay disconnected after transient gateway downtime until an external restart, which is a behavior regression introduced by this change.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes the unbounded reconnect loop in GatewayClient.scheduleReconnect() (issue #45469) by introducing a reconnectCount field that is incremented on every retry and checked against a configurable maxReconnectRetries option (defaulting to 10). When the limit is exceeded, onConnectError fires and stop() is called to flush pending requests and close the WebSocket cleanly. The counter correctly resets to 0 on successful hello-ok connection.

Key findings:

  • Logic bug — NaN/Infinity bypasses the cap: this.opts.maxReconnectRetries ?? 10 only substitutes 10 for null/undefined. If a caller passes NaN or Infinity, the value passes through and ++reconnectCount > NaN always evaluates to false, silently re-introducing the infinite loop this PR is meant to fix. Every other numeric option in the file is validated with typeof x === "number" && Number.isFinite(x) and the same pattern should be used here.
  • No tests for the new behaviour: The existing 20 tests all pass, but none of them exercise maxReconnectRetries. The retry-cap path (count exceeding limit, onConnectError firing, reset on hello-ok) is entirely untested, which creates a regression risk for future changes to scheduleReconnect.

Confidence Score: 3/5

  • Needs a small fix before merging — NaN/Infinity can bypass the retry cap and re-introduce the original infinite-loop bug.
  • The overall approach (counter + configurable limit + reset on success) is sound and the integration with stop() is correct. However, the missing numeric validation means a caller passing NaN silently breaks the fix, and there are no tests to catch regressions.
  • src/gateway/client.ts — specifically the maxRetries computation at line 643 and the absence of test coverage for the new code path.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-644

Comment:
**`NaN`/`Infinity` bypasses the retry cap entirely**

`this.opts.maxReconnectRetries ?? 10` uses the nullish coalescing operator, which only substitutes `10` for `null`/`undefined`. If a caller passes `maxReconnectRetries: NaN` or `maxReconnectRetries: Infinity`, the nullish check passes through the bad value and the cap is silently broken:

- `NaN ?? 10``NaN`; then `++reconnectCount > NaN``false` forever → infinite loop (re-introduces the original bug)
- `Infinity ?? 10``Infinity`; the count can never exceed `Infinity` → infinite loop

Every other numeric option in this file is validated with `typeof x === "number" && Number.isFinite(x)` (see `requestTimeoutMs`, `connectDelayMs`, `tickWatchMinIntervalMs`). The same pattern should be applied here:

```suggestion
    const rawMax = this.opts.maxReconnectRetries;
    const maxRetries =
      typeof rawMax === "number" && Number.isFinite(rawMax) && rawMax >= 0
        ? Math.floor(rawMax)
        : 10;
    if (++this.reconnectCount > maxRetries) {
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-648

Comment:
**No test coverage for the new retry-cap behaviour**

The PR description notes that the existing 20 tests pass, but none of them exercise `maxReconnectRetries`. There are no test cases that verify:

- The client stops after exactly `maxReconnectRetries` attempts
- `onConnectError` fires with the "limit reached" message
- `reconnectCount` resets to `0` on a successful `hello-ok`
- `maxReconnectRetries: 0` stops on the first reconnect attempt

Given that this feature exists specifically to prevent the zombie-process regression from #45469, having automated tests would prevent regressions from future refactors of `scheduleReconnect`.

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: d37906b

Comment thread src/gateway/client.ts Outdated
Comment on lines +643 to +644
const maxRetries = this.opts.maxReconnectRetries ?? 10;
if (++this.reconnectCount > maxRetries) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NaN/Infinity bypasses the retry cap entirely

this.opts.maxReconnectRetries ?? 10 uses the nullish coalescing operator, which only substitutes 10 for null/undefined. If a caller passes maxReconnectRetries: NaN or maxReconnectRetries: Infinity, the nullish check passes through the bad value and the cap is silently broken:

  • NaN ?? 10NaN; then ++reconnectCount > NaNfalse forever → infinite loop (re-introduces the original bug)
  • Infinity ?? 10Infinity; the count can never exceed Infinity → infinite loop

Every other numeric option in this file is validated with typeof x === "number" && Number.isFinite(x) (see requestTimeoutMs, connectDelayMs, tickWatchMinIntervalMs). The same pattern should be applied here:

Suggested change
const maxRetries = this.opts.maxReconnectRetries ?? 10;
if (++this.reconnectCount > maxRetries) {
const rawMax = this.opts.maxReconnectRetries;
const maxRetries =
typeof rawMax === "number" && Number.isFinite(rawMax) && rawMax >= 0
? Math.floor(rawMax)
: 10;
if (++this.reconnectCount > maxRetries) {
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-644

Comment:
**`NaN`/`Infinity` bypasses the retry cap entirely**

`this.opts.maxReconnectRetries ?? 10` uses the nullish coalescing operator, which only substitutes `10` for `null`/`undefined`. If a caller passes `maxReconnectRetries: NaN` or `maxReconnectRetries: Infinity`, the nullish check passes through the bad value and the cap is silently broken:

- `NaN ?? 10``NaN`; then `++reconnectCount > NaN``false` forever → infinite loop (re-introduces the original bug)
- `Infinity ?? 10``Infinity`; the count can never exceed `Infinity` → infinite loop

Every other numeric option in this file is validated with `typeof x === "number" && Number.isFinite(x)` (see `requestTimeoutMs`, `connectDelayMs`, `tickWatchMinIntervalMs`). The same pattern should be applied here:

```suggestion
    const rawMax = this.opts.maxReconnectRetries;
    const maxRetries =
      typeof rawMax === "number" && Number.isFinite(rawMax) && rawMax >= 0
        ? Math.floor(rawMax)
        : 10;
    if (++this.reconnectCount > maxRetries) {
```

How can I resolve this? If you propose a fix, please make it concise.

Comment thread src/gateway/client.ts Outdated
Comment on lines +643 to +648
const maxRetries = this.opts.maxReconnectRetries ?? 10;
if (++this.reconnectCount > maxRetries) {
const err = new Error(`gateway reconnect limit reached (${maxRetries} attempts)`);
this.opts.onConnectError?.(err);
this.stop();
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test coverage for the new retry-cap behaviour

The PR description notes that the existing 20 tests pass, but none of them exercise maxReconnectRetries. There are no test cases that verify:

  • The client stops after exactly maxReconnectRetries attempts
  • onConnectError fires with the "limit reached" message
  • reconnectCount resets to 0 on a successful hello-ok
  • maxReconnectRetries: 0 stops on the first reconnect attempt

Given that this feature exists specifically to prevent the zombie-process regression from #45469, having automated tests would prevent regressions from future refactors of scheduleReconnect.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/client.ts
Line: 643-648

Comment:
**No test coverage for the new retry-cap behaviour**

The PR description notes that the existing 20 tests pass, but none of them exercise `maxReconnectRetries`. There are no test cases that verify:

- The client stops after exactly `maxReconnectRetries` attempts
- `onConnectError` fires with the "limit reached" message
- `reconnectCount` resets to `0` on a successful `hello-ok`
- `maxReconnectRetries: 0` stops on the first reconnect attempt

Given that this feature exists specifically to prevent the zombie-process regression from #45469, having automated tests would prevent regressions from future refactors of `scheduleReconnect`.

How can I resolve this? If you propose a fix, please make it concise.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a8ad33384

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/client.ts Outdated
Comment on lines +643 to +647
const maxRetries = this.opts.maxReconnectRetries ?? 10;
if (++this.reconnectCount > maxRetries) {
const err = new Error(`gateway reconnect limit reached (${maxRetries} attempts)`);
this.opts.onConnectError?.(err);
this.stop();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore unbounded default reconnect policy

Using this.opts.maxReconnectRetries ?? 10 changes every existing GatewayClient caller that does not set this new option from infinite reconnect to a hard stop after 10 retries (~3 minutes with current backoff), and stop() then permanently sets closed. This is a behavior regression for long-lived processes that explicitly depend on automatic recovery (for example runNodeHost documents "keep retrying" in src/node-host/runner.ts:211 and then blocks forever), so transient gateway downtime now leaves those clients disconnected until an external restart.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c7d03f28e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/gateway/client.ts Outdated
@@ -637,6 +640,16 @@ export class GatewayClient {
if (this.closed) {
return;
}
const maxRetries = this.opts.maxReconnectRetries ?? 10;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep reconnect retries unbounded by default

Using this.opts.maxReconnectRetries ?? 10 changes existing GatewayClient callers that do not set the new option from infinite recovery to a hard stop after roughly a few minutes of outage, because exceeding the cap calls stop() and permanently flips closed. I checked long-lived callers in src/node-host/runner.ts (the process blocks forever after startup and comments that reconnect is handled by the client) and src/acp/server.ts (onClose logic assumes transient disconnects are followed by auto-reconnect), so transient gateway downtime now leaves these services disconnected until an external restart.

Useful? React with 👍 / 👎.

…loses openclaw#45469)

Add reconnectCount with configurable maxReconnectRetries (default 10).
When exhausted, fires onConnectError and stops the client cleanly.
Counter resets on successful connection (hello-ok).

Co-authored-by: Copilot <[email protected]>
@Br1an67

Br1an67 commented Mar 17, 2026

Copy link
Copy Markdown
Contributor Author

Closing to manage active PR count. Will reopen when slot is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug P2] scheduleReconnect() has no max retry limit — infinite reconnect loop

1 participant