Skip to content

fix(agents): circuit-break consecutive LLM idle timeouts to cap retry-loop cost (#76293)#76345

Merged
steipete merged 1 commit into
openclaw:mainfrom
adhirajhangal:fix/76293-llm-idle-timeout-circuit-breaker
May 3, 2026
Merged

fix(agents): circuit-break consecutive LLM idle timeouts to cap retry-loop cost (#76293)#76345
steipete merged 1 commit into
openclaw:mainfrom
adhirajhangal:fix/76293-llm-idle-timeout-circuit-breaker

Conversation

@adhirajhangal

Copy link
Copy Markdown
Contributor

What

Adds a circuit breaker to streamWithIdleTimeout that caps how many consecutive idle timeouts can fire on the same wrapper instance before subsequent baseFn invocations are refused. Default cap is 5. Counter resets on any successful chunk, so slow-but-responsive streams keep working.

Why

Per #76293, today a single idle timeout propagates up, the upstream retry layer (pi-agent-core) re-invokes streamFn, the retry also times out (or fails fast) against the same wedged provider, and the cycle repeats unbounded. The reporter saw 761-1384 paid Anthropic Sonnet 4.6 calls in 60 seconds across two real incidents, costing $20-30 each before any external monitor caught it. Auto-recharge on the provider account masks the spike at the time of incident, so users only find out at billing.

The fix lives at the OpenClaw streamFn wrapper level so it applies to every caller without coordinating with the upstream retry semantics. Worst case under the new default: 5 paid model calls before lockout. At Anthropic Sonnet 4.6 list pricing for the heartbeat-context size in the report, that is roughly $0.10-0.30 per incident vs $20-30 today.

How it behaves

  • streamFn called, first idle timeout fires, counter = 1, normal abort path runs as before
  • Retry, second timeout, counter = 2, ... counter = 5
  • 6th retry: wrapper synchronously throws LLM idle timeout circuit breaker tripped: ... without calling baseFn, and routes the same error through onIdleTimeout so the existing idleTimeoutTrigger in attempt.ts aborts the run via the same path it already takes for a regular timeout
  • Any subsequent retry in the same wrapper instance: same synchronous refusal until the wrapper instance is replaced (next attempt creates a fresh wrapper)
  • A chunk on any invocation: counter reset to 0, breaker re-armed

What did not change

  • src/agents/pi-embedded-runner/run/attempt.ts call site is unchanged. The default of 5 takes effect everywhere via the function default
  • pi-agent-core retry semantics: untouched. Even if the upstream layer keeps retrying, our wrapper just keeps refusing without any paid call. Each refused call is a synchronous throw, costing nothing
  • onIdleTimeout contract: still fires once per real timeout AND once per circuit-breaker trip, so the existing abortRun path runs unchanged. The error message lets observability tools distinguish the two cases (circuit breaker tripped vs the existing no response from model)

Tests

Five new vitest cases in src/agents/pi-embedded-runner/run/llm-idle-timeout.test.ts under describe("circuit breaker - issue #76293"):

  1. Default cap (DEFAULT_MAX_CONSECUTIVE_LLM_IDLE_TIMEOUTS = 5) holds under a 50-attempt simulated retry storm. baseFn.mock.calls.length lands at exactly 5
  2. Explicit smaller cap (3) holds under 20 simulated retries
  3. maxConsecutiveTimeouts: 0 disables the breaker (escape hatch for power users who want the old behavior). 12 simulated retries produce 12 baseFn calls
  4. The trip is reported via onIdleTimeout with a distinct circuit breaker tripped message so the surrounding abort path keeps working and observability can split the two cases
  5. A single chunk between timeouts resets the counter so legitimate slow streams are not falsely tripped (5 attempts: timeout / timeout / chunk / timeout / timeout, all 5 reach baseFn because the chunk resets the counter)

What I haven't verified

I'm working from a sandbox without pnpm installed, so I haven't been able to run pnpm test or pnpm test:contracts:agents locally. The unit tests are written and reviewed by hand against the wrapper logic. Relying on PR CI to confirm. If anything fails I'll iterate.

AI / vibe-coding disclosure

  • AI-assisted with Claude Opus 4.7 (Anthropic). Marked per CONTRIBUTING.md
  • Lightly tested: 5 unit tests added and hand-reviewed against the wrapper, NOT validated against a live model run or full pnpm test suite
  • I do not have codex installed in the sandbox so could not run codex review --base origin/main locally. Will address Codex review feedback on the PR
  • Will resolve clawsweeper / Codex review conversations as they come in

Closes #76293

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M labels May 3, 2026
@clawsweeper

clawsweeper Bot commented May 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The branch adds an outer embedded-run idle-timeout breaker, a pure helper with unit tests, and a changelog entry for the linked runaway cost bug.

Reproducibility: yes. By source inspection, current main recreates the idle watchdog per embedded attempt and only has a broad 32-160 iteration backstop, while #76293 supplies concrete zero-output timeout storm evidence; I did not run a live paid-provider reproduction.

Next step before merge
No repair job: this is an active contributor PR with no discrete actionable defect from this pass, so the remaining path is normal review and CI gating.

Security
Cleared: The diff only changes local TypeScript run-loop logic, a colocated unit test, and CHANGELOG.md; it does not touch dependencies, workflows, package scripts, permissions, or secrets.

Review details

Best possible solution:

Land the run-loop-level breaker after exact-head checks and maintainer review, then let the linked bug close through the merge.

Do we have a high-confidence way to reproduce the issue?

Yes. By source inspection, current main recreates the idle watchdog per embedded attempt and only has a broad 32-160 iteration backstop, while #76293 supplies concrete zero-output timeout storm evidence; I did not run a live paid-provider reproduction.

Is this the best way to solve the issue?

Yes. The current PR head uses the maintainable boundary: breaker state lives in the outer embedded run where it survives attempt/profile retries, and the helper tests cover cap/reset behavior including partial tool-argument token dribbles.

What I checked:

Likely related people:

  • steipete: Current-main blame for the idle-timeout wrapper points to Peter Steinberger, and the PR head was force-pushed by steipete to address the latest breaker semantics and validation notes. (role: recent maintainer and final PR updater; confidence: high; commits: 5ab4e6f9d9, bc820665d879; files: src/agents/pi-embedded-runner/run.ts, src/agents/pi-embedded-runner/run/llm-idle-timeout.ts, src/agents/pi-embedded-runner/run/idle-timeout-breaker.ts)
  • liuy: Prior ClawSweeper history for this PR identifies liuy as the author of the original LLM streaming idle-timeout wrapper and tests that this PR works around at the run-loop level. (role: introduced idle-timeout watchdog; confidence: medium; commits: 84b72e66b918; files: src/agents/pi-embedded-runner/run/attempt.ts, src/agents/pi-embedded-runner/run/llm-idle-timeout.ts, src/agents/pi-embedded-runner/run/llm-idle-timeout.test.ts)
  • simonusa: Prior review context connects simonusa to recent embedded-run timeout classification and assistant failover changes adjacent to the retry and timeout paths. (role: recent adjacent timeout maintainer; confidence: medium; commits: 2605490dbd30; files: src/agents/pi-embedded-runner/run.ts, src/agents/pi-embedded-runner/run/assistant-failover.ts, src/agents/pi-embedded-runner/run/types.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against fc570d0e58f7.

@adhirajhangal

Copy link
Copy Markdown
Contributor Author

Pushed f3a4a5d to fix the check-test-types failure I introduced. Was: driveTimeouts cast {} to Parameters<typeof params.baseFn>[N], but the passed-in baseFn has the broad ReturnType<typeof vi.fn> type which does not satisfy Parameters<>'s (...args: any) => any constraint, so tsc bailed with TS2344 on lines 268-270. Now: import StreamFn from @mariozechner/pi-agent-core (same import the production file uses), derive StreamFnArgs = Parameters<StreamFn>, and cast the tuple straight to that.

The other two failures are pre-existing on main and unrelated to this PR:

  • check-additional-boundaries is the lint:tmp:no-raw-channel-fetch check flagging raw fetch() in extensions/chutes/models.ts:536,543 and extensions/huggingface/models.ts:143. I haven't touched any of those files
  • checks-fast-contracts-plugins-c is extensions/tlon keeping the unused dependency @tloncorp/tlon-skill. Same, not mine

Per CONTRIBUTING.md ("Do not submit test or CI-config fixes for failures already red on main CI"), leaving those two alone for the maintainer team to track.

@adhirajhangal

Copy link
Copy Markdown
Contributor Author

Pushed 06e7ef9 addressing the clawsweeper [P2] and [P3] findings.

[P2] Counter moved from wrapper to outer run loop. The bot was right: the per-attempt session and streamWithIdleTimeout wrapper installed at attempt.ts:2145 get recreated on every iteration of runEmbeddedAttemptWithBackend in run.ts:927, so a wrapper-local counter resets every time and the cap never trips on the actual retry path. v1 was theater. v3 puts the state in run.ts next to sameModelIdleTimeoutRetries (so it survives across attempts AND across profile/model failover), updates it right after the attempt is destructured (run.ts:1188), and bails through the existing handleRetryLimitExhaustion path with a distinct error message when the cap is hit. The wrapper at llm-idle-timeout.ts is restored to its original upstream shape, no behavior change there.

The "no model output" gate. The breaker only counts attempts where attemptUsage.output === 0 AND idleTimedOut, so a slow-but-responsive stream that produced 200 tokens then idle-timed-out at the tail is NOT treated as a wedged provider. Resets the moment any attempt produces output. Reserved specifically for the truly-no-output wedge that drove the reporter's incidents.

Tests rewritten as integration tests in src/agents/pi-embedded-runner/run.idle-timeout-breaker.test.ts using the existing run.overflow-compaction.harness:

  1. 12 wedged attempts queued, mockedRunEmbeddedAttempt called exactly 5 times, error message matches /cost-runaway breaker tripped/
  2. 8 productive-but-timed-out attempts queued, breaker stays disarmed
  3. 4 wedged + 1 productive + 4 wedged queued, breaker stays disarmed because the productive attempt resets the counter between the two runs of 4

The 5 wrapper-level unit tests from v1 were removed (the behavior they covered no longer exists in the wrapper). llm-idle-timeout.test.ts is back at the upstream shape.

[P3] CHANGELOG entry. Added an Unreleased Fixes line under Agents/idle-timeout referencing #76293, matching the existing voice in that section.

Acceptance commands per the bot's review:

  • pnpm test src/agents/pi-embedded-runner/run/llm-idle-timeout.test.ts src/agents/pi-embedded-runner/run.idle-timeout-breaker.test.ts (substituted my new file for the compaction one since this is the run-loop test for this fix)
  • pnpm exec oxfmt --check and pnpm check:changed would be next, but pnpm is not installed in my sandbox so relying on PR CI for those.

Diff is now +186/-232 (net negative). Marking as ready for re-review.

@adhirajhangal
adhirajhangal force-pushed the fix/76293-llm-idle-timeout-circuit-breaker branch from 61d131d to 2aa59c9 Compare May 3, 2026 01:43
@adhirajhangal

Copy link
Copy Markdown
Contributor Author

Pushed 27ff168 addressing clawsweeper [P3]: added Thanks @ThePuma312 to the changelog entry, since they filed #76293. No code change, just the credit.

Bot's [P2] / architectural concern from the previous review is fully resolved on the rebased branch — counter lives at the run loop level next to sameModelIdleTimeoutRetries, not in the wrapper, integration tests cover the actual loop via the existing harness. Re-review summary marked patch as correct with confidence 0.86.

CI on the prior commit was 77 success / 21 skipped / 1 neutral with one in-progress check (the pre-existing main-failure that doesn't gate this PR). Will let it ride.

@adhirajhangal

Copy link
Copy Markdown
Contributor Author

Pushed c87d455 addressing both clawsweeper [P2] findings on the integration test.

You were right on two counts: (1) the v3 integration test asserted exactly 5 mocked attempts but with the harness default mockedResolveAuthProfileOrder of [], the run loop only allows ~2 attempts before bailing through assistantFailoverOutcome.throw, so the assertion was unreachable. (2) On top of that, the file naming run.idle-timeout-breaker.test.ts did not match the existing agents test shard's glob, so the test was never being executed in CI in the first place. Two layers of broken.

Fix: extracted the breaker into a pure helper at src/agents/pi-embedded-runner/run/idle-timeout-breaker.ts with MAX_CONSECUTIVE_IDLE_TIMEOUTS_BEFORE_OUTPUT, createIdleTimeoutBreakerState(), and stepIdleTimeoutBreaker(state, input, opts?). run.ts owns the state at the outer-loop scope (next to sameModelIdleTimeoutRetries), calls the helper after each attempt is destructured, and bails through the existing handleRetryLimitExhaustion path when step.tripped is true. Same product behavior as v3, just with the logic in a place that can be exercised directly.

New tests at src/agents/pi-embedded-runner/run/idle-timeout-breaker.test.ts (sibling shard to the existing run/llm-idle-timeout.test.ts so it actually gets picked up). 9 cases covering: default cap matches the constant the run loop reads from; one wedged attempt does not trip; five wedged in a row trips; explicit smaller cap respected; cap=0 escape hatch; slow-but-alive (timeout but produced output) never trips; productive attempt between two runs of 4 wedged resets the counter; non-timeout error attempts neither poison nor reset; defensive coercion of NaN/negative/Infinity outputTokens to 0.

Deleted the v3 run.idle-timeout-breaker.test.ts (the unreachable, never-executed one).

CHANGELOG entry from v3 unchanged (this is a refactor of the implementation, not a behavior change). Same product fix.

@martingarramon

Copy link
Copy Markdown
Contributor

The breaker location is correct: outer-loop scope survives same-model retries and attempt boundaries, helper is pure + well-tested, five-call worst-case bound on the reported $20-30 incident pattern.

Tool-call-only attempts may falsely reset the breaker. attemptUsage?.output (run.ts:1180) reflects aggregate provider output tokens, not text-only. Tool-call argument deltas stream incrementally before completion (openai-transport-stream.ts:477,484) and the idle watchdog races each iterator next() (llm-idle-timeout.ts:121,129), so a wedged provider that emits partial tool args before stalling would report outputTokens > 0 → reset to 0 (helper line 333). Worth a test case for idleTimedOut: true, outputTokens > 0 from tool-args only and a decision on whether to count text-output-only.

Failover state persistence is split. Profile rotation within one runEmbeddedPiAgent invocation persists the counter (outer-loop scope). Model fallback goes through runAgentAttemptagent-command.ts:985command/attempt-execution.ts:561 → fresh runEmbeddedPiAgent invocation, which resets idleTimeoutBreakerState. The CHANGELOG claim "fires regardless of failover" likely doesn't hold for the model-fallback path. Confirm and either tighten the changelog or hoist state to the surviving scope.

Observability — distinct trip signal. The trip routes through handleRetryLimitExhaustion and reuses generic retry_limit (retry-limit.ts:27,41). Operators reading logs/metrics can't distinguish a cost-runaway trip from ordinary retry exhaustion. Worth a distinct log tag ([idle-timeout-circuit-breaker-tripped]) or error-kind metadata so dashboards can alert separately.

Hardcoded cap (non-blocking). Helper accepts options.cap but the call site uses the default. Existing timeouts surface through config (types.agent-defaults.ts:340/384/496); follow-up agents.idleTimeoutBreakerCap would let ops teams tune by paid-vs-free-tier.

@steipete
steipete force-pushed the fix/76293-llm-idle-timeout-circuit-breaker branch from c87d455 to bc82066 Compare May 3, 2026 13:07
@steipete

steipete commented May 3, 2026

Copy link
Copy Markdown
Contributor

Addressed the tool-call-only reset concern in bc82066.

  • src/agents/pi-embedded-runner/run.ts now resets the breaker only for completed model progress: non-empty assistant text, completed regular/hosted tool calls, messaging-tool delivery evidence, or completed item lifecycle.
  • src/agents/pi-embedded-runner/run/idle-timeout-breaker.ts intentionally ignores raw outputTokens as the reset signal, so billed partial tool-argument deltas before an idle stall still count toward the breaker.
  • src/agents/pi-embedded-runner/run/idle-timeout-breaker.test.ts adds the regression case for idleTimedOut: true with positive outputTokens but no completed progress.
  • Also tightened the changelog wording to “same embedded run” and added a distinct [idle-timeout-circuit-breaker-tripped] log tag.

Local/Testbox proof:

  • pnpm test src/agents/pi-embedded-runner/run/idle-timeout-breaker.test.ts
  • pnpm exec oxfmt --check --threads=1 src/agents/pi-embedded-runner/run.ts src/agents/pi-embedded-runner/run/idle-timeout-breaker.ts src/agents/pi-embedded-runner/run/idle-timeout-breaker.test.ts
  • Testbox pnpm check:changed on tbx_01kqpz2cptpq5fbvvbcywvmht5

@steipete
steipete merged commit 8f20d03 into openclaw:main May 3, 2026
101 checks passed
@steipete

steipete commented May 3, 2026

Copy link
Copy Markdown
Contributor

Landed via rebase onto main.

  • Gate: exact-head CI green; local helper test and Testbox pnpm check:changed passed.
  • Source head: bc82066
  • Landed commit: 8f20d03

Thanks @adhirajhangal!

lxe pushed a commit to lxe/openclaw that referenced this pull request May 6, 2026
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: [Bug] LLM idle timeout triggers unbounded retry loop, causing massive cost spikes

3 participants