fix(agents): circuit-break consecutive LLM idle timeouts to cap retry-loop cost (#76293)#76345
Conversation
|
Codex review: needs maintainer review before merge. Summary Reproducibility: yes. By source inspection, current main recreates the idle watchdog per embedded attempt and only has a broad 32-160 iteration backstop, while #76293 supplies concrete zero-output timeout storm evidence; I did not run a live paid-provider reproduction. Next step before merge Security Review detailsBest possible solution: Land the run-loop-level breaker after exact-head checks and maintainer review, then let the linked bug close through the merge. Do we have a high-confidence way to reproduce the issue? Yes. By source inspection, current main recreates the idle watchdog per embedded attempt and only has a broad 32-160 iteration backstop, while #76293 supplies concrete zero-output timeout storm evidence; I did not run a live paid-provider reproduction. Is this the best way to solve the issue? Yes. The current PR head uses the maintainable boundary: breaker state lives in the outer embedded run where it survives attempt/profile retries, and the helper tests cover cap/reset behavior including partial tool-argument token dribbles. What I checked:
Likely related people:
Codex review notes: model gpt-5.5, reasoning high; reviewed against fc570d0e58f7. |
|
Pushed f3a4a5d to fix the The other two failures are pre-existing on
Per CONTRIBUTING.md ("Do not submit test or CI-config fixes for failures already red on main CI"), leaving those two alone for the maintainer team to track. |
|
Pushed 06e7ef9 addressing the clawsweeper [P2] and [P3] findings. [P2] Counter moved from wrapper to outer run loop. The bot was right: the per-attempt session and The "no model output" gate. The breaker only counts attempts where Tests rewritten as integration tests in
The 5 wrapper-level unit tests from v1 were removed (the behavior they covered no longer exists in the wrapper). [P3] CHANGELOG entry. Added an Unreleased Fixes line under Acceptance commands per the bot's review:
Diff is now +186/-232 (net negative). Marking as ready for re-review. |
61d131d to
2aa59c9
Compare
|
Pushed 27ff168 addressing clawsweeper [P3]: added Bot's [P2] / architectural concern from the previous review is fully resolved on the rebased branch — counter lives at the run loop level next to CI on the prior commit was 77 success / 21 skipped / 1 neutral with one in-progress check (the pre-existing main-failure that doesn't gate this PR). Will let it ride. |
|
Pushed c87d455 addressing both clawsweeper [P2] findings on the integration test. You were right on two counts: (1) the v3 integration test asserted exactly 5 mocked attempts but with the harness default Fix: extracted the breaker into a pure helper at New tests at Deleted the v3 CHANGELOG entry from v3 unchanged (this is a refactor of the implementation, not a behavior change). Same product fix. |
|
The breaker location is correct: outer-loop scope survives same-model retries and attempt boundaries, helper is pure + well-tested, five-call worst-case bound on the reported $20-30 incident pattern. Tool-call-only attempts may falsely reset the breaker. Failover state persistence is split. Profile rotation within one Observability — distinct trip signal. The trip routes through Hardcoded cap (non-blocking). Helper accepts |
Co-authored-by: adhiraj <[email protected]>
c87d455 to
bc82066
Compare
|
Addressed the tool-call-only reset concern in bc82066.
Local/Testbox proof:
|
|
Landed via rebase onto
Thanks @adhirajhangal! |
Co-authored-by: adhiraj <[email protected]>
Co-authored-by: adhiraj <[email protected]>
Co-authored-by: adhiraj <[email protected]>
Co-authored-by: adhiraj <[email protected]>
Co-authored-by: adhiraj <[email protected]>
What
Adds a circuit breaker to
streamWithIdleTimeoutthat caps how many consecutive idle timeouts can fire on the same wrapper instance before subsequentbaseFninvocations are refused. Default cap is 5. Counter resets on any successful chunk, so slow-but-responsive streams keep working.Why
Per #76293, today a single idle timeout propagates up, the upstream retry layer (pi-agent-core) re-invokes
streamFn, the retry also times out (or fails fast) against the same wedged provider, and the cycle repeats unbounded. The reporter saw 761-1384 paid Anthropic Sonnet 4.6 calls in 60 seconds across two real incidents, costing $20-30 each before any external monitor caught it. Auto-recharge on the provider account masks the spike at the time of incident, so users only find out at billing.The fix lives at the OpenClaw
streamFnwrapper level so it applies to every caller without coordinating with the upstream retry semantics. Worst case under the new default: 5 paid model calls before lockout. At Anthropic Sonnet 4.6 list pricing for the heartbeat-context size in the report, that is roughly $0.10-0.30 per incident vs $20-30 today.How it behaves
LLM idle timeout circuit breaker tripped: ...without callingbaseFn, and routes the same error throughonIdleTimeoutso the existingidleTimeoutTriggerinattempt.tsaborts the run via the same path it already takes for a regular timeoutWhat did not change
src/agents/pi-embedded-runner/run/attempt.tscall site is unchanged. The default of 5 takes effect everywhere via the function defaultonIdleTimeoutcontract: still fires once per real timeout AND once per circuit-breaker trip, so the existingabortRunpath runs unchanged. The error message lets observability tools distinguish the two cases (circuit breaker trippedvs the existingno response from model)Tests
Five new vitest cases in
src/agents/pi-embedded-runner/run/llm-idle-timeout.test.tsunderdescribe("circuit breaker - issue #76293"):DEFAULT_MAX_CONSECUTIVE_LLM_IDLE_TIMEOUTS = 5) holds under a 50-attempt simulated retry storm.baseFn.mock.calls.lengthlands at exactly 5maxConsecutiveTimeouts: 0disables the breaker (escape hatch for power users who want the old behavior). 12 simulated retries produce 12baseFncallsonIdleTimeoutwith a distinctcircuit breaker trippedmessage so the surrounding abort path keeps working and observability can split the two casesbaseFnbecause the chunk resets the counter)What I haven't verified
I'm working from a sandbox without pnpm installed, so I haven't been able to run
pnpm testorpnpm test:contracts:agentslocally. The unit tests are written and reviewed by hand against the wrapper logic. Relying on PR CI to confirm. If anything fails I'll iterate.AI / vibe-coding disclosure
codex review --base origin/mainlocally. Will address Codex review feedback on the PRCloses #76293