fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted]#94137
Conversation
|
CI status note (round 1):
Nothing actionable from my side until maintainer review/approval. Happy to address any review feedback. |
|
Codex review: needs maintainer review before merge. Reviewed July 9, 2026, 12:09 PM ET / 16:09 UTC. Summary PR surface: Source +45, Tests +85. Total +130 across 2 files. Reproducibility: yes. at source level. Current main catches Codex app-server closed-client and exit errors in Review metrics: 1 noteworthy metric.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Land this PR after maintainer acceptance of the bounded backoff/error-shape tradeoff and a final current-base check, or ask for a readiness-gate variant if maintainers prefer explicit server readiness over time-based retry admission. Do we have a high-confidence way to reproduce the issue? Yes, at source level. Current main catches Codex app-server closed-client and exit errors in Is this the best way to solve the issue? Yes, with one maintainer tradeoff. The fix lives in the Codex app-server startup owner and reuses the shared backoff/sleep helper, while maintainers still need to decide whether bounded sleep is preferable to an explicit readiness gate. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 4f9a371c9530. Label changesLabel justifications:
Evidence reviewedPR surface: Source +45, Tests +85. Total +130 across 2 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
Real behavior proof upgraded in response to The previous capture reimplemented the retry loop ( Observed on the same
Full terminal captures are in the
|
…ustion (openclaw#83959) The Codex app-server startup retry loop re-acquired the shared client immediately after a connection-close, before the replacement app-server process had time to bind, and exhausted a 3-attempt budget in the same failing window. Scheduled/background turns then failed even though the route became viable moments later. Add a bounded, abortable backoff between startup retries, raise the budget from 3 to 5, and throw a distinct CodexAppServerStartupExhausted Error (original error preserved as cause) when the window is exhausted, so callers can distinguish startup lifecycle exhaustion from a mid-turn client close. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2746c6f to
40c0d9f
Compare
|
This pull request has been automatically marked as stale due to inactivity. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted] This is item 1/1 in the current shard. Shard 1/4. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
Summary
startCodexAttemptThreadcleared the shared client and retried immediately — before the replacement app-server process had bound/become ready — under a tight 3-attempt budget. All retries landed in the same failing window and exhausted, so a scheduled/background turn failed even though the Codex route became viable moments later. Cron automation reported job failure with no task-level problem. (Issue Codex app-server startup retries can exhaust before replacement server is ready #83959)embedded-agent-runner/run.ts(already correct, untouched) and the shared-client eviction work tracked under Codex harness app-server shared client is evicted across agents because cache key includes agentDir #79495.attempt-startup.ts(budget 3→5, addedcomputeBackoff+sleepWithAbortbackoff, newCodexAppServerStartupExhaustedError) and the two new regression tests.Linked context
Closes #83959
Related #79495 (shared Codex app-server client eviction across agents — broader, separate).
Was this requested by a maintainer or owner? No — external contributor fix for an open P1.
Real behavior proof (required for external PRs)
startCodexAttemptThread(the production function this PR changes) against a REAL local app-server subprocess spawned by the REALCodexAppServerClient.start->createStdioTransport->child_process.spawn. The subprocess speaks the same newline-delimited JSON-RPC framing as the real Codex app-server; on the first N attempts it closes its connection during theinitializehandshake (real child exit -> realcodex app-server exited:error -> realisCodexAppServerConnectionClosedError-> the real startup retry loop), and completes the handshake on attempt N+1. It is a stand-in for thecodexbinary (no Codex account/backend is available on this host); it is NOT a reimplementation of the retry loop and does not stubstartCodexAttemptThread.pnpm exec tsx .tmp/capture-83959-real.tswithCLOSES_BEFORE_SUCCESS=4(recovery) andCLOSES_BEFORE_SUCCESS=99(persistent-close exhaustion), run on unpatched main and on this PR branch. Driver:.tmp/capture-83959-real.ts; stand-in app-server subprocess:.tmp/fake-codex-app-server.mjs.codex app-server exited: code=1 signal=null(plainError, no distinct exhaustion type). AFTER (this PR, budget 5 + bounded abortable backoff) — the same 4 leading closes were survived: startup recovered on the 5th attempt with 4 monotonically increasing gaps [1962, 2531, 3399, 5340] ms that track the realcomputeBackoffpolicy [532, 1179, 2019, 4000] ms (policy value + spawn/cleanup overhead), and the turn succeeded. Persistent-close AFTER — exhausted at 5 with the distinctCodexAppServerStartupExhaustedError("codex app-server startup exhausted after 5 connection-close retries"), the originalcodex app-server exited:preserved ascause, instead of the raw close error.codexapp-server process backed by a real Codex/LLM account (no local Codex harness or API key on this Windows host). The stand-in subprocess reproduces the connection-close-during-startup condition and the JSON-RPC initialize/thread-start handshake but does not execute Codex model turns. macOS/Linux runtime not exercised locally.computeBackoff+sleepWithAbortwall-clock delays, realCodexAppServerStartupExhaustedError); it is not the realcodexbinary because no Codex backend is available on this host.Tests and validation
pnpm exec oxfmt --check(changed files, clean);pnpm tsgo:extensions(clean);pnpm tsgo:extensions:test(clean);pnpm exec oxlint --type-aware(changed files, clean);node scripts/run-vitest.mjs extensions/codex/src/app-server/attempt-startup.test.ts -t "83959"(2 passed).attempt-startup.test.tscovering the previously-untested startup connection-close retry loop — (1) survives N transient closes with bounded backoff and recovers, assertingsleepWithAbortis awaited between attempts with bounded delays; (2) persistent closes produce the distinctCodexAppServerStartupExhaustedErrorwith the original error ascause, and the loop is bounded (no infinite retry).pnpm check:host-env-policy:swiftskipped on Windows (no Swift).Risk checklist
Did user-visible behavior change?
Yes— a Codex startup that previously failed after 3 immediate retries now retries up to 5 times with a bounded backoff (worst case a few extra seconds before a transient-close recovery or a cleaner failure).Did config, environment, or migration behavior change?
No.Did security, auth, secrets, network, or tool execution behavior change?
No— the change only affects the Codex app-server startup retry timing/error shape; no auth, secret, network, or tool-execution surface touched.extensions/codex/src/**is not a CODEOWNERS secops path.What is the highest-risk area? Added latency on a Codex startup that closes repeatedly during warm-up (bounded to ~8s worst case across 5 attempts).
How is that risk mitigated? Backoff only fires after a connection-close failure (never on the success path);
sleepWithAbortis abortable so a cancelled/aborted run fails fast instead of stalling; the budget is still bounded (5) so a permanently-broken server fails rather than retrying forever; the distinct exhaustion error preserves the original cause for diagnostics.Current review state
What is the next action? Awaiting maintainer review. The real behavior proof was upgraded (see above) from a synthetic retry-loop model to a real local app-server subprocess driving the actual
startCodexAttemptThread;@clawsweeper re-reviewrequested.What is still waiting on author, maintainer, CI, or external proof? External-fork heavy CI shards require maintainer workflow approval to run. A maintainer proof/latency override is also an acceptable path per ClawSweeper's option 2 if the stand-in-subprocess proof is deemed sufficient.
Which bot or reviewer comments were addressed? Upgraded the real behavior proof in response to ClawSweeper's
status: 📣 needs proof("supplies tests and a synthetic terminal model, but not after-fix live OpenClaw/Codex app-server startup proof").