fix(codex): bound shared app-server startup waits by zhangguiping-xydt · Pull Request #89442 · openclaw/openclaw

zhangguiping-xydt · 2026-06-02T12:37:15Z

What Problem This Solves

Codex app-server startup can stall while the shared client is waiting for the protocol initialize response or post-initialize authentication. Those waits were not independently bounded per caller, so a short caller could either receive a generic outer startup error or retire a physical client that a longer-lived caller still needed.

This PR bounds those shared startup phases per acquirer and reports the phase that actually timed out. It hardens the initialize path related to #84567, but does not close that issue: the reporter's authenticated OAuth/recurring-cron failure has not been reproduced here.

Why This Change Was Made

Codex permits one connection-scoped initialize handshake. The physical app-server startup therefore remains shared, while each OpenClaw caller applies its own initialize and authentication deadline. A caller that times out releases only its pending ownership; the transport closes only when no other pending acquire or active lease remains.

The shared startup state now exposes separate initialized and ready phases. That preserves accurate codex app-server initialize timed out versus codex app-server authentication timed out diagnostics, while the attempt owner passes the configured app-server request timeout into the client factory. The abandoned-startup retry guard remains in place.

The earlier synthetic cron harness was removed. Focused regressions now cover real stdio initialize stalls, concurrent short/long waiters, authentication stalls, claimant-aware cleanup, and late factory results.

User Impact

Stalled Codex app-server initialize and authentication operations fail within the configured request budget with specific diagnostics. A short-lived startup attempt no longer kills a shared client that another caller can still initialize and use. Successful startup, provider/model routing, OAuth profile selection, protocol payloads, and public configuration remain unchanged.

Evidence

Exact reviewed head: 24b6bba99aaeb3d8dbf9a3dc898e92e7a597dce6.
Focused sanitized-AWS tests: run_024de96a615f, 3 files / 43 tests passed.
Sanitized-AWS changed gate: run_09142eb448be, all guards, production/test types, lint, DB/runtime, and import-cycle checks passed.
Real built Gateway -> naturally scheduled isolated cron -> Codex stdio initialize-stall proof: run_ac03877e9405. Both initialize-only child processes exited, the run and persisted job carried the specific initialize-timeout diagnostic, and the Gateway remained healthy.
Exact-head hosted CI: run 29150400559, 56 relevant checks passed, 0 failed.
Source-blind behavior-contract validation: pass, 0.94 confidence; reporter-account OAuth reproduction remains out of scope.
Fresh autoreview: clean; no accepted or actionable findings remain.

clawsweeper · 2026-06-02T12:38:39Z

Codex review: needs changes before merge. Reviewed July 11, 2026, 7:13 AM ET / 11:13 UTC.

Summary
The PR refactors shared Codex app-server startup to expose initialize and ready phases, adds per-acquire timeout and abandonment handling, and expands startup concurrency and isolated-cron regression coverage.

PR surface: Source +52, Tests +201. Total +253 across 4 files.

Reproducibility: yes. A real stdio child can receive the actual initialize request and stall, and the production isolated-cron path has a clear outer watchdog that can mask that phase.

Review metrics: 1 noteworthy metric.

Startup deadline sources: 2 independent budgets used. The outer startupTimeoutMs and inner fixed requestTimeoutMs can expire in a different order, which is central to the diagnostic this PR intends to fix.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #84567
Summary: This PR is the candidate diagnostic fix for the isolated cron report; the earlier initialize-hang report shares the lifecycle cause, while the other open PR addresses connection-close retry exhaustion.

Members:

canonical: [Bug]: Codex bundled harness initialize still hangs in 2026.5.18 isolated cron — surfaces via #64744 timeout-wrapping as 'isolated agent setup timed out before runner start' #84567 - It contains the concrete isolated cron reproduction and generic setup-timeout symptom targeted by this PR.
same_root_cause: [Bug]: codex/gpt-5.4 via bundled Codex harness hangs indefinitely on initialize in v2026.4.10 (macOS, codex-cli 0.118.0 and 0.120.0) #64744 - It established the unbounded Codex initialize wait that should receive phase-specific timeout attribution.
partial_overlap: fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted] #94137 - It modifies the same startup retry loop for connection-close exhaustion, but does not solve remaining-deadline attribution.
adjacent_distinct: Fix isolated cron cold runner setup timeout #86893 - It proposed extending the cron watchdog window rather than attributing a stall inside Codex initialize.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Carry one absolute outer startup deadline through initialize and authentication.
[P2] Add a regression where pre-initialize work consumes part of the budget before initialize stalls.
Recheck integration with the startup retry changes in fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted] #94137.

Risk before merge

[P1] Merging as-is can still surface isolated agent setup timed out before runner start when start-context work consumes part of the outer budget before initialize stalls.
[P1] The two sequential full-duration waits can extend a shared acquire beyond the intended remaining startup budget, increasing Codex startup latency before abandonment.
[P1] The patch overlaps extensions/codex/src/app-server/attempt-startup.ts with fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted] #94137, so whichever lands second needs an integration review of retry and abandonment ordering.

Maintainer options:

Carry the real remaining deadline (recommended)
Compute the outer startup deadline once, consume its remainder in shared-client initialize/auth waits, and add a regression with delayed pre-initialize setup followed by a stalled initialize.
Accept generic timeout races
Merge the current implementation knowing the outer generic startup timeout can still beat the initialize-specific diagnostic after earlier setup delay.
Pause for adjacent startup integration
Wait for the retry work in fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted] #94137 and combine both startup state-machine changes in one reviewed patch.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Preserve the shared per-caller ownership and abandonment model, but pass the outer startup watchdog's actual remaining deadline into client acquisition; recompute remaining time across initialize and authentication, and add a regression where pre-initialize work consumes budget before initialize stalls.

Next step before merge

The blocker is a narrow deadline-accounting defect in the touched startup path with a clear regression-test shape.

Security
Cleared: The patch introduces no concrete security or supply-chain regression; it changes only Codex startup lifecycle code and tests.

Review findings

[P1] Carry the outer startup deadline into initialize — extensions/codex/src/app-server/attempt-startup.ts:210

Review details

Best possible solution:

Capture one absolute outer startup deadline before setup begins, pass its remaining milliseconds into the shared acquire, and recompute the remainder before initialize/auth phase waits so the specific lifecycle error wins without extending the total watchdog budget.

Do we have a high-confidence way to reproduce the issue?

Yes. A real stdio child can receive the actual initialize request and stall, and the production isolated-cron path has a clear outer watchdog that can mask that phase.

Is this the best way to solve the issue?

No, not yet. Moving timeout attribution into the initialize owner is the right boundary, but the implementation must carry the outer watchdog's actual remaining deadline rather than introduce independent fixed-duration waits.

Full review comments:

[P1] Carry the outer startup deadline into initialize — extensions/codex/src/app-server/attempt-startup.ts:210
timeoutMs here is the fixed app-server request timeout, not the remaining withCodexStartupTimeout budget. If start-context/factory work consumes part of the outer window, the generic outer watchdog can still fire before this inner timer, reproducing the diagnostic masking this PR claims to fix. Compute an absolute startup deadline at the outer owner and pass/recompute its remainder for initialize (and auth), with a delayed-pre-initialize regression.
Confidence: 0.96

Overall correctness: patch is incorrect
Overall confidence: 0.95

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f8aa995856ec.

Label changes

Label changes:

add merge-risk: 🚨 availability: The PR changes timeout ordering, retries, and shared-client retirement in the Codex app-server startup path.
add proof: sufficient: Contributor real behavior proof is sufficient. The contributor provides after-fix real-stdio subprocess output and production isolated-cron route proof; an additional delayed-pre-initialize case is needed for the correctness finding, not to satisfy the general proof gate.
add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The contributor provides after-fix real-stdio subprocess output and production isolated-cron route proof; an additional delayed-pre-initialize case is needed for the correctness finding, not to satisfy the general proof gate.

Label justifications:

P1: Isolated cron Codex jobs can deterministically fail before runner start, blocking real scheduled workflows.
merge-risk: 🚨 availability: The PR changes timeout ordering, retries, and shared-client retirement in the Codex app-server startup path.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The contributor provides after-fix real-stdio subprocess output and production isolated-cron route proof; an additional delayed-pre-initialize case is needed for the correctness finding, not to satisfy the general proof gate.
proof: sufficient: Contributor real behavior proof is sufficient. The contributor provides after-fix real-stdio subprocess output and production isolated-cron route proof; an additional delayed-pre-initialize case is needed for the correctness finding, not to satisfy the general proof gate.

Evidence reviewed

PR surface:

Source +52, Tests +201. Total +253 across 4 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	86	34	+52
Tests	2	212	11	+201
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	4	298	45	+253

Acceptance criteria:

[P1] node scripts/run-vitest.mjs extensions/codex/src/app-server/shared-client.test.ts.
[P1] node scripts/run-vitest.mjs extensions/codex/src/app-server/attempt-startup.test.ts.
[P1] node scripts/run-vitest.mjs src/cron/isolated-agent.codex-app-server.test.ts.
[P1] git diff --check.

What I checked:

Wrong deadline source: The startup attempt passes params.appServer.requestTimeoutMs into the shared-client factory, while the enclosing owner is governed separately by params.startupTimeoutMs; no remaining outer deadline is computed or forwarded. (extensions/codex/src/app-server/attempt-startup.ts:210, 24b6bba99aae)
Independent phase timers: The shared acquire applies the full supplied timeout first to startup.initialized and then again to startup.ready, rather than consuming one remaining absolute startup deadline. (extensions/codex/src/app-server/shared-client.ts:215, 24b6bba99aae)
Configured values can coincide: Codex requestTimeoutMs defaults to 60,000 ms, while isolated startup has its own separately resolved startup timeout; using the fixed request timeout does not preserve elapsed pre-initialize budget. (extensions/codex/src/app-server/config.ts:794, f8aa995856ec)
Upstream initialize is a real awaited protocol phase: Codex app-server dispatches Initialize asynchronously before allowing initialized requests, so a client-side bounded wait at this phase is appropriate; upstream does not supply OpenClaw's outer cron deadline. (../codex/codex-rs/app-server/src/message_processor.rs:764, unknown)
Strong but incomplete proof: The PR body demonstrates a child that stalls directly in initialize and an isolated route with successful initialize, but does not cover elapsed factory/start-context time followed by an initialize stall—the case needed to prove the remaining outer deadline wins with the specific error. (extensions/codex/src/app-server/attempt-startup.test.ts:1, 24b6bba99aae)
Current-head CI: Real behavior proof, build, lint, types, dependency/security guards, and most test shards passed; two broad test shards were still pending when inspected. (24b6bba99aae)

Likely related people:

steipete: He is assigned on the live PR and authored the current-head shared-startup lifecycle and test refinements. (role: current assignee and recent lifecycle contributor; confidence: high; commits: 031d9eecc810, 4107da1c2027, 074b9259812e; files: extensions/codex/src/app-server/shared-client.ts, extensions/codex/src/app-server/attempt-startup.ts, extensions/codex/src/app-server/shared-client.test.ts)
Peter Steinberger: Git history ties the shared Codex app-server lifecycle to the original harness disposal and app-server module split. (role: feature introducer and refactor author; confidence: high; commits: c643e3c72de7, 8d72aafdbb8d; files: extensions/codex/src/app-server/shared-client.ts, extensions/codex/src/app-server/attempt-startup.ts)
chengzhichao-xydt: Current-main blame shows recent work across the same shared-client timeout and initialization surface, making them useful for follow-up context. (role: recent adjacent contributor; confidence: medium; commits: a11449211a2c; files: extensions/codex/src/app-server/shared-client.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (5 earlier review cycles)

reviewed 2026-06-28T07:34:13.613Z sha f1693a2 :: needs maintainer review before merge. :: none
reviewed 2026-07-05T15:10:09.645Z sha e08b763 :: needs maintainer review before merge. :: none
reviewed 2026-07-05T18:39:38.882Z sha e08b763 :: needs maintainer review before merge. :: none
reviewed 2026-07-07T16:19:56.440Z sha ad61a71 :: needs maintainer review before merge. :: none
reviewed 2026-07-09T13:10:31.371Z sha 19a5def :: needs maintainer review before merge. :: none

zhangguiping-xydt · 2026-07-05T18:25:31Z

@clawsweeper re-review

clawsweeper · 2026-07-05T18:25:34Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Co-authored-by: 张贵萍0668001030 <[email protected]>

steipete · 2026-07-11T11:57:57Z

Land-ready at exact head 24b6bba99aaeb3d8dbf9a3dc898e92e7a597dce6.

What changed:

Shared Codex app-server startup keeps one physical initialize/auth pipeline while each caller owns its own phase deadline.
A timed-out caller releases only its claim; the physical client closes only when no pending acquire or active lease remains.
Isolated startup keeps caller-owned teardown, the configured request timeout reaches the client factory, and abandoned startup does not retry.
Focused regressions cover concurrent short/long waiters, initialize/auth stalls, late factory results, claimant-aware cleanup, and a real stdio child.

Proof:

Focused sanitized AWS: run_024de96a615f, 3 files / 43 tests passed.
Changed gate: run_09142eb448be, all guards, production/test types, lint, DB/runtime, and import-cycle checks passed.
Live built Gateway and naturally scheduled isolated cron: run_ac03877e9405, exit 0. Run error, diagnostics, and persisted job state carried codex app-server initialize timed out; both initialize-only child processes exited; Gateway health remained OK.
Hosted exact-head CI: run 29150400559, 56 relevant checks passed, 0 failed.
Fresh autoreview: clean. Source-blind behavior-contract validation: pass at 0.94 confidence.

Known proof gap: the reporter-account OAuth/recurring-cron path from #84567 was not reproduced. This lands bounded initialize/auth hardening and leaves #84567 open.

CHANGELOG.md is release-owned; release-note context is captured in the PR body.

steipete · 2026-07-11T11:59:06Z

Merged via squash.

Prepared head SHA: 24b6bba99aaeb3d8dbf9a3dc898e92e7a597dce6
Landed commit: 745cfc028edb2d659c382d88629a70b1ee2a45fc

openclaw-barnacle Bot added extensions: codex size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 2, 2026

openclaw-barnacle Bot added size: S and removed size: XS labels Jun 2, 2026

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 2, 2026

mbelinky mentioned this pull request Jun 3, 2026

feat(cron): support command jobs #89712

Merged

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 7, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 7, 2026

zhangguiping-xydt force-pushed the feat/issue-84567 branch from 8931293 to 2c374aa Compare June 9, 2026 00:01

openclaw-barnacle Bot added size: L and removed size: S labels Jun 9, 2026

zhangguiping-xydt force-pushed the feat/issue-84567 branch 2 times, most recently from ad61a71 to 19a5def Compare July 9, 2026 12:32

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(codex): backoff startup connection-close retries and surface exhaustion (#83959) [AI-assisted] #94137

Open

steipete self-assigned this Jul 11, 2026

steipete force-pushed the feat/issue-84567 branch 2 times, most recently from 9a8d0bd to 81eaca9 Compare July 11, 2026 10:56

zhangguiping-xydt and others added 6 commits July 11, 2026 12:05

fix(codex): isolated cron reports Codex startup stalls

6f388f2

fix(codex): isolated cron reports Codex startup stalls

0523402

fix(codex): bound shared app-server startup waits

031d9ee

Co-authored-by: 张贵萍0668001030 <[email protected]>

test(codex): satisfy startup lifecycle checks

4107da1

test(codex): align deferred auth test type

074b925

test(codex): type auth mock as async void

24b6bba

steipete force-pushed the feat/issue-84567 branch from 81eaca9 to 24b6bba Compare July 11, 2026 11:05

github-actions Bot mentioned this pull request Jul 11, 2026

📡 Upstream Digest — 2026-07-11 12:59 UTC curtismercier/openclaw-mods#1261

Open

This was referenced Jul 11, 2026

fix(cli,doctor): use own-property checks for config field lookups #104512

Closed

chore(i18n): refresh native locales #104541

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(codex): bound shared app-server startup waits#89442

fix(codex): bound shared app-server startup waits#89442
steipete merged 6 commits into
openclaw:mainfrom
zhangguiping-xydt:feat/issue-84567

zhangguiping-xydt commented Jun 2, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

zhangguiping-xydt commented Jul 5, 2026

Uh oh!

clawsweeper Bot commented Jul 5, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 11, 2026

Uh oh!

steipete commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

zhangguiping-xydt commented Jun 2, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangguiping-xydt commented Jul 5, 2026

Uh oh!

clawsweeper Bot commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 11, 2026

Uh oh!

steipete commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhangguiping-xydt commented Jun 2, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

clawsweeper Bot commented Jul 5, 2026 •

edited

Loading