Repository navigation
Description
Split out from #53, which fixed the test-side flake. This one is about production code and is deliberately left untouched by PR #54.
What I found
While stress-testing the #53 fix I pushed past the point where scaled test budgets help. Beyond ~10 concurrent test processes on a 10-core box (~1.6x oversubscription), failures stop being test-timeout-shaped and start reporting:
Error: codex app-server initialize timed out.
at almost exactly 10003 ms — i.e. the production constant, not a test budget:
plugins/codex/scripts/lib/app-server.mjs:34—const SPAWNED_INITIALIZE_TIMEOUT_MS = 10000;plugins/codex/scripts/lib/app-server.mjs:280—timeoutMs: this.options.spawnedInitializeTimeoutMs ?? SPAWNED_INITIALIZE_TIMEOUT_MS
Why this is worth a separate issue
To be clear about severity: a spawn that takes >10s genuinely is a problem, and the timeout firing is a truthful signal, not a bug in itself. I am not proposing we simply raise the number.
The gap is that there is no way for a user to adjust it. The spawnedInitializeTimeoutMs option exists, but grepping shows nothing in production sets it — the only caller is a test:
plugins/codex/scripts/lib/app-server.mjs:280 this.options.spawnedInitializeTimeoutMs ?? SPAWNED_INITIALIZE_TIMEOUT_MS
tests/spawned-client-initialize-timeout.test.mjs:25 spawnedInitializeTimeoutMs: 300
So on a genuinely slow or heavily loaded machine — a busy CI runner, a container with a tight CPU quota, a cold start where the codex binary is not in page cache — /codex:* fails with a message the user cannot act on and cannot tune.
There is already precedent for exactly this shape of escape hatch: CODEX_COMPANION_BROKER_IDLE_TIMEOUT_MS is a documented env override for the broker idle timer (see plugins/codex/CHANGELOG.md, plugins/codex/scripts/app-server-broker.mjs:10).
Reproduction
On a 10-core box, from a checkout with the #53 fix applied:
for i in $(seq 1 16); do
(node --test tests/broker-smoke.test.mjs tests/broker-client-initialize-timeout.test.mjs > /tmp/v3-$i.txt 2>&1) &
done; wait
grep -h "Error:" /tmp/v3-*.txt | sort | uniq -cObserved:
14 Error: Timed out waiting for IPC ready after 30000ms.
10 Error: codex app-server initialize timed out.
The second line is this issue. (The first is the scaled test budget also being exceeded at that extreme, which is expected and out of scope.)
Possible directions
Roughly in order of how much I would trust them:
- Env override, mirroring the broker idle timer — e.g.
CODEX_COMPANION_SPAWNED_INITIALIZE_TIMEOUT_MS, documented in the CHANGELOG. Smallest change, matches existing convention, gives users an out. - Improve the error message so it names the deadline and how to raise it. Right now
codex app-server initialize timed out.says nothing about 10s or about any knob. Cheap and worth doing regardless of 1. - Progress-based deadline — reset the clock while the spawned process is demonstrably making progress (stderr output, partial handshake) instead of a flat wall-clock budget. Most correct, most work; only worth it if 1 and 2 prove insufficient.
I would lean toward 1 + 2 together.
Scope note
No change proposed here, and PR #54 intentionally does not touch this constant — altering it changes product behavior, which was outside the scope of the test-flake fix.
Activity
fix: make the spawned app-server handshake deadline overridable (#55) (…