Skip to content

Spawned app-server initialize has a hard 10s deadline with no user-facing override #55

Closed

Description

@axisrow

Split out from #53, which fixed the test-side flake. This one is about production code and is deliberately left untouched by PR #54.

What I found

While stress-testing the #53 fix I pushed past the point where scaled test budgets help. Beyond ~10 concurrent test processes on a 10-core box (~1.6x oversubscription), failures stop being test-timeout-shaped and start reporting:

Error: codex app-server initialize timed out.

at almost exactly 10003 ms — i.e. the production constant, not a test budget:

  • plugins/codex/scripts/lib/app-server.mjs:34 — const SPAWNED_INITIALIZE_TIMEOUT_MS = 10000;
  • plugins/codex/scripts/lib/app-server.mjs:280 — timeoutMs: this.options.spawnedInitializeTimeoutMs ?? SPAWNED_INITIALIZE_TIMEOUT_MS

Why this is worth a separate issue

To be clear about severity: a spawn that takes >10s genuinely is a problem, and the timeout firing is a truthful signal, not a bug in itself. I am not proposing we simply raise the number.

The gap is that there is no way for a user to adjust it. The spawnedInitializeTimeoutMs option exists, but grepping shows nothing in production sets it — the only caller is a test:

plugins/codex/scripts/lib/app-server.mjs:280   this.options.spawnedInitializeTimeoutMs ?? SPAWNED_INITIALIZE_TIMEOUT_MS
tests/spawned-client-initialize-timeout.test.mjs:25   spawnedInitializeTimeoutMs: 300

So on a genuinely slow or heavily loaded machine — a busy CI runner, a container with a tight CPU quota, a cold start where the codex binary is not in page cache — /codex:* fails with a message the user cannot act on and cannot tune.

There is already precedent for exactly this shape of escape hatch: CODEX_COMPANION_BROKER_IDLE_TIMEOUT_MS is a documented env override for the broker idle timer (see plugins/codex/CHANGELOG.md, plugins/codex/scripts/app-server-broker.mjs:10).

Reproduction

On a 10-core box, from a checkout with the #53 fix applied:

for i in $(seq 1 16); do
  (node --test tests/broker-smoke.test.mjs tests/broker-client-initialize-timeout.test.mjs > /tmp/v3-$i.txt 2>&1) &
done; wait
grep -h "Error:" /tmp/v3-*.txt | sort | uniq -c

Observed:

14   Error: Timed out waiting for IPC ready after 30000ms.
10   Error: codex app-server initialize timed out.

The second line is this issue. (The first is the scaled test budget also being exceeded at that extreme, which is expected and out of scope.)

Possible directions

Roughly in order of how much I would trust them:

  1. Env override, mirroring the broker idle timer — e.g. CODEX_COMPANION_SPAWNED_INITIALIZE_TIMEOUT_MS, documented in the CHANGELOG. Smallest change, matches existing convention, gives users an out.
  2. Improve the error message so it names the deadline and how to raise it. Right now codex app-server initialize timed out. says nothing about 10s or about any knob. Cheap and worth doing regardless of 1.
  3. Progress-based deadline — reset the clock while the spawned process is demonstrably making progress (stderr output, partial handshake) instead of a flat wall-clock budget. Most correct, most work; only worth it if 1 and 2 prove insufficient.

I would lean toward 1 + 2 together.

Scope note

No change proposed here, and PR #54 intentionally does not touch this constant — altering it changes product behavior, which was outside the scope of the test-flake fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions