Skip to content

fix(broker): bound broker handshake so a wedged broker can't hang the client (closes #29) - #30

Merged
axisrow merged 2 commits into
mainfrom
fix/broker-busy-hang-29
Jul 20, 2026
Merged

axisrow merged 2 commits into
mainfrom
fix/broker-busy-hang-29

Conversation

@axisrow

@axisrow axisrow commented Jul 20, 2026

Copy link
Copy Markdown
Owner

What

When the shared broker is wedged — connect accepted, but initialize never answered — BrokerCodexAppServerClient had no deadline on net.createConnection or request("initialize"), so a second /codex:* command hung forever. This is the broker-busy hang reported in #29 (mirror of upstream openai#509).

Root cause (reproduced, not guessed)

TDD-first: spawnWedgedServer (a real net.createServer that accepts but never answers) + a test driving the real client path CodexAppServerClient.connect → BrokerCodexAppServerClient.initialize. Before the fix, connect resolved but the initialize request waited indefinitely → winner: "HANG" after 2000ms.

The broker itself does not hang: socket.on("data", async cb) doesn't block the event loop, and the busy-guard returns -32001 immediately. The hang was on the client side.

Fix

  • Bound connect (2s default) and initialize (5s default) in BrokerCodexAppServerClient.initialize, with a new BROKER_HANDSHAKE_TIMEOUT_CODE ("EBROKERTIMEOUT"). Both overridable via brokerConnectTimeoutMs / brokerInitializeTimeoutMs options. Timers are unref'd.
  • withAppServer now falls back to a direct app-server on EBROKERTIMEOUT in addition to BROKER_BUSY_RPC_CODE / ENOENT / ECONNREFUSED.

Pattern follows the existing sendBrokerShutdown bounded-wait (broker-lifecycle.mjs).

Verification

  • New test tests/broker-client-initialize-timeout.test.mjs: RED before fix (HANG), GREEN after (EBROKERTIMEOUT in 506ms).
  • Broker suite (9 tests) green.
  • npm test: no new failures vs main (4 pre-existing failures unchanged).

Follow-up (not in this PR)

SpawnedCodexAppServerClient.initialize has the same unbounded shape. A deeper fix would put requestTimeoutMs on AppServerClientBase.request() and collapse the withAppServer OR-chain into a single broker-transport-error predicate. Filed as a follow-up to keep this PR scoped to the reported broker-path hang.

🤖 Generated with Claude Code

axisrow and others added 2 commits July 20, 2026 22:36
… client (closes #29)

When the shared broker is wedged (connect accepted but initialize never
answered), BrokerCodexAppServerClient had no deadline on net.createConnection
or request("initialize"), so a second /codex:* command hung forever — the
broker-busy hang seen in upstream openai#509.

- Bound connect (2s default) and initialize (5s default) with a new
  BROKER_HANDSHAKE_TIMEOUT_CODE ("EBROKERTIMEOUT"); both overridable via
  brokerConnectTimeoutMs / brokerInitializeTimeoutMs options. Timers are
  unref'd so they don't pin the event loop.
- withAppServer now falls back to a direct app-server on EBROKERTIMEOUT in
  addition to BROKER_BUSY_RPC_CODE / ENOENT / ECONNREFUSED.
- RED-first: spawnWedgedServer helper + broker-client-initialize-timeout test
  reproduces the hang (connect ok, no initialize answer → EBROKERTIMEOUT).

Follow-up (not in this PR): SpawnedCodexAppServerClient.initialize has the
same unbounded shape; a deeper fix would put requestTimeoutMs on
AppServerClientBase.request() and collapse the withAppServer OR-chain.

Co-Authored-By: Claude <[email protected]>
…-review critical)

Codex adversarial review caught that the handshake timeout rejects inside
CodexAppServerClient.connect() — before `client` is assigned in withAppServer —
so the `client?.transport === "broker"` guard was always false for EBROKERTIMEOUT.
The fallback clause was dead code: a wedged broker made every withAppServer-backed
command throw after 5s instead of retrying directly.

Tag initialization errors with the chosen transport in connect(), and let the
fallback match on error.transport when client is still null. Adds an end-to-end
test (wedged broker → withAppServer → direct fallback via fake-codex).

Co-Authored-By: Claude <[email protected]>
@axisrow

axisrow commented Jul 20, 2026

Copy link
Copy Markdown
Owner Author

📋 Review summary — all cycles

Cycle Reviewer Finding Verdict Resolution
1 codex Broker timeout fallback is unreachable — handshake rejects before client assigned, so client.transport is null and EBROKERTIMEOUT fallback never fires FIX Fixed in 536774d (connect tags error.transport; withAppServer matches on error.transport when client is null; e2e test RED→GREEN)
1 claude (/review) SpawnedCodexAppServerClient.initialize has the same unbounded shape SKIP Already noted in PR body as follow-up (out of scope for broker-path #29)
1 simplify (4 agents) unref timers; drop redundant connected flag; try/finally → .finally applied All applied pre-review
2 codex (no findings) approve —
2 claude (/review) (clean) approve —

Totals: 1 FIX (resolved), 1 SKIP (follow-up), 0 UNVERIFIED.
Verdict: both reviewers approve at head 536774d. Ready to merge (local mode — merge is yours to trigger).

@axisrow
axisrow merged commit b057f64 into main Jul 20, 2026
@axisrow
axisrow deleted the fix/broker-busy-hang-29 branch July 20, 2026 18:09
axisrow added a commit that referenced this pull request Jul 21, 2026
…-review critical)

Codex adversarial review caught that `error.code = REQUEST_TIMEOUT_CODE` on a
plain `new Error(...)` fails the repo's mandatory `npm run build`
(TS2339: Property 'code' does not exist on type 'Error'). CI runs this build,
so the head could not pass its shipping gate. The same pre-existing breakage
in BrokerCodexAppServerClient's handshakeError (landed in #30) is fixed too.

Cast the errors to ProtocolError (extended to include `code?: string`), the
same pattern createProtocolError already uses.

Co-Authored-By: Claude <[email protected]>
axisrow added a commit that referenced this pull request Jul 21, 2026
… hang the client (closes #31) (#32)

* fix(app-server): bound request() so a wedged spawned app-server can't hang the client (closes #31)

SpawnedCodexAppServerClient.initialize had the same unbounded handshake shape
PR #30 fixed only on the broker path: request("initialize") had no deadline,
so a spawned codex app-server that accepted stdin but never answered hung the
client forever. RED-first: a stalled-initialize fake-codex behavior reproduces
the hang (connect() never settles within 2s).

Deeper fix (as promised in #31 / upstream openai#509), not a third copy of the
bounded-handshake pattern:

- AppServerClientBase.request(method, params, options) gains an optional
  timeoutMs (+ onTimeout for transport cleanup). Optional/undefined-default, so
  all existing 2-arg call sites are unchanged. Timer is unref'd, cleared in
  handleLine/handleExit. Timeout rejects with BROKER_HANDSHAKE_TIMEOUT_CODE and
  removes the pending entry; a late answer for that id is a no-op in handleLine.
- SpawnedCodexAppServerClient.initialize now passes timeoutMs (10s default,
  spawnedInitializeTimeoutMs override), and kills the child on handshake
  failure so a timeout doesn't orphan the spawned process.
- BrokerCodexAppServerClient.initialize drops its inline Promise.race and uses
  request(timeoutMs, onTimeout=socket.destroy) — one mechanism, both transports.
- CodexAppServerClient.connect tags broker-transport init failures with
  brokerFatal; withAppServer's fallback OR-chain collapses to
  (broker-busy) || (brokerFatal).

Verified: RED→GREEN (304ms, EBROKERTIMEOUT); broker suite 11/11 green; npm test
unchanged (4 pre-existing failures).

Co-Authored-By: Claude <[email protected]>

* fix(broker): restore post-handshake broker/busy direct fallback (cycle-review critical)

Codex adversarial review caught that collapsing withAppServer's fallback onto
error.transport broke the #29 path: broker/busy (-32001) arrives AFTER a
successful handshake, while the error is created by createProtocolError with
rpcCode but NO transport (transport is tagged only on handshake failure in
connect()'s catch). So every overlapping /codex:* command failed for the
duration of a long-running broker turn instead of falling back to direct.

Restore client?.transport === "broker" for the busy branch (post-handshake,
client is assigned); keep error.brokerFatal for handshake failures (client is
null, tagged in connect). Adds spawnBusyBroker helper + an end-to-end test
that drives a successful initialize, then -32001 on the first RPC, and
verifies the operation completes via the direct fallback.

Co-Authored-By: Claude <[email protected]>

* fix(app-server): make timeout-error code assignments type-safe (cycle-review critical)

Codex adversarial review caught that `error.code = REQUEST_TIMEOUT_CODE` on a
plain `new Error(...)` fails the repo's mandatory `npm run build`
(TS2339: Property 'code' does not exist on type 'Error'). CI runs this build,
so the head could not pass its shipping gate. The same pre-existing breakage
in BrokerCodexAppServerClient's handshakeError (landed in #30) is fixed too.

Cast the errors to ProtocolError (extended to include `code?: string`), the
same pattern createProtocolError already uses.

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: axisrow <[email protected]>
Co-authored-by: Claude <[email protected]>
axisrow added a commit that referenced this pull request Jul 21, 2026
Ships accumulated fixes since fork.4:
- #29 broker-busy hang (PR #30): bound broker handshake
- #31 spawned-app-server handshake (PR #32): bound request() primitive
- openai#508 config-driven sandbox + --read-only (PR #33) + resume-sandbox fail-closed
- #34 model/effort in /codex:rescue launch line (PR #35)
- #28 turn-timeout gaps (already in fork.4 base)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant