Repository navigation
Conversation
The broker proxies every request to the single `codex app-server` child it spawns at startup. If that child exits, appClient can no longer serve any request: request() writes to a closed stdin and its promise never resolves, so the caller hangs indefinitely (a submitted turn/task stays at "starting" and is never answered). The broker itself stays alive as a zombie because its listening socket still accepts connections, so ensureBrokerSession() keeps reusing it and every subsequent request wedges the same way. Register an exit handler on appClient.exitPromise that shuts the broker down and exits once the child is gone, so the stale socket/pid-file are removed and the next connection spawns a fresh, working broker. A `terminating` guard keeps the handler from racing an intentional shutdown (broker/shutdown, SIGTERM/SIGINT). Co-Authored-By: Claude Opus 4.8 <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e7300ed9c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (terminating) { | ||
| return; |
There was a problem hiding this comment.
Return the in-flight shutdown instead of exiting early
If a second shutdown trigger arrives while the first cleanup is still awaiting appClient.close() or server.close(), this new early return makes that caller continue directly to its process.exit(...) path and abort the original cleanup before it can unlink the broker socket/pid file. This can happen with two broker/shutdown requests, or a SIGTERM during app-server-exit cleanup, leaving stale broker state or an orphaned child; keep and return a shared shutdown promise rather than treating an in-progress shutdown as complete.
Useful? React with 👍 / 👎.
…nai#453) The broker spawns one codex app-server child and proxies every request to it. If that child dies, AppServerClientBase.handleExit() resolved exitPromise but never set this.closed, so the next request() passed the closed-guard, wrote to a dead stdin, and hung forever. The broker stayed up as a zombie (listening socket still accepted connections), and ensureBrokerSession() kept reusing it — every task/review from that workspace wedged. Fix: register an appClient.exitPromise handler that runs shutdown(server) and process.exit(1) when the child is gone, removing the stale socket + pid file so the next connect spawns a fresh broker. A `terminating` flag guards against racing the intentional shutdown paths. Also (defense-in-depth, same root cause at the class level): - request()/notify() now treat exitResolved like closed, so a post-exit call can't register a pending promise on a dead transport. - shutdown()'s server.close() wait is now bounded (1s grace, then force destroy sockets), so a peer that never closes can't hang the exit. Race A (signal arriving mid-shutdown could theoretically preempt the unlink) was probed with a dedicated regression test (10 iterations, child-exit racing SIGTERM). It did not reproduce — the shared shutdown() sets shuttingDown synchronously before any await, so both paths funnel through one cleanup. No centralized coordinator added without a failing test (TDD: no architecture without red). Tested: tests/broker-exit.test.mjs (red before fix, green after, mutation-verified). Upstream openai#453 ships no test; this fork adds its own. Co-Authored-By: Claude <[email protected]>
The "child exit races with SIGTERM" test (added with the openai#453 port) was a probe for a theoretical Race A (signal arriving mid-shutdown preempts the socket unlink). It was stable in isolation but flaked under full-suite load — the 25ms/50ms timing window shifts when other tests compete for CPU. Per the TDD rule that guided openai#453 ("no architecture without a failing test"), Race A was never proven as a stable bug: 30 isolated iterations never leaked. The centralized coordinator was correctly NOT added. This probe test adds CI flakiness without guarding a real defect, so remove it. The core openai#453 regression test (broker exits + cleans up when the app-server child dies) stays — that one is deterministic and proven. Co-Authored-By: Claude <[email protected]>
Covers everything merged since fork.2: gpt-5.6 aliases (sol/terra/luna), natural-language model/effort recognition in the command .md files, the openai#453 port (broker self-terminate on app-server child exit), the openai#522 parity fix (review no longer rejects positional focus text), the openai#497 port (dead-PID reconciliation, --cwd workspace pin, cancellation safety), and the BrokerController refactor (deterministic in-process broker tests). Base stays 1.0.6 — that is what upstream main carries and the last real release (v1.0.6). The 1.0.7 that had crept into package.json came from the openai#497 port (the PR author's aspirational number), not from any release. Co-Authored-By: Claude <[email protected]>
Audited against open upstream reports on openai/codex-plugin-cc; each defect was reproduced or verified against this fork and has a regression test in tests/substrate.test.mjs (dedicated fake app-server). - openai#302 unbounded waits: every RPC has a bounded wall-clock timeout (120s default, command/exec aware, CODEX_COMPANION_RPC_TIMEOUT_MS, 0 disables), and requests on a dead connection fail immediately. The turn watchdog does not use a silence timeout (legitimate turns are silent for long periods): it fails fast when the connection closes, and after quiet periods asks the app-server via thread/read whether the thread is still active, recovering a turn whose completion event was lost and failing only an app-server that stops answering. - openai#453 zombie broker: the broker exits when its app-server child dies, so the next caller starts a healthy broker instead of wedging. - openai#706/openai#707 retained subscriptions: the broker tracks per-socket thread ownership (subagents inherit their parent's owners) and sends thread/unsubscribe when the last owner disconnects; requests for a thread wait (bounded) for its in-flight unsubscribe. Simpler than openai#707 because this broker already serializes active requests and streams. - openai#762 + openai#768 together: broker acquisition is serialized; a broker whose process is alive is never torn down or killed because it missed a probe (the caller uses a private app-server); metadata is cleared only when the process is provably gone; only a just-spawned broker that never became ready is killed, identity-checked. SessionEnd's kill is identity-checked. - openai#574 RC3: non-retryable errors fail a turn even when completion is inferred. The upstream fix (any error fails) is wrong: error notifications carry willRetry, and retried turns can succeed. - openai#775: fileChange start events without a change list no longer throw. - openai#740 reproduced with real Codex 0.144.1: live thread/resume ignores the requested sandbox in both directions (write-capable threads stayed writable after a read-only resume), and a per-turn override persists to later turns. Every turn now sends an explicit sandboxPolicy. Also fixes a same-process withFileLockAsync bug (a second async holder treated the first as stale) and makes the test harness stop brokers when a test process is killed by a signal. Validation: npm test 156/156, tsc clean, no stray processes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Problem
The app-server broker (
scripts/app-server-broker.mjs) spawns exactly onecodex app-serverchild at startup and proxies every request to it. If that child exits at any point, the broker becomes a zombie that accepts connections but can never serve a request again:AppServerClientBase.handleExit()rejects in-flight requests but does not setclosed, and nothing respawns the child.request()therefore passes theif (this.closed)guard, registers a pending promise, and writes to the dead child's stdin — the promise never resolves.await appClient.request(...)in the broker's data handler hangs forever, the client sees its submittedturn/start/review/startsit at "starting" indefinitely, and it is eventually reported lost.ensureBrokerSession()(whose readiness check is a bare socket connect) keeps reusing the same wedged broker, so every subsequent Codex review/task from that workspace wedges the same way until the broker is killed manually.We observed this repeatedly on a busy multi-session machine: brokers whose app-server child had died (e.g. reaped by an unrelated process sweeper, OOM, or a crash) lingered for many hours with zero children, and every Codex review routed to them hung.
Fix
Register a handler on
appClient.exitPromisethat shuts the broker down and exits (code 1) as soon as the app-server child is gone. This removes the stale unix socket + pid-file, so the nextensureBrokerSession()sees a dead endpoint and spawns a fresh, working broker instead of binding the zombie. Aterminatingflag guards against racing an intentional shutdown (broker/shutdown,SIGTERM/SIGINT).Net effect: a broker whose backing app-server dies now fails closed and self-heals on next use, instead of silently swallowing every future request.
Testing
node --checkpasses.codex app-serverchild, and confirmed the broker process exits and its socket is removed (vs. the unpatched build, where the broker stays up childless and the next request hangs).broker/shutdownpath.Small, self-contained change; no interface or behavior change on the healthy path.