Skip to content

Terminate broker when its app-server child exits (fixes wedged/zombie broker hangs) - #453

Open
lselva123 wants to merge 1 commit into
openai:mainfrom
lselva123:fix/broker-self-terminate-on-child-exit
Open

lselva123 wants to merge 1 commit into
openai:mainfrom
lselva123:fix/broker-self-terminate-on-child-exit

Conversation

@lselva123

Copy link
Copy Markdown

Problem

The app-server broker (scripts/app-server-broker.mjs) spawns exactly one codex app-server child at startup and proxies every request to it. If that child exits at any point, the broker becomes a zombie that accepts connections but can never serve a request again:

  • AppServerClientBase.handleExit() rejects in-flight requests but does not set closed, and nothing respawns the child.
  • The next request() therefore passes the if (this.closed) guard, registers a pending promise, and writes to the dead child's stdin — the promise never resolves.
  • So await appClient.request(...) in the broker's data handler hangs forever, the client sees its submitted turn/start / review/start sit at "starting" indefinitely, and it is eventually reported lost.
  • Because the broker's listening socket is still up, ensureBrokerSession() (whose readiness check is a bare socket connect) keeps reusing the same wedged broker, so every subsequent Codex review/task from that workspace wedges the same way until the broker is killed manually.

We observed this repeatedly on a busy multi-session machine: brokers whose app-server child had died (e.g. reaped by an unrelated process sweeper, OOM, or a crash) lingered for many hours with zero children, and every Codex review routed to them hung.

Fix

Register a handler on appClient.exitPromise that shuts the broker down and exits (code 1) as soon as the app-server child is gone. This removes the stale unix socket + pid-file, so the next ensureBrokerSession() sees a dead endpoint and spawns a fresh, working broker instead of binding the zombie. A terminating flag guards against racing an intentional shutdown (broker/shutdown, SIGTERM/SIGINT).

Net effect: a broker whose backing app-server dies now fails closed and self-heals on next use, instead of silently swallowing every future request.

Testing

  • node --check passes.
  • Spawned a broker from the patched script, killed its codex app-server child, and confirmed the broker process exits and its socket is removed (vs. the unpatched build, where the broker stays up childless and the next request hangs).
  • Verified the guard prevents a double-shutdown on the normal broker/shutdown path.

Small, self-contained change; no interface or behavior change on the healthy path.

The broker proxies every request to the single `codex app-server` child it
spawns at startup. If that child exits, appClient can no longer serve any
request: request() writes to a closed stdin and its promise never resolves, so
the caller hangs indefinitely (a submitted turn/task stays at "starting" and is
never answered). The broker itself stays alive as a zombie because its listening
socket still accepts connections, so ensureBrokerSession() keeps reusing it and
every subsequent request wedges the same way.

Register an exit handler on appClient.exitPromise that shuts the broker down and
exits once the child is gone, so the stale socket/pid-file are removed and the
next connection spawns a fresh, working broker. A `terminating` guard keeps the
handler from racing an intentional shutdown (broker/shutdown, SIGTERM/SIGINT).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@lselva123
lselva123 requested a review from a team July 8, 2026 08:08

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e7300ed9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +104 to +105
if (terminating) {
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Return the in-flight shutdown instead of exiting early

If a second shutdown trigger arrives while the first cleanup is still awaiting appClient.close() or server.close(), this new early return makes that caller continue directly to its process.exit(...) path and abort the original cleanup before it can unlink the broker socket/pid file. This can happen with two broker/shutdown requests, or a SIGTERM during app-server-exit cleanup, leaving stale broker state or an orphaned child; keep and return a shared shutdown promise rather than treating an in-progress shutdown as complete.

Useful? React with 👍 / 👎.

axisrow added a commit to axisrow/codex-plugin-cc that referenced this pull request Jul 18, 2026
…nai#453)

The broker spawns one codex app-server child and proxies every request to
it. If that child dies, AppServerClientBase.handleExit() resolved
exitPromise but never set this.closed, so the next request() passed the
closed-guard, wrote to a dead stdin, and hung forever. The broker stayed
up as a zombie (listening socket still accepted connections), and
ensureBrokerSession() kept reusing it — every task/review from that
workspace wedged.

Fix: register an appClient.exitPromise handler that runs shutdown(server)
and process.exit(1) when the child is gone, removing the stale socket +
pid file so the next connect spawns a fresh broker. A `terminating` flag
guards against racing the intentional shutdown paths.

Also (defense-in-depth, same root cause at the class level):
- request()/notify() now treat exitResolved like closed, so a post-exit
  call can't register a pending promise on a dead transport.
- shutdown()'s server.close() wait is now bounded (1s grace, then force
  destroy sockets), so a peer that never closes can't hang the exit.

Race A (signal arriving mid-shutdown could theoretically preempt the
unlink) was probed with a dedicated regression test (10 iterations,
child-exit racing SIGTERM). It did not reproduce — the shared shutdown()
sets shuttingDown synchronously before any await, so both paths funnel
through one cleanup. No centralized coordinator added without a failing
test (TDD: no architecture without red).

Tested: tests/broker-exit.test.mjs (red before fix, green after,
mutation-verified). Upstream openai#453 ships no test; this fork adds its own.

Co-Authored-By: Claude <[email protected]>
axisrow added a commit to axisrow/codex-plugin-cc that referenced this pull request Jul 18, 2026
axisrow added a commit to axisrow/codex-plugin-cc that referenced this pull request Jul 18, 2026
The "child exit races with SIGTERM" test (added with the openai#453 port) was a
probe for a theoretical Race A (signal arriving mid-shutdown preempts the
socket unlink). It was stable in isolation but flaked under full-suite
load — the 25ms/50ms timing window shifts when other tests compete for
CPU.

Per the TDD rule that guided openai#453 ("no architecture without a failing
test"), Race A was never proven as a stable bug: 30 isolated iterations
never leaked. The centralized coordinator was correctly NOT added. This
probe test adds CI flakiness without guarding a real defect, so remove
it. The core openai#453 regression test (broker exits + cleans up when the
app-server child dies) stays — that one is deterministic and proven.

Co-Authored-By: Claude <[email protected]>
axisrow added a commit to axisrow/codex-plugin-cc that referenced this pull request Jul 18, 2026
Covers everything merged since fork.2: gpt-5.6 aliases (sol/terra/luna),
natural-language model/effort recognition in the command .md files, the openai#453
port (broker self-terminate on app-server child exit), the openai#522 parity fix
(review no longer rejects positional focus text), the openai#497 port (dead-PID
reconciliation, --cwd workspace pin, cancellation safety), and the
BrokerController refactor (deterministic in-process broker tests).

Base stays 1.0.6 — that is what upstream main carries and the last real
release (v1.0.6). The 1.0.7 that had crept into package.json came from the
openai#497 port (the PR author's aspirational number), not from any release.

Co-Authored-By: Claude <[email protected]>
ApexAiOfficial added a commit to ApexAiOfficial/codex-plugin-cc that referenced this pull request Sep 25, 2026
Audited against open upstream reports on openai/codex-plugin-cc; each defect
was reproduced or verified against this fork and has a regression test in
tests/substrate.test.mjs (dedicated fake app-server).

- openai#302 unbounded waits: every RPC has a bounded wall-clock timeout (120s
  default, command/exec aware, CODEX_COMPANION_RPC_TIMEOUT_MS, 0 disables),
  and requests on a dead connection fail immediately. The turn watchdog does
  not use a silence timeout (legitimate turns are silent for long periods):
  it fails fast when the connection closes, and after quiet periods asks the
  app-server via thread/read whether the thread is still active, recovering a
  turn whose completion event was lost and failing only an app-server that
  stops answering.
- openai#453 zombie broker: the broker exits when its app-server child dies, so the
  next caller starts a healthy broker instead of wedging.
- openai#706/openai#707 retained subscriptions: the broker tracks per-socket thread
  ownership (subagents inherit their parent's owners) and sends
  thread/unsubscribe when the last owner disconnects; requests for a thread
  wait (bounded) for its in-flight unsubscribe. Simpler than openai#707 because this
  broker already serializes active requests and streams.
- openai#762 + openai#768 together: broker acquisition is serialized; a broker whose
  process is alive is never torn down or killed because it missed a probe
  (the caller uses a private app-server); metadata is cleared only when the
  process is provably gone; only a just-spawned broker that never became
  ready is killed, identity-checked. SessionEnd's kill is identity-checked.
- openai#574 RC3: non-retryable errors fail a turn even when completion is
  inferred. The upstream fix (any error fails) is wrong: error notifications
  carry willRetry, and retried turns can succeed.
- openai#775: fileChange start events without a change list no longer throw.
- openai#740 reproduced with real Codex 0.144.1: live thread/resume ignores the
  requested sandbox in both directions (write-capable threads stayed writable
  after a read-only resume), and a per-turn override persists to later turns.
  Every turn now sends an explicit sandboxPolicy.

Also fixes a same-process withFileLockAsync bug (a second async holder
treated the first as stale) and makes the test harness stop brokers when a
test process is killed by a signal.

Validation: npm test 156/156, tsc clean, no stray processes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant