Skip to content

Commit 14123d3

Browse files
ruan330claude
andcommitted
fix: resolve captureTurn when app-server disconnects mid-turn
Root cause ---------- `captureTurn` (plugins/codex/scripts/lib/codex.mjs) blocks on `state.completion`. Two existing resolution paths both require server cooperation: A. `turn/completed` notification from Codex app-server B. `finalAnswerSeen + scheduleInferredCompletion` (fires 250ms after an `agentMessage` with `phase === "final_answer"`) When the Codex CLI exits mid-turn — rate limit, OOM, internal error, token limit — neither fires: the only `agentMessage` had `phase == "plan"` so `finalAnswerSeen` stayed `false`, and no `turn/completed` was ever sent. `state.completion` hangs indefinitely. The node companion process itself exits cleanly via IPC EOF, bypassing `runTrackedJob`'s try/catch, leaving a zombie job record (`status: "running"`) and no final verdict. Observed three times in the wild on 2026-04-18 while running `/codex:adversarial-review` from Claude Code. In broker mode (`app-server-broker.mjs`) the bug was worse: the broker did not watch the upstream `appClient`, so when the Codex CLI died the broker kept accepting socket connections. Clients got neither `turn/completed` nor a socket EOF. Fix (three coupled changes, one semantic unit) ----------------------------------------------- 1. `captureTurn` subscribes to `client.exitPromise` as a third resolution path. If a `final_answer` agentMessage has already been captured AND no subagent work is outstanding, honor the existing inferred-success semantics — call `completeTurn(state, null, { inferred: true })` without the 250ms debounce, which is meaningless on a dead socket. Only when no terminal signal was captured do we synthesize a `failed` turn so `buildResultStatus` returns non-zero. 2. `app-server-broker.mjs` subscribes to the upstream `appClient.exitPromise` and tears down the server when the Codex CLI dies. Connected clients receive socket EOF and the watchdog in #1 resolves their `captureTurn`. Broker exits with status 1 so supervisors can restart it. 3. The broker watchdog skips when `shuttingDown` is true. Without this guard, intentional teardown paths (`broker/shutdown`, `SIGTERM`, `SIGINT`) all call `shutdown(server)` → `appClient.close()` which resolves the same `exitPromise`, re-entering `shutdown()` and racing `process.exit(0)` with `process.exit(1)`. Why the obvious small fixes were rejected ------------------------------------------ - Relax `finalAnswerSeen` to any `lifecycle === "completed"` agentMessage: changes semantics for well-behaved turns; can commit prematurely on intermediate plan-phase messages. - Only a max-turn timeout: loses `lastAgentMessage` diagnostic, fires on legitimately long turns, and doesn't catch fast disconnects. - Broker-side timeout alone: doesn't help direct-transport callers. Tests ----- Three regression tests in `tests/runtime.test.mjs` against new fake behaviors in `tests/fake-codex-fixture.mjs`: * `adversarial review resolves when app-server disconnects before turn/completed` — plan-phase agentMessage then disconnect; asserts non-zero exit + rendered output mentions the disconnect. * `adversarial review preserves final_answer when app-server exits before turn/completed` — final_answer agentMessage then disconnect; asserts exit 0 + structured findings survive. * `broker exits cleanly on broker/shutdown without tripping the upstream-exit watchdog` — sends `broker/shutdown` RPC to a non-detached broker, asserts exit 0. Baseline suite: 88 tests, 82 pass, 6 fail pre-existing (status/result caching in test state — unrelated). With this patch: 90 tests, 85 pass, 6 fail. +3 new passing tests, no regression. Upstream pointer ---------------- Why the Codex CLI can exit mid-turn without flushing `turn/completed` is orthogonal to this PR and belongs in `openai/codex`. The watchdog here makes the companion resilient regardless. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 6a5c2ba commit 14123d3

4 files changed

Lines changed: 254 additions & 2 deletions

File tree

‎plugins/codex/scripts/app-server-broker.mjs‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,16 @@ async function main() {
9999
}
100100
}
101101

102+
// Tracks whether `shutdown()` is running so the upstream-exit watchdog
103+
// below can tell intentional teardown (`broker/shutdown`, SIGTERM, SIGINT)
104+
// apart from the upstream app-server dying on its own. Both paths resolve
105+
// `appClient.exitPromise` via `appClient.close()`; without this flag the
106+
// watchdog would re-enter `shutdown()` during graceful teardown and race
107+
// `process.exit(0)` with `process.exit(1)`.
108+
let shuttingDown = false;
109+
102110
async function shutdown(server) {
111+
shuttingDown = true;
103112
for (const socket of sockets) {
104113
socket.end();
105114
}
@@ -115,6 +124,26 @@ async function main() {
115124

116125
appClient.setNotificationHandler(routeNotification);
117126

127+
// Propagate upstream codex app-server exit so connected clients see a
128+
// socket EOF instead of waiting forever for a turn/completed. Without
129+
// this, a codex CLI crash (rate limit, OOM, internal error) leaves the
130+
// broker idle-alive: the socket to each client stays open but no more
131+
// notifications ever arrive, so `captureTurn` hangs. Closing the server
132+
// triggers the client-side `exitPromise` resolution path that
133+
// `captureTurn` now uses as a watchdog.
134+
//
135+
// The `shuttingDown` guard is required — `shutdown()` itself closes
136+
// `appClient`, which resolves the same `exitPromise` this handler is
137+
// listening on. Without the guard, every graceful termination would
138+
// trigger this watchdog, re-enter `shutdown()`, and race the intentional
139+
// `process.exit(0)` with `process.exit(1)`.
140+
appClient.exitPromise.then(() => {
141+
if (shuttingDown) {
142+
return;
143+
}
144+
shutdown(server).catch(() => {}).finally(() => process.exit(1));
145+
});
146+
118147
const server = net.createServer((socket) => {
119148
sockets.add(socket);
120149
socket.setEncoding("utf8");

‎plugins/codex/scripts/lib/codex.mjs‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -575,6 +575,51 @@ async function captureTurn(client, threadId, startRequest, options = {}) {
575575
applyTurnNotification(state, message);
576576
});
577577

578+
// Resolve captureTurn when the app-server disconnects before sending a
579+
// terminal signal. The existing resolution paths both require server
580+
// cooperation: (a) a `turn/completed` notification, or (b) a final_answer
581+
// phase agentMessage that arms `scheduleInferredCompletion`. Neither fires
582+
// when the Codex CLI exits mid-turn (rate limit, OOM, internal error) —
583+
// observed in 2026-04-18 adversarial-review runs where the CLI emitted a
584+
// plan-phase agentMessage, ran tool calls, then exited cleanly via IPC EOF.
585+
// Without this watchdog, `state.completion` hangs, the companion script
586+
// exits via its own IPC close handling, and runTrackedJob's try/catch is
587+
// never reached — leaving a zombie job and no final verdict.
588+
//
589+
// Success preservation: if a `final_answer` agentMessage already arrived
590+
// AND no subagent work is outstanding, the turn is authoritatively
591+
// complete — the disconnect is just Codex closing the door behind a
592+
// successful response. Mirror `scheduleInferredCompletion`'s success
593+
// semantics (same flag, same pending-work check) without waiting for
594+
// its 250ms debounce, which can't help a dead socket. Only when no
595+
// terminal signal was captured do we synthesize a `failed` turn.
596+
client.exitPromise.then(() => {
597+
if (state.completed) {
598+
return;
599+
}
600+
const hasFinalAnswer =
601+
state.finalAnswerSeen &&
602+
state.pendingCollaborations.size === 0 &&
603+
state.activeSubagentTurns.size === 0;
604+
if (hasFinalAnswer) {
605+
completeTurn(state, null, { inferred: true });
606+
return;
607+
}
608+
state.error = state.error ?? client.exitError ?? {
609+
message: "codex app-server disconnected before the turn completed."
610+
};
611+
emitProgress(
612+
state.onProgress,
613+
"App-server disconnected before turn/completed; marking turn as failed.",
614+
"failed"
615+
);
616+
completeTurn(
617+
state,
618+
{ id: state.turnId ?? "disconnected-turn", status: "failed" },
619+
{ inferred: true }
620+
);
621+
});
622+
578623
try {
579624
const response = await startRequest();
580625
options.onResponse?.(response, state);

‎tests/fake-codex-fixture.mjs‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -533,6 +533,52 @@ rl.on("line", (line) => {
533533
interruptibleTurns.set(turnId, { threadId: thread.id, timer });
534534
} else if (BEHAVIOR === "slow-task") {
535535
emitTurnCompletedLater(thread.id, turnId, items, 400);
536+
} else if (BEHAVIOR === "disconnect-before-completion") {
537+
// Reproduces the real-world Codex CLI exit path: plan-phase
538+
// agentMessage (phase !== "final_answer") is emitted, then the
539+
// server disconnects without ever sending turn/completed or a
540+
// final_answer agentMessage. Neither existing captureTurn
541+
// resolution path (turn/completed notification OR finalAnswerSeen
542+
// inferred timer) fires → captureTurn hangs indefinitely.
543+
send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } });
544+
send({
545+
method: "item/completed",
546+
params: {
547+
threadId: thread.id,
548+
turnId,
549+
item: {
550+
type: "agentMessage",
551+
id: "msg_" + turnId,
552+
text: "I'm going to inspect the target and return a verdict.",
553+
phase: "plan"
554+
}
555+
}
556+
});
557+
// Flush stdout then disconnect cleanly, matching observed upstream
558+
// exit mode (exit 0 via IPC EOF; no protocol-level error).
559+
process.stdout.write("", () => process.exit(0));
560+
} else if (BEHAVIOR === "final-answer-then-exit") {
561+
// The captureTurn watchdog must distinguish "disconnected with
562+
// authoritative completion signal" (success) from "disconnected
563+
// with no terminal signal" (failure). Emit a completed
564+
// final_answer agentMessage and then exit before sending
565+
// turn/completed — this is the success-path race the watchdog
566+
// must preserve instead of demoting to failed.
567+
send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } });
568+
send({
569+
method: "item/completed",
570+
params: {
571+
threadId: thread.id,
572+
turnId,
573+
item: {
574+
type: "agentMessage",
575+
id: "msg_" + turnId,
576+
text: payload,
577+
phase: "final_answer"
578+
}
579+
}
580+
});
581+
process.stdout.write("", () => process.exit(0));
536582
} else {
537583
emitTurnCompleted(thread.id, turnId, items);
538584
}

‎tests/runtime.test.mjs‎

Lines changed: 134 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,20 @@
11
import fs from "node:fs";
2+
import os from "node:os";
23
import path from "node:path";
34
import test from "node:test";
45
import assert from "node:assert/strict";
5-
import { spawn } from "node:child_process";
6+
import { spawn, spawnSync } from "node:child_process";
67
import { fileURLToPath } from "node:url";
78

89
import { buildEnv, installFakeCodex } from "./fake-codex-fixture.mjs";
910
import { initGitRepo, makeTempDir, run } from "./helpers.mjs";
10-
import { loadBrokerSession, saveBrokerSession } from "../plugins/codex/scripts/lib/broker-lifecycle.mjs";
11+
import {
12+
loadBrokerSession,
13+
saveBrokerSession,
14+
sendBrokerShutdown,
15+
waitForBrokerEndpoint
16+
} from "../plugins/codex/scripts/lib/broker-lifecycle.mjs";
17+
import { createBrokerEndpoint } from "../plugins/codex/scripts/lib/broker-endpoint.mjs";
1118
import { resolveStateDir } from "../plugins/codex/scripts/lib/state.mjs";
1219

1320
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
@@ -300,6 +307,131 @@ test("adversarial review asks Codex to inspect larger diffs itself", () => {
300307
assert.doesNotMatch(state.lastTurnStart.prompt, /PROMPT_SELF_COLLECT_[ABC]/);
301308
});
302309

310+
test("adversarial review resolves when app-server disconnects before turn/completed", () => {
311+
// Regression: Codex CLI can exit cleanly (token limit / rate limit / internal
312+
// error) after emitting a non-final_answer agentMessage but before sending
313+
// turn/completed. Neither existing captureTurn resolution path fires — the
314+
// Promise hangs and the companion script exits via IPC EOF without writing a
315+
// final verdict, leaving a zombie job record.
316+
//
317+
// Observed in the wild on 2026-04-18 (MeeePtt favorites-fix commit). A passing
318+
// test asserts captureTurn can handle "app-server closed unexpectedly".
319+
const repo = makeTempDir();
320+
const binDir = makeTempDir();
321+
installFakeCodex(binDir, "disconnect-before-completion");
322+
initGitRepo(repo);
323+
fs.writeFileSync(path.join(repo, "README.md"), "hello\n");
324+
run("git", ["add", "README.md"], { cwd: repo });
325+
run("git", ["commit", "-m", "init"], { cwd: repo });
326+
fs.writeFileSync(path.join(repo, "README.md"), "hello again\n");
327+
328+
// Explicit 10s timeout guards against the hang on unfixed code — without
329+
// this, the test would block until node --test's outer limit and report
330+
// "hung" instead of "disconnected".
331+
const result = spawnSync("node", [SCRIPT, "adversarial-review"], {
332+
cwd: repo,
333+
env: buildEnv(binDir),
334+
encoding: "utf8",
335+
timeout: 10_000
336+
});
337+
338+
assert.notEqual(result.signal, "SIGTERM",
339+
`adversarial-review hung after app-server disconnect (stdout=${result.stdout}, stderr=${result.stderr})`);
340+
assert.notEqual(result.status, 0,
341+
"adversarial-review must surface disconnect as non-zero exit");
342+
assert.match(
343+
`${result.stdout}\n${result.stderr}`,
344+
/disconnect|closed|unexpectedly/i,
345+
"rendered output should mention the disconnect"
346+
);
347+
});
348+
349+
test("adversarial review preserves final_answer when app-server exits before turn/completed", () => {
350+
// Companion to the disconnect-before-completion test: when the
351+
// app-server emits a completed final_answer agentMessage and then exits
352+
// cleanly (no turn/completed notification), the captureTurn watchdog
353+
// must NOT demote the successful turn to failed. `finalAnswerSeen` is
354+
// the authoritative "work finished" signal — the disconnect is just
355+
// Codex closing the door, not a failure.
356+
const repo = makeTempDir();
357+
const binDir = makeTempDir();
358+
installFakeCodex(binDir, "final-answer-then-exit");
359+
initGitRepo(repo);
360+
fs.mkdirSync(path.join(repo, "src"));
361+
fs.writeFileSync(path.join(repo, "src", "app.js"), "export const value = items[0];\n");
362+
run("git", ["add", "src/app.js"], { cwd: repo });
363+
run("git", ["commit", "-m", "init"], { cwd: repo });
364+
fs.writeFileSync(path.join(repo, "src", "app.js"), "export const value = items[0].id;\n");
365+
366+
const result = spawnSync("node", [SCRIPT, "adversarial-review"], {
367+
cwd: repo,
368+
env: buildEnv(binDir),
369+
encoding: "utf8",
370+
timeout: 10_000
371+
});
372+
373+
assert.notEqual(result.signal, "SIGTERM",
374+
`adversarial-review hung after final_answer + exit (stderr=${result.stderr})`);
375+
assert.equal(result.status, 0,
376+
`final_answer before disconnect must stay exit 0 (stdout=${result.stdout}, stderr=${result.stderr})`);
377+
assert.match(
378+
result.stdout,
379+
/Missing empty-state guard/,
380+
"structured findings from the final_answer payload must survive the disconnect"
381+
);
382+
});
383+
384+
test("broker exits cleanly on broker/shutdown without tripping the upstream-exit watchdog", async () => {
385+
// Guards against a regression introduced while fixing the
386+
// "disconnect-before-completion" hang: the upstream-exit watchdog added
387+
// to app-server-broker.mjs listens on appClient.exitPromise, which is
388+
// resolved both by Codex CLI crashes AND by shutdown()'s own
389+
// appClient.close(). Without an `isShuttingDown` guard the watchdog
390+
// re-enters shutdown() during broker/shutdown (or SIGTERM/SIGINT) and
391+
// races the intentional process.exit(0) with process.exit(1).
392+
const binDir = makeTempDir();
393+
installFakeCodex(binDir);
394+
const cwd = makeTempDir();
395+
396+
// Spawn the broker directly (not detached) so the test can observe its
397+
// real exit code — the shared-broker lifecycle normally detaches the
398+
// process which hides exit code from supervisors.
399+
const brokerScript = path.join(PLUGIN_ROOT, "scripts", "app-server-broker.mjs");
400+
const sessionDir = fs.mkdtempSync(path.join(os.tmpdir(), "broker-test-"));
401+
const endpoint = createBrokerEndpoint(sessionDir);
402+
const pidFile = path.join(sessionDir, "broker.pid");
403+
404+
const broker = spawn(
405+
"node",
406+
[brokerScript, "serve", "--endpoint", endpoint, "--cwd", cwd, "--pid-file", pidFile],
407+
{ env: buildEnv(binDir), stdio: ["ignore", "pipe", "pipe"] }
408+
);
409+
410+
const exited = new Promise((resolve) => {
411+
broker.on("exit", (code, signal) => resolve({ code, signal }));
412+
});
413+
414+
try {
415+
const ready = await waitForBrokerEndpoint(endpoint, 5_000);
416+
assert.ok(ready, "broker should become reachable");
417+
418+
await sendBrokerShutdown(endpoint);
419+
const { code, signal } = await exited;
420+
421+
assert.equal(signal, null, "broker should exit without a signal");
422+
assert.equal(
423+
code,
424+
0,
425+
"broker/shutdown must return exit 0 — non-zero indicates the upstream-exit watchdog tripped during graceful teardown"
426+
);
427+
} finally {
428+
if (broker.exitCode === null && broker.signalCode === null) {
429+
broker.kill("SIGKILL");
430+
}
431+
fs.rmSync(sessionDir, { recursive: true, force: true });
432+
}
433+
});
434+
303435
test("review includes reasoning output when the app server returns it", () => {
304436
const repo = makeTempDir();
305437
const binDir = makeTempDir();

0 commit comments

Comments
 (0)