Skip to content

Commit cd6cb84

Browse files
axisrowclaude
andcommitted
fix: split the turn budgets and stop opting out of liveness deltas
Two findings from #43, both consequences of the idle-timeout redesign in #41. They share the same subsystem and the same tests, so they land together. Finding 1 — a foreground turn could outlive the host's SIGKILL. At base, FOREGROUND_TURN_TIMEOUT_MS (110s) was a non-resettable race against the whole turn, so a foreground turn was always cut before Claude Code's ~120s Bash SIGKILL and captureTurn's catch reached interruptAppServerTurn in time. #41 made that deadline resettable, so a turn that keeps producing events now sails past 110s to the host kill: the catch never runs, no turn/interrupt is sent, and the broker keeps executing a possibly write-capable turn nothing can reach. Three of the four links in that chain predate #41 (see #42); the reachability is what #41 introduced. Fix: split the two budgets instead of collapsing them. - The resettable idle budget is unchanged (110s FG / 600s BG). - The hard wall-clock ceiling becomes an option instead of the fixed HARD_WALL_CLOCK_CEILING_MS constant. The companion resolves it per host: foreground -> FOREGROUND_TURN_TIMEOUT_MS (110s, below the Bash kill), background -> 45 min (no external ceiling to collide with). - CODEX_TURN_HARD_CEILING_MS overrides it, for tests and for hosts whose external kill differs. No CLI flag: the ceiling is a property of the host, not something a caller should tune per run. - hardCeilingMs threads alongside turnTimeoutMs through buildTaskRequest / executeTaskRun / executeReviewRun -> runAppServerTurn / runAppServerReview -> captureTurn. The existing catch already interrupts on any deadline rejection, so it is untouched. Foreground at defaults is behaviourally back to the pre-#41 ceiling, while a lowered idle budget still catches a genuine stall faster. Finding 2 — the liveness signal for long reasoning was opted out. Of the nine delta branches added in #41, only three could ever fire. item/agentMessage/delta, item/reasoning/summaryTextDelta and item/reasoning/textDelta were listed in DEFAULT_CAPABILITIES.optOutNotificationMethods, so the server never sent them. That is precisely the gpt-5.6-sol / effort=xhigh case #40 was filed for: during a long reasoning stretch those deltas are the ONLY liveness signal — no item boundary arrives for minutes — so the idle deadline fired and killed a turn that was demonstrably alive. Fix: drop those three from the opt-out list. item/reasoning/summaryPartAdded stays opted out — it is a boundary marker, not a delta. Also removed the command/exec/outputDelta and process/outputDelta cases: their params carry only a connection-scoped processId / processHandle and no threadId (verified against the generated protocol types), so belongsToTurn can never route them and the cases were dead code. item/fileChange/outputDelta is kept despite being deprecated upstream — an older app-server may still emit it, and treating it as activity costs nothing. Tests (TDD-first, each confirmed red before / green after): - long-progress-hits-wall-clock-ceiling fixture: streams an outputDelta every 500ms for ~20s, so the idle budget is reset continuously and only the ceiling can stop the turn. * with CODEX_TURN_HARD_CEILING_MS=3500 the run is cut at ~5s citing the ceiling, the job is failed, and the fixture records lastInterrupt. Red pre-fix: the turn ran to completion (~21s). * the same fixture through the foreground path with the ceiling lowered must also be cut and interrupted — red pre-fix for the same reason. - reasoning-delta-idle-ok fixture: streams reasoning summary deltas across the idle budget. The turn must complete, and the handshake capabilities recorded by the fixture must no longer opt out of the three methods, while still opting out of summaryPartAdded. Red pre-fix on the opt-out assertion. npm test: 184 tests, 184 pass, 0 fail. npm run build passes. Version metadata untouched at 1.0.6-fork.6. Closes #43 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8
1 parent ed9b4e3 commit cd6cb84

5 files changed

Lines changed: 296 additions & 20 deletions

File tree

‎plugins/codex/scripts/codex-companion.mjs‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,17 @@ const TASK_WORKER_RECORD_WAIT_TIMEOUT_MS = 1000;
7676
const FOREGROUND_TURN_TIMEOUT_MS = 110000;
7777
// Background runs have no external Bash ceiling — give them the full default budget.
7878
const DEFAULT_TURN_TIMEOUT_MS = 600000;
79+
// Hard wall-clock ceiling: a second, non-resettable failsafe under the idle
80+
// budget above. The idle deadline is reset by every notification, so a turn
81+
// that keeps producing events can run indefinitely — correct for background
82+
// work, but fatal in the foreground, where Claude Code's Bash tool SIGKILLs
83+
// node at ~120s. A foreground turn that sails past that is killed before
84+
// captureTurn's catch can send turn/interrupt, orphaning a live, possibly
85+
// write-capable turn on the broker. So the foreground ceiling stays below the
86+
// host kill (matching FOREGROUND_TURN_TIMEOUT_MS), while background keeps a
87+
// generous ceiling that only catches a runaway turn resetting its own timer
88+
// forever. Closes #43.
89+
const DEFAULT_TURN_HARD_CEILING_MS = 45 * 60 * 1000;
7990
const REASONING_EFFORTS = ["none", "minimal", "low", "medium", "high", "xhigh", "max", "ultra"];
8091
const VALID_REASONING_EFFORTS = new Set(REASONING_EFFORTS);
8192
const MODEL_ALIASES = new Map([
@@ -422,6 +433,7 @@ async function executeReviewRun(request) {
422433
model: request.model,
423434
effort: request.effort,
424435
turnTimeoutMs: request.turnTimeoutMs,
436+
hardCeilingMs: request.hardCeilingMs,
425437
onProgress: request.onProgress
426438
});
427439
const payload = {
@@ -471,6 +483,7 @@ async function executeReviewRun(request) {
471483
onProgress: request.onProgress,
472484
persistThread: true,
473485
turnTimeoutMs: request.turnTimeoutMs,
486+
hardCeilingMs: request.hardCeilingMs,
474487
threadName: `Codex Companion Review: ${context.target.label}`.slice(0, 80)
475488
});
476489
const parsed = parseStructuredOutput(result.finalMessage, {
@@ -551,6 +564,7 @@ async function executeTaskRun(request) {
551564
onProgress: request.onProgress,
552565
persistThread: true,
553566
turnTimeoutMs: request.turnTimeoutMs,
567+
hardCeilingMs: request.hardCeilingMs,
554568
threadName: resumeThreadId ? null : buildPersistentTaskThreadName(request.prompt || DEFAULT_CONTINUE_PROMPT)
555569
});
556570

@@ -662,7 +676,7 @@ function buildTaskJob(workspaceRoot, taskMetadata, write) {
662676
});
663677
}
664678

665-
function buildTaskRequest({ cwd, model, effort, prompt, write, readOnly, resumeLast, jobId, turnTimeoutMs }) {
679+
function buildTaskRequest({ cwd, model, effort, prompt, write, readOnly, resumeLast, jobId, turnTimeoutMs, hardCeilingMs }) {
666680
return {
667681
cwd,
668682
model,
@@ -672,7 +686,8 @@ function buildTaskRequest({ cwd, model, effort, prompt, write, readOnly, resumeL
672686
readOnly,
673687
resumeLast,
674688
jobId,
675-
turnTimeoutMs
689+
turnTimeoutMs,
690+
hardCeilingMs
676691
};
677692
}
678693

@@ -726,6 +741,19 @@ function resolveTurnTimeoutMsFromOptions(options) {
726741
return options.background ? DEFAULT_TURN_TIMEOUT_MS : FOREGROUND_TURN_TIMEOUT_MS;
727742
}
728743

744+
// Resolve the hard wall-clock ceiling (see DEFAULT_TURN_HARD_CEILING_MS).
745+
// Precedence mirrors resolveTurnTimeoutMsFromOptions: env override first (for
746+
// tests and non-Claude hosts with a different external ceiling), then the
747+
// foreground/background default. There is no CLI flag — the ceiling is a
748+
// safety property of the host, not something a caller should tune per-run.
749+
function resolveTurnHardCeilingMsFromOptions(options) {
750+
const fromEnv = Number(process.env.CODEX_TURN_HARD_CEILING_MS);
751+
if (Number.isFinite(fromEnv) && fromEnv > 0) {
752+
return fromEnv;
753+
}
754+
return options.background ? DEFAULT_TURN_HARD_CEILING_MS : FOREGROUND_TURN_TIMEOUT_MS;
755+
}
756+
729757
function readTaskPrompt(cwd, options, positionals) {
730758
if (options["prompt-file"]) {
731759
return fs.readFileSync(path.resolve(cwd, options["prompt-file"]), "utf8");
@@ -837,6 +865,7 @@ async function handleReviewCommand(argv, config) {
837865
focusText,
838866
reviewName: config.reviewName,
839867
turnTimeoutMs: resolveTurnTimeoutMsFromOptions(options),
868+
hardCeilingMs: resolveTurnHardCeilingMsFromOptions(options),
840869
onProgress: progress
841870
}),
842871
{ json: options.json }
@@ -895,7 +924,8 @@ async function handleTask(argv) {
895924
readOnly,
896925
resumeLast,
897926
jobId: job.id,
898-
turnTimeoutMs: resolveTurnTimeoutMsFromOptions(options)
927+
turnTimeoutMs: resolveTurnTimeoutMsFromOptions(options),
928+
hardCeilingMs: resolveTurnHardCeilingMsFromOptions(options)
899929
});
900930
const { payload } = enqueueBackgroundTask(cwd, job, request);
901931
outputCommandResult(payload, renderQueuedTaskLaunch(payload), options.json);
@@ -916,6 +946,7 @@ async function handleTask(argv) {
916946
resumeLast,
917947
jobId: job.id,
918948
turnTimeoutMs: resolveTurnTimeoutMsFromOptions(options),
949+
hardCeilingMs: resolveTurnHardCeilingMsFromOptions(options),
919950
onProgress: progress
920951
}),
921952
{ json: options.json }

‎plugins/codex/scripts/lib/app-server.mjs‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,15 @@ const DEFAULT_CLIENT_INFO = {
4444
const DEFAULT_CAPABILITIES = {
4545
experimentalApi: false,
4646
requestAttestation: false,
47-
optOutNotificationMethods: [
48-
"item/agentMessage/delta",
49-
"item/reasoning/summaryTextDelta",
50-
"item/reasoning/summaryPartAdded",
51-
"item/reasoning/textDelta"
52-
]
47+
// Only opt out of notifications we neither render nor need. The streaming
48+
// text deltas (agentMessage, reasoning summary/text) are deliberately NOT
49+
// opted out: captureTurn's idle deadline treats every notification as proof
50+
// of life, and during a long reasoning stretch these deltas are the ONLY
51+
// liveness signal the server emits — no item boundary arrives for minutes.
52+
// Opting them out made the client blind to a working agent and killed the
53+
// turn mid-reasoning (the gpt-5.6-sol/effort=xhigh case behind #40).
54+
// summaryPartAdded stays opted out: it is a boundary marker, not a delta.
55+
optOutNotificationMethods: ["item/reasoning/summaryPartAdded"]
5356
};
5457

5558
function buildJsonRpcError(code, message, data) {

‎plugins/codex/scripts/lib/codex.mjs‎

Lines changed: 47 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -68,11 +68,19 @@ const DEFAULT_TURN_TIMEOUT_MS = 600000;
6868
// progress/delta notification, so a turn that keeps producing events — even
6969
// slowly, e.g. gpt-5.6-sol at effort=xhigh on a large diff routinely runs
7070
// well past 10 minutes while still making progress — is never killed by the
71-
// idle budget alone. This ceiling exists only to catch the case the idle
72-
// budget can't: a turn that resets its own idle timer forever without ever
73-
// completing (e.g. a runaway tool-call loop). It is intentionally generous
74-
// and is not meant to be tuned per-call the way the idle budget is.
75-
const HARD_WALL_CLOCK_CEILING_MS = 45 * 60 * 1000;
71+
// idle budget alone.
72+
//
73+
// The ceiling therefore serves two different purposes depending on the host:
74+
// - background: catch the case the idle budget can't — a turn that resets
75+
// its own idle timer forever without ever completing (runaway tool-call
76+
// loop). Generous by design; this default applies.
77+
// - foreground: also stay below the host's own kill. Claude Code's Bash
78+
// tool SIGKILLs node at ~120s, and a foreground turn that outlives that
79+
// is killed before captureTurn's catch can send turn/interrupt — leaving
80+
// a live, possibly write-capable turn running on the broker with no way
81+
// to reach it. The companion passes its 110s foreground budget as the
82+
// ceiling so the interrupt always wins the race. See #43.
83+
const DEFAULT_HARD_WALL_CLOCK_CEILING_MS = 45 * 60 * 1000;
7684

7785
// Resolve the per-turn idle budget at CALL time, not import time. The
7886
// companion sets CODEX_TURN_TIMEOUT_MS (e.g. the foreground budget, below
@@ -84,7 +92,7 @@ const HARD_WALL_CLOCK_CEILING_MS = 45 * 60 * 1000;
8492
// every turn/started, item/started, and item/completed notification (see
8593
// applyTurnNotification -> resetIdleDeadline). A turn that keeps producing
8694
// events can run indefinitely; only a gap this long with no events at all
87-
// trips it. A separate, much larger HARD_WALL_CLOCK_CEILING_MS backstops a
95+
// trips it. A separate wall-clock ceiling (resolveHardCeilingMs) backstops a
8896
// turn that never stops resetting its own idle timer.
8997
function resolveTurnTimeoutMs(options = {}) {
9098
const fromOptions = Number(options.turnTimeoutMs);
@@ -98,6 +106,23 @@ function resolveTurnTimeoutMs(options = {}) {
98106
return DEFAULT_TURN_TIMEOUT_MS;
99107
}
100108

109+
// Resolve the hard wall-clock ceiling at call time, mirroring
110+
// resolveTurnTimeoutMs. Precedence: explicit option (the companion passes
111+
// 110s for foreground, 45min for background) > CODEX_TURN_HARD_CEILING_MS
112+
// env override (tests, non-Claude hosts with a different external kill) >
113+
// the generous default. See DEFAULT_HARD_WALL_CLOCK_CEILING_MS.
114+
function resolveHardCeilingMs(options = {}) {
115+
const fromOptions = Number(options.hardCeilingMs);
116+
if (Number.isFinite(fromOptions) && fromOptions > 0) {
117+
return fromOptions;
118+
}
119+
const fromEnv = Number(process.env.CODEX_TURN_HARD_CEILING_MS);
120+
if (Number.isFinite(fromEnv) && fromEnv > 0) {
121+
return fromEnv;
122+
}
123+
return DEFAULT_HARD_WALL_CLOCK_CEILING_MS;
124+
}
125+
101126
function cleanCodexStderr(stderr) {
102127
return stderr
103128
.split(/\r?\n/)
@@ -625,15 +650,21 @@ function applyTurnNotification(state, message) {
625650
// ever emitting another item/started or item/completed in between. Treat
626651
// them as activity too, or a genuinely progressing item gets interrupted
627652
// mid-flight — the exact failure mode this idle timeout exists to avoid.
653+
// Every method below carries threadId + turnId, so belongsToTurn can route
654+
// it to this turn. Two sibling methods deliberately do NOT appear here:
655+
// command/exec/outputDelta and process/outputDelta are connection-scoped
656+
// (their params carry only processId / processHandle, no threadId), so
657+
// belongsToTurn can never match them and a case for them is dead code.
658+
// item/fileChange/outputDelta is kept although upstream deprecated it
659+
// ("the server no longer emits this notification") — an older app-server
660+
// may still send it, and treating it as activity costs nothing.
628661
case "item/commandExecution/outputDelta":
629662
case "item/fileChange/outputDelta":
630663
case "item/mcpToolCall/progress":
631664
case "item/agentMessage/delta":
632665
case "item/plan/delta":
633666
case "item/reasoning/summaryTextDelta":
634667
case "item/reasoning/textDelta":
635-
case "command/exec/outputDelta":
636-
case "process/outputDelta":
637668
state.resetIdleDeadline?.();
638669
break;
639670
case "error":
@@ -710,12 +741,15 @@ async function captureTurn(client, threadId, startRequest, options = {}) {
710741
state.resetIdleDeadline = armIdleDeadline;
711742

712743
// Coarse wall-clock failsafe: never reset, catches a turn that keeps
713-
// resetting its own idle timer forever without ever completing.
744+
// resetting its own idle timer forever without ever completing. In the
745+
// foreground it also keeps the turn inside the host's own kill window so
746+
// the interrupt in the catch below can still run (#43).
747+
const hardCeilingMs = resolveHardCeilingMs(options);
714748
let wallClockTimer = null;
715749
const wallClockCeiling = new Promise((_resolve, reject) => {
716750
wallClockTimer = setTimeout(() => {
717-
reject(new Error(`codex turn exceeded the ${HARD_WALL_CLOCK_CEILING_MS}ms hard wall-clock ceiling.`));
718-
}, HARD_WALL_CLOCK_CEILING_MS);
751+
reject(new Error(`codex turn exceeded the ${hardCeilingMs}ms hard wall-clock ceiling.`));
752+
}, hardCeilingMs);
719753
wallClockTimer.unref?.();
720754
});
721755

@@ -1234,6 +1268,7 @@ export async function runAppServerReview(cwd, options = {}) {
12341268
{
12351269
cwd,
12361270
turnTimeoutMs: options.turnTimeoutMs,
1271+
hardCeilingMs: options.hardCeilingMs,
12371272
onProgress: options.onProgress,
12381273
onResponse(response, state) {
12391274
if (response.reviewThreadId) {
@@ -1376,6 +1411,7 @@ export async function runAppServerTurn(cwd, options = {}) {
13761411
{
13771412
cwd,
13781413
turnTimeoutMs: options.turnTimeoutMs,
1414+
hardCeilingMs: options.hardCeilingMs,
13791415
onProgress: options.onProgress,
13801416
onResponse() {
13811417
if (!options.effort) {

‎tests/fake-codex-fixture.mjs‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -828,6 +828,96 @@ rl.on("line", (line) => {
828828
setTimeout(tick, gapMs);
829829
};
830830
setTimeout(tick, gapMs);
831+
} else if (BEHAVIOR === "long-progress-hits-wall-clock-ceiling") {
832+
// Continuously-progressing turn: one item/started, then a
833+
// commandExecution outputDelta every 500ms for ~20s. Every delta
834+
// resets the idle deadline, so the idle budget alone never fires --
835+
// only the separate hard wall-clock ceiling can stop this turn.
836+
// Registered as interruptible so turn/interrupt is observable.
837+
send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } });
838+
send({
839+
method: "item/started",
840+
params: {
841+
threadId: thread.id,
842+
turnId,
843+
item: { type: "commandExecution", id: "cmd_" + turnId, command: "very long step", status: "inProgress" }
844+
}
845+
});
846+
const ceilingGapMs = 500;
847+
const ceilingDeltaCount = 40;
848+
let ceilingStep = 0;
849+
const ceilingTick = () => {
850+
ceilingStep += 1;
851+
send({
852+
method: "item/commandExecution/outputDelta",
853+
params: { threadId: thread.id, turnId, itemId: "cmd_" + turnId, delta: "chunk " + ceilingStep + "\\n" }
854+
});
855+
if (ceilingStep >= ceilingDeltaCount) {
856+
send({
857+
method: "item/completed",
858+
params: {
859+
threadId: thread.id,
860+
turnId,
861+
item: { type: "commandExecution", id: "cmd_" + turnId, command: "very long step", status: "completed" }
862+
}
863+
});
864+
send({
865+
method: "item/completed",
866+
params: { threadId: thread.id, turnId, item: { type: "agentMessage", id: "msg_" + turnId, text: payload, phase: "final_answer" } }
867+
});
868+
send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } });
869+
interruptibleTurns.delete(turnId);
870+
return;
871+
}
872+
const nextTimer = setTimeout(ceilingTick, ceilingGapMs);
873+
interruptibleTurns.set(turnId, { threadId: thread.id, timer: nextTimer });
874+
};
875+
const firstCeilingTimer = setTimeout(ceilingTick, ceilingGapMs);
876+
interruptibleTurns.set(turnId, { threadId: thread.id, timer: firstCeilingTimer });
877+
} else if (BEHAVIOR === "reasoning-delta-idle-ok") {
878+
// The gpt-5.6-sol/xhigh case #40 was filed for: a long reasoning
879+
// stretch whose ONLY liveness signal is reasoning summary deltas.
880+
// Those are opt-ed out at handshake by default, so before the fix
881+
// the client never sees them and the idle budget kills the turn.
882+
send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } });
883+
send({
884+
method: "item/started",
885+
params: {
886+
threadId: thread.id,
887+
turnId,
888+
item: { type: "reasoning", id: "rsn_" + turnId, status: "inProgress" }
889+
}
890+
});
891+
const reasoningGapMs = 700;
892+
const reasoningDeltaCount = 6;
893+
let reasoningStep = 0;
894+
const reasoningTick = () => {
895+
reasoningStep += 1;
896+
send({
897+
method: "item/reasoning/summaryTextDelta",
898+
params: {
899+
threadId: thread.id,
900+
turnId,
901+
itemId: "rsn_" + turnId,
902+
delta: "thinking " + reasoningStep + "\\n",
903+
summaryIndex: 0
904+
}
905+
});
906+
if (reasoningStep >= reasoningDeltaCount) {
907+
send({
908+
method: "item/completed",
909+
params: { threadId: thread.id, turnId, item: { type: "reasoning", id: "rsn_" + turnId, status: "completed" } }
910+
});
911+
send({
912+
method: "item/completed",
913+
params: { threadId: thread.id, turnId, item: { type: "agentMessage", id: "msg_" + turnId, text: payload, phase: "final_answer" } }
914+
});
915+
send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } });
916+
return;
917+
}
918+
setTimeout(reasoningTick, reasoningGapMs);
919+
};
920+
setTimeout(reasoningTick, reasoningGapMs);
831921
} else {
832922
emitTurnCompleted(thread.id, turnId, items);
833923
}

0 commit comments

Comments
 (0)