Repository navigation
fix: split the turn budgets and stop opting out of liveness deltas - #46
Conversation
Two findings from #43, both consequences of the idle-timeout redesign in #41. They share the same subsystem and the same tests, so they land together. Finding 1 — a foreground turn could outlive the host's SIGKILL. At base, FOREGROUND_TURN_TIMEOUT_MS (110s) was a non-resettable race against the whole turn, so a foreground turn was always cut before Claude Code's ~120s Bash SIGKILL and captureTurn's catch reached interruptAppServerTurn in time. #41 made that deadline resettable, so a turn that keeps producing events now sails past 110s to the host kill: the catch never runs, no turn/interrupt is sent, and the broker keeps executing a possibly write-capable turn nothing can reach. Three of the four links in that chain predate #41 (see #42); the reachability is what #41 introduced. Fix: split the two budgets instead of collapsing them. - The resettable idle budget is unchanged (110s FG / 600s BG). - The hard wall-clock ceiling becomes an option instead of the fixed HARD_WALL_CLOCK_CEILING_MS constant. The companion resolves it per host: foreground -> FOREGROUND_TURN_TIMEOUT_MS (110s, below the Bash kill), background -> 45 min (no external ceiling to collide with). - CODEX_TURN_HARD_CEILING_MS overrides it, for tests and for hosts whose external kill differs. No CLI flag: the ceiling is a property of the host, not something a caller should tune per run. - hardCeilingMs threads alongside turnTimeoutMs through buildTaskRequest / executeTaskRun / executeReviewRun -> runAppServerTurn / runAppServerReview -> captureTurn. The existing catch already interrupts on any deadline rejection, so it is untouched. Foreground at defaults is behaviourally back to the pre-#41 ceiling, while a lowered idle budget still catches a genuine stall faster. Finding 2 — the liveness signal for long reasoning was opted out. Of the nine delta branches added in #41, only three could ever fire. item/agentMessage/delta, item/reasoning/summaryTextDelta and item/reasoning/textDelta were listed in DEFAULT_CAPABILITIES.optOutNotificationMethods, so the server never sent them. That is precisely the gpt-5.6-sol / effort=xhigh case #40 was filed for: during a long reasoning stretch those deltas are the ONLY liveness signal — no item boundary arrives for minutes — so the idle deadline fired and killed a turn that was demonstrably alive. Fix: drop those three from the opt-out list. item/reasoning/summaryPartAdded stays opted out — it is a boundary marker, not a delta. Also removed the command/exec/outputDelta and process/outputDelta cases: their params carry only a connection-scoped processId / processHandle and no threadId (verified against the generated protocol types), so belongsToTurn can never route them and the cases were dead code. item/fileChange/outputDelta is kept despite being deprecated upstream — an older app-server may still emit it, and treating it as activity costs nothing. Tests (TDD-first, each confirmed red before / green after): - long-progress-hits-wall-clock-ceiling fixture: streams an outputDelta every 500ms for ~20s, so the idle budget is reset continuously and only the ceiling can stop the turn. * with CODEX_TURN_HARD_CEILING_MS=3500 the run is cut at ~5s citing the ceiling, the job is failed, and the fixture records lastInterrupt. Red pre-fix: the turn ran to completion (~21s). * the same fixture through the foreground path with the ceiling lowered must also be cut and interrupted — red pre-fix for the same reason. - reasoning-delta-idle-ok fixture: streams reasoning summary deltas across the idle budget. The turn must complete, and the handshake capabilities recorded by the fixture must no longer opt out of the three methods, while still opting out of summaryPartAdded. Red pre-fix on the opt-out assertion. npm test: 184 tests, 184 pass, 0 fail. npm run build passes. Version metadata untouched at 1.0.6-fork.6. Closes #43 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8
The comment pointed at HARD_WALL_CLOCK_CEILING_MS, which this branch renamed to DEFAULT_HARD_WALL_CLOCK_CEILING_MS, and described the ceiling as "much larger" — no longer true in the foreground, where it now equals the 110s foreground budget so it stays under the host's Bash SIGKILL. Comment-only; no behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8
🔍 Local review (cycle 1)Reviewed locally (
On Codex's critical findingEvery factual premise checks out, and one sub-claim is understated:
But it is not a regression from this PR, and blocking on it would block a strict improvement. Pre-#41 ( So: real weakness, correctly identified, wrong severity for this diff. Filing as a follow-up in #42's territory:
On the SKIPs
Fixed in |
📋 Review summary — all cyclesLocal review (
Totals: 1 FIX (resolved), 4 SKIP, 0 UNVERIFIED. On the two Codex criticalsBoth were verified in detail rather than taken at face value, and both hold factually — one of them is even understated (I found setup RPCs to be fully unbounded, not the "~10s" Codex estimated, because
The decisive point: at this PR's base the foreground had no wall-clock bound at all (#41 left the ceiling at a fixed 45 min), so a delta-producing foreground turn was SIGKILLed with no interrupt attempted whatsoever. This PR converts that into "an interrupt is attempted inside the host kill window" — strictly better, or at worst equal when the peer accepts and then goes silent. That last case is real and worth closing, so both findings plus the two deferred minors are now tracked in #47, with the one-line Cleanup passNo-op. The one remaining candidate — renaming the idle error's "turn budget" wording — would require updating three test assertions (
Review complete — merge is yours to trigger (local mode does not auto-merge). |
Summary
Closes #43 — both confirmed findings from the follow-up to #41. They live in the same subsystem and share the same fixtures and tests, so they land together.
Finding 1: a foreground turn could outlive the host's SIGKILL
At base,
FOREGROUND_TURN_TIMEOUT_MS(110s) was a non-resettable race against the whole turn, so a foreground turn was always cut before Claude Code's ~120s Bash SIGKILL andcaptureTurn's catch reachedinterruptAppServerTurnin time.#41 made that deadline resettable. A foreground turn that keeps producing events now sails past 110s to the host kill: the catch never runs, no
turn/interruptis sent, and the broker keeps executing a possibly write-capable turn that nothing can reach. Three of the four links in that chain predate #41 (tracked in #42) — the reachability is what #41 introduced.Fix: split the budgets rather than collapse them
HARD_WALL_CLOCK_CEILING_MSconstant. The companion resolves it per host: foreground →FOREGROUND_TURN_TIMEOUT_MS(110s, below the Bash kill), background → 45 min (no external ceiling to collide with).CODEX_TURN_HARD_CEILING_MSoverrides it, for tests and for hosts whose external kill differs. No CLI flag — the ceiling is a property of the host, not something a caller should tune per run.hardCeilingMsthreads alongsideturnTimeoutMsthroughbuildTaskRequest/executeTaskRun/executeReviewRun→runAppServerTurn/runAppServerReview→captureTurn. The existing catch already interrupts on any deadline rejection, so it is untouched.Foreground at defaults is behaviourally back to the pre-#41 ceiling, while a lowered idle budget still catches a genuine stall faster.
Finding 2: the liveness signal for long reasoning was opted out
Of the nine delta branches added in #41, only three could ever fire.
item/agentMessage/delta,item/reasoning/summaryTextDeltaanditem/reasoning/textDeltawere listed inDEFAULT_CAPABILITIES.optOutNotificationMethods, so the server never sent them.That is precisely the
gpt-5.6-sol/effort=xhighcase #40 was filed for: during a long reasoning stretch those deltas are the only liveness signal — no item boundary arrives for minutes — so the idle deadline fired and killed a turn that was demonstrably alive. The fix shipped in #41 did not actually cover its own motivating case.item/reasoning/summaryPartAddedstays opted out — it is a boundary marker, not a delta.command/exec/outputDeltaandprocess/outputDeltacases: their params carry only a connection-scopedprocessId/processHandleand nothreadId(verified against the generated protocol types in this repo), sobelongsToTurncan never route them and the cases were dead code.item/fileChange/outputDeltaalthough upstream deprecated it ("the server no longer emits this notification") — an older app-server may still send it, and treating it as activity costs nothing.Test plan
TDD-first; each assertion confirmed red before / green after.
Two new fixture behaviors in
tests/fake-codex-fixture.mjs:long-progress-hits-wall-clock-ceiling— streams anitem/commandExecution/outputDeltaevery 500ms for ~20s, so the idle budget is reset continuously and only the ceiling can stop the turn. Registered as interruptible soturn/interruptis observable.reasoning-delta-idle-ok— streamsitem/reasoning/summaryTextDeltaacross the idle budget, with no other liveness signal.Three tests in
tests/runtime.test.mjs:a continuously-progressing turn is still bounded by the hard wall-clock ceiling, and the turn is interrupted— withCODEX_TURN_HARD_CEILING_MS=3500the run is cut at ~5s citing the ceiling, the job isfailed, and the fixture recordslastInterrupt. Red pre-fix: the turn ran to completion (~21s).a foreground turn's wall-clock ceiling stays below the host's Bash SIGKILL— the same fixture through the foreground path with the ceiling lowered must also be cut and interrupted. Red pre-fix for the same reason. (Asserts observable behaviour rather than the resolved number, so the ceiling doesn't have to leak into user-facing output.)a long reasoning stretch keeps the turn alive: reasoning deltas are not opted out at handshake— the turn must complete, and the handshake capabilities recorded by the fixture must no longer contain the three methods while still containingsummaryPartAdded. Red pre-fix on the opt-out assertion.npm test— 184 tests, 184 pass, 0 failnpm run build— passesnpm run check-version— all metadata matches1.0.6-fork.6Version
No bump — manifests stay at
1.0.6-fork.6. Suggestion for whoever cuts the next release: this changes foreground timeout behaviour and the handshake capabilities, so it likely deserves its own version.Out of scope
Findings 3 and 4 from #43 are not touched here: #3 is already resolved (merged as #45), and #4 (
buildEnvnot pinningCODEX_COMPANION_SESSION_ID) is latent and unrelated to the timeout subsystem. The "installed plugin not updated" note in #43 is a release question, not a code change.🤖 Generated with Claude Code
https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8