Skip to content

fix: split the turn budgets and stop opting out of liveness deltas - #46

Merged
axisrow merged 2 commits into
mainfrom
fix/split-budgets-43
Aug 2, 2026
Merged

axisrow merged 2 commits into
mainfrom
fix/split-budgets-43

Conversation

@axisrow

@axisrow axisrow commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #43 — both confirmed findings from the follow-up to #41. They live in the same subsystem and share the same fixtures and tests, so they land together.

Finding 1: a foreground turn could outlive the host's SIGKILL

At base, FOREGROUND_TURN_TIMEOUT_MS (110s) was a non-resettable race against the whole turn, so a foreground turn was always cut before Claude Code's ~120s Bash SIGKILL and captureTurn's catch reached interruptAppServerTurn in time.

#41 made that deadline resettable. A foreground turn that keeps producing events now sails past 110s to the host kill: the catch never runs, no turn/interrupt is sent, and the broker keeps executing a possibly write-capable turn that nothing can reach. Three of the four links in that chain predate #41 (tracked in #42) — the reachability is what #41 introduced.

Fix: split the budgets rather than collapse them

  • The resettable idle budget is unchanged (110s FG / 600s BG).
  • The hard wall-clock ceiling becomes an option instead of the fixed HARD_WALL_CLOCK_CEILING_MS constant. The companion resolves it per host: foreground → FOREGROUND_TURN_TIMEOUT_MS (110s, below the Bash kill), background → 45 min (no external ceiling to collide with).
  • CODEX_TURN_HARD_CEILING_MS overrides it, for tests and for hosts whose external kill differs. No CLI flag — the ceiling is a property of the host, not something a caller should tune per run.
  • hardCeilingMs threads alongside turnTimeoutMs through buildTaskRequest / executeTaskRun / executeReviewRun → runAppServerTurn / runAppServerReview → captureTurn. The existing catch already interrupts on any deadline rejection, so it is untouched.

Foreground at defaults is behaviourally back to the pre-#41 ceiling, while a lowered idle budget still catches a genuine stall faster.

Finding 2: the liveness signal for long reasoning was opted out

Of the nine delta branches added in #41, only three could ever fire. item/agentMessage/delta, item/reasoning/summaryTextDelta and item/reasoning/textDelta were listed in DEFAULT_CAPABILITIES.optOutNotificationMethods, so the server never sent them.

That is precisely the gpt-5.6-sol / effort=xhigh case #40 was filed for: during a long reasoning stretch those deltas are the only liveness signal — no item boundary arrives for minutes — so the idle deadline fired and killed a turn that was demonstrably alive. The fix shipped in #41 did not actually cover its own motivating case.

  • Dropped those three from the opt-out list. item/reasoning/summaryPartAdded stays opted out — it is a boundary marker, not a delta.
  • Removed the command/exec/outputDelta and process/outputDelta cases: their params carry only a connection-scoped processId / processHandle and no threadId (verified against the generated protocol types in this repo), so belongsToTurn can never route them and the cases were dead code.
  • Kept item/fileChange/outputDelta although upstream deprecated it ("the server no longer emits this notification") — an older app-server may still send it, and treating it as activity costs nothing.

Test plan

TDD-first; each assertion confirmed red before / green after.

Two new fixture behaviors in tests/fake-codex-fixture.mjs:

  • long-progress-hits-wall-clock-ceiling — streams an item/commandExecution/outputDelta every 500ms for ~20s, so the idle budget is reset continuously and only the ceiling can stop the turn. Registered as interruptible so turn/interrupt is observable.
  • reasoning-delta-idle-ok — streams item/reasoning/summaryTextDelta across the idle budget, with no other liveness signal.

Three tests in tests/runtime.test.mjs:

  • a continuously-progressing turn is still bounded by the hard wall-clock ceiling, and the turn is interrupted — with CODEX_TURN_HARD_CEILING_MS=3500 the run is cut at ~5s citing the ceiling, the job is failed, and the fixture records lastInterrupt. Red pre-fix: the turn ran to completion (~21s).

  • a foreground turn's wall-clock ceiling stays below the host's Bash SIGKILL — the same fixture through the foreground path with the ceiling lowered must also be cut and interrupted. Red pre-fix for the same reason. (Asserts observable behaviour rather than the resolved number, so the ceiling doesn't have to leak into user-facing output.)

  • a long reasoning stretch keeps the turn alive: reasoning deltas are not opted out at handshake — the turn must complete, and the handshake capabilities recorded by the fixture must no longer contain the three methods while still containing summaryPartAdded. Red pre-fix on the opt-out assertion.

  • npm test — 184 tests, 184 pass, 0 fail

  • npm run build — passes

  • npm run check-version — all metadata matches 1.0.6-fork.6

Version

No bump — manifests stay at 1.0.6-fork.6. Suggestion for whoever cuts the next release: this changes foreground timeout behaviour and the handshake capabilities, so it likely deserves its own version.

Out of scope

Findings 3 and 4 from #43 are not touched here: #3 is already resolved (merged as #45), and #4 (buildEnv not pinning CODEX_COMPANION_SESSION_ID) is latent and unrelated to the timeout subsystem. The "installed plugin not updated" note in #43 is a release question, not a code change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8

axisrow and others added 2 commits August 2, 2026 18:44
Two findings from #43, both consequences of the idle-timeout redesign in
#41. They share the same subsystem and the same tests, so they land together.

Finding 1 — a foreground turn could outlive the host's SIGKILL.

At base, FOREGROUND_TURN_TIMEOUT_MS (110s) was a non-resettable race against
the whole turn, so a foreground turn was always cut before Claude Code's
~120s Bash SIGKILL and captureTurn's catch reached interruptAppServerTurn in
time. #41 made that deadline resettable, so a turn that keeps producing
events now sails past 110s to the host kill: the catch never runs, no
turn/interrupt is sent, and the broker keeps executing a possibly
write-capable turn nothing can reach. Three of the four links in that chain
predate #41 (see #42); the reachability is what #41 introduced.

Fix: split the two budgets instead of collapsing them.

  - The resettable idle budget is unchanged (110s FG / 600s BG).
  - The hard wall-clock ceiling becomes an option instead of the fixed
    HARD_WALL_CLOCK_CEILING_MS constant. The companion resolves it per host:
    foreground -> FOREGROUND_TURN_TIMEOUT_MS (110s, below the Bash kill),
    background -> 45 min (no external ceiling to collide with).
  - CODEX_TURN_HARD_CEILING_MS overrides it, for tests and for hosts whose
    external kill differs. No CLI flag: the ceiling is a property of the
    host, not something a caller should tune per run.
  - hardCeilingMs threads alongside turnTimeoutMs through buildTaskRequest /
    executeTaskRun / executeReviewRun -> runAppServerTurn / runAppServerReview
    -> captureTurn. The existing catch already interrupts on any deadline
    rejection, so it is untouched.

Foreground at defaults is behaviourally back to the pre-#41 ceiling, while a
lowered idle budget still catches a genuine stall faster.

Finding 2 — the liveness signal for long reasoning was opted out.

Of the nine delta branches added in #41, only three could ever fire.
item/agentMessage/delta, item/reasoning/summaryTextDelta and
item/reasoning/textDelta were listed in
DEFAULT_CAPABILITIES.optOutNotificationMethods, so the server never sent
them. That is precisely the gpt-5.6-sol / effort=xhigh case #40 was filed
for: during a long reasoning stretch those deltas are the ONLY liveness
signal — no item boundary arrives for minutes — so the idle deadline fired
and killed a turn that was demonstrably alive.

Fix: drop those three from the opt-out list. item/reasoning/summaryPartAdded
stays opted out — it is a boundary marker, not a delta.

Also removed the command/exec/outputDelta and process/outputDelta cases:
their params carry only a connection-scoped processId / processHandle and no
threadId (verified against the generated protocol types), so belongsToTurn
can never route them and the cases were dead code. item/fileChange/outputDelta
is kept despite being deprecated upstream — an older app-server may still
emit it, and treating it as activity costs nothing.

Tests (TDD-first, each confirmed red before / green after):

  - long-progress-hits-wall-clock-ceiling fixture: streams an outputDelta
    every 500ms for ~20s, so the idle budget is reset continuously and only
    the ceiling can stop the turn.
      * with CODEX_TURN_HARD_CEILING_MS=3500 the run is cut at ~5s citing the
        ceiling, the job is failed, and the fixture records lastInterrupt.
        Red pre-fix: the turn ran to completion (~21s).
      * the same fixture through the foreground path with the ceiling lowered
        must also be cut and interrupted — red pre-fix for the same reason.
  - reasoning-delta-idle-ok fixture: streams reasoning summary deltas across
    the idle budget. The turn must complete, and the handshake capabilities
    recorded by the fixture must no longer opt out of the three methods,
    while still opting out of summaryPartAdded. Red pre-fix on the opt-out
    assertion.

npm test: 184 tests, 184 pass, 0 fail. npm run build passes. Version
metadata untouched at 1.0.6-fork.6.

Closes #43

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8
The comment pointed at HARD_WALL_CLOCK_CEILING_MS, which this branch
renamed to DEFAULT_HARD_WALL_CLOCK_CEILING_MS, and described the ceiling
as "much larger" — no longer true in the foreground, where it now equals
the 110s foreground budget so it stays under the host's Bash SIGKILL.

Comment-only; no behaviour change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FQBJirbWNKWhnpFUnWogB8
@axisrow

axisrow commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

🔍 Local review (cycle 1)

Reviewed locally (/review + Codex companion gpt-5.6-sol/xhigh), no bots pinged.

Verdict Reviewer Finding Location
SKIP codex Hard ceiling starts too late to precede the host SIGKILL (critical, conf. 0.97) lib/codex.mjs:747
FIX claude Stale comment references the renamed HARD_WALL_CLOCK_CEILING_MS codex-companion.mjs:727
SKIP claude Idle error says "turn budget", ambiguous against the ceiling message lib/codex.mjs:733
SKIP claude Second ceiling test's foreground property is a comment, not an assertion runtime.test.mjs:3364

On Codex's critical finding

Every factual premise checks out, and one sub-claim is understated:

  • The ceiling is armed inside captureTurn (lib/codex.mjs:747), after connect → startThread/resume → validateReasoningSelection. The clock genuinely starts late.
  • Setup is not merely "~10s" — it is unbounded. AppServerClientBase.request (app-server.mjs:101-131) only arms a timer when the caller passes timeoutMs, and the only callers that do are the two initialize paths. thread/start, thread/resume and validateReasoningSelection have no deadline. Bounded handshake alone is additive to ~17s (broker 2s+5s, spawned fallback 10s).
  • interruptAppServerTurn (lib/codex.mjs:1180-1223) opens a fresh connection and issues turn/interrupt with no timeoutMs, inside a window that by construction has ~0s left. This is the sharpest part of the finding.

But it is not a regression from this PR, and blocking on it would block a strict improvement.

Pre-#41 (c33cf49^:lib/codex.mjs:645-650) armed its 110s total foreground budget at the identical point — inside captureTurn, after the same setup. And at this PR's base (ed9b4e3), #41 had left the ceiling as a fixed 45-minute constant, so the foreground currently has no wall-clock bound at all: a delta-producing foreground turn runs until SIGKILL. This PR moves the foreground case from no bound to a bound that holds in the realistic case (110s + ~1-3s setup ≈ 113s, inside the ~120s kill). The invariant Codex names — "the ceiling fires strictly before the host kill, with reserved cleanup budget" — has never been held by any revision of this code.

So: real weakness, correctly identified, wrong severity for this diff. Filing as a follow-up in #42's territory:

  1. Pass timeoutMs to turn/interrupt in interruptAppServerTurn — cheapest, highest-value; closes the unbounded-cleanup hole.
  2. Anchor an absolute deadline at command entry and subtract elapsed setup before arming the ceiling.
  3. Bound the thread/start / thread/resume / validateReasoningSelection RPCs.

On the SKIPs

  • Ambiguous error message — accurate, and slightly worse after this PR since both numbers are now 110000 in the foreground; the two messages differ only by their trailing phrase. Already recorded as a deliberate deferred SKIP in Follow-up to #41: foreground hard-ceiling regression + dead delta branches (protocol research attached) #43. Diagnostic strings only; the tests discriminate on /hard wall-clock ceiling/i.
  • Test assertion — the redundancy is overstated (both ceiling tests are foreground, so the second is a near-duplicate rather than an unenforced claim). No assertion is available without leaking the resolved ceiling into user-facing output or exporting a module-private resolver purely for the test. The genuinely uncovered case is the opposite one — that --background resolves to the 45-minute default — which would need a 45-minute run or a new seam. Noted on the follow-up.

Fixed in b3ecf07. npm test 184/184, npm run build green.

@axisrow

axisrow commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

📋 Review summary — all cycles

Local review (/review + Codex companion gpt-5.6-sol/xhigh). Two cycles; cycle 2 clean.

Cycle Reviewer Finding Verdict Resolution
1 codex Hard ceiling starts too late to precede the host SIGKILL (critical) SKIP Pre-existing; filed as #47
1 claude Stale comment references the renamed HARD_WALL_CLOCK_CEILING_MS FIX Fixed in b3ecf07
1 claude Idle error says "turn budget", ambiguous against the ceiling message SKIP Deferred in #43; noted on #47
1 claude Second ceiling test's foreground property is a comment, not an assertion SKIP No seam available; noted on #47
2 codex Hard ceiling can hit the host kill while awaiting an unbounded interrupt (critical) SKIP Pre-existing; filed as #47
2 claude — — Clean round, no findings

Totals: 1 FIX (resolved), 4 SKIP, 0 UNVERIFIED.

On the two Codex criticals

Both were verified in detail rather than taken at face value, and both hold factually — one of them is even understated (I found setup RPCs to be fully unbounded, not the "~10s" Codex estimated, because request() only arms a timer when a caller passes timeoutMs). Neither is a regression from this PR:

  • Round 1 (ceiling armed after setup): pre-fix: measure turn timeout as agent idle time, not total wall-clock #41 (c33cf49^:lib/codex.mjs:645-650) armed its 110s budget at the identical point, after the same setup sequence.
  • Round 2 (unbounded turn/interrupt): interruptAppServerTurn is byte-identical to base ed9b4e3; this PR does not touch it. And the catch containing it was already reachable at base at the same ~110s via the idle deadline — so this PR adds a second trigger into an already-reachable path, not new reachability.

The decisive point: at this PR's base the foreground had no wall-clock bound at all (#41 left the ceiling at a fixed 45 min), so a delta-producing foreground turn was SIGKILLed with no interrupt attempted whatsoever. This PR converts that into "an interrupt is attempted inside the host kill window" — strictly better, or at worst equal when the peer accepts and then goes silent.

That last case is real and worth closing, so both findings plus the two deferred minors are now tracked in #47, with the one-line timeoutMs on turn/interrupt as the cheapest first step.

Cleanup pass

No-op. The one remaining candidate — renaming the idle error's "turn budget" wording — would require updating three test assertions (runtime.test.mjs:3203,3253,3422), which is scope creep for a diagnostic string already recorded as a deliberate SKIP in #43. Moved to #47 instead.

npm test 184/184, npm run build green at b3ecf07.

Review complete — merge is yours to trigger (local mode does not auto-merge).

@axisrow
axisrow merged commit fa3c630 into main Aug 2, 2026
@axisrow
axisrow deleted the fix/split-budgets-43 branch August 2, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Follow-up to #41: foreground hard-ceiling regression + dead delta branches (protocol research attached)

1 participant