Repository navigation
Description
Problem
Interrupting a running turn can take several seconds before the server settles the execution and projects its terminal state. In one observed run, shell/tool cancellation was timestamped at 16:33:49.400Z, while the assistant terminal failure was not created until 16:34:00.946Z and became visible around 16:34:01Z.
This is actual server-side settlement latency, not just delayed client feedback.
Observed behavior
- The user interrupts a turn with an active shell/provider step.
- Tool cancellation begins promptly.
- The interrupt request remains pending while cleanup and durable publication settle.
- The execution does not reach its durable interrupted terminal for roughly 11.5 seconds.
Several unrelated sessions resumed logging at approximately the same time, suggesting shared service contention rather than only per-session shell teardown.
Current evidence
Two independent blocking paths are confirmed in the runner:
- Interrupted settlement waits for
captureStepEnd(). A blocked end snapshot prevents both the run and interrupt request from settling. - Durable events across sessions share a single SQLite transaction permit. An unrelated open transaction can prevent the interrupted step from durably settling.
The historical timestamp pattern points more strongly at durable-event/SQLite contention than end-snapshot work: event timestamps are assigned before durable publication waits, and unrelated sessions showed synchronized progress after the stall. This is not conclusive because phase-level timing was absent.
Other unbounded cleanup paths remain relevant, including shell close waiting and provider stream cancellation.
Desired outcome
Interrupt settlement should have bounded latency under snapshot, database, shell, and provider contention without weakening:
- durable event ordering
- exact changed-file attribution
- Undo/revert behavior for partial file mutations
- single terminal-event semantics
Recommended investigation
- Add phase timing around tool terminal publication, tool-fiber joins, end-snapshot capture, changed-file calculation, step terminal publication, execution terminal publication, and shell/provider finalizers.
- Record SQLite transaction-permit queue time separately from transaction execution and listener time.
- Reproduce the long-tail interrupt with instrumentation.
- Fix the measured blocker rather than masking it in clients.
- Add a latency regression test using deterministic gates, not wall-clock sleeps.
Potential directions to evaluate after measurement:
- Replace native SQLite busy waiting with cooperative, bounded retry so the Bun event loop remains schedulable.
- Reduce or isolate work performed while holding the global durable transaction permit.
- Bound interrupted end-snapshot work while preserving the exact pre/post boundary needed by Undo.
- Bound shell and provider cancellation finalizers with explicit escalation and observable failure states.
Test seams
The exploration branch interrupt-latency-exploration contains deterministic characterization tests in packages/core/test/session-runner.test.ts for:
- blocked interrupted end-snapshot capture
- unrelated database transaction contention during interrupted settlement
The client-side Stopping... prototype from #42115 was intentionally closed because it improved perceived responsiveness but did not reduce server latency.
Activity
cms19859230182-lang commentedon Aug 14, 2026
I'll take this one — starting with the phase-timing investigation, then fixing the measured blocker. I'll reuse the deterministic gates from the exploration branch.
neriousy commentedon Oct 21, 2026
Partly done: #43552 makes the interrupt request return as soon as the interruption is accepted, so the client no longer waits for cleanup. Settling on the server still has no time limit and no per-phase timing, and #42539 (cooperative SQLite retry) was closed unmerged. Closing this tracker. If interrupt latency comes up again, timings from a real run are the best starting point.