Skip to content

Commit 7f79a37

Browse files
EdwardLiuycclaude
andauthored
M0-4 · Cancellation probe findings (#56)
Fills docs/M0-FINDINGS.md §M0-4 and folds the results into the plan and stories. Probe tooling kept on spike/m0-4-cancel-probe (never merged). Cancellation is fast and clean on all three harnesses: the turn ends within 64 ms of session/cancel, nothing arrives afterwards, and the session stays usable. M1c-1 therefore gets a 2 s grace and never needs to kill. Two results the plan did not anticipate: - copilot answers `end_turn` for a turn it cancelled, contrary to ACP's MUST and undocumented by GitHub. Filed upstream as github/copilot-cli#4561. M1b-2 must key status on (stopReason, did we cancel, did the watchdog fire). - every harness accepts a second session/prompt mid-turn: none rejects it, the in-flight turn dies, both requests answer `end_turn`, and on opencode and copilot the text lands in the running turn's context, where both models read it as a possible prompt injection (github/copilot-cli#4555 has the mechanism). One prompt per session per task; there is no safe nudge. Closes #5. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_013ZVZAP4MdJ69xjyDHg6RVx
1 parent ae6a885 commit 7f79a37

3 files changed

Lines changed: 187 additions & 11 deletions

File tree

‎docs/M0-FINDINGS.md‎

Lines changed: 137 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -351,9 +351,145 @@ content on the floor.
351351
- `unknown` reads as `internal` / `none` for warning purposes. `omp`, `pi` and `gemini`
352352
are uninstalled here and are **unknown by absence, not by test**.
353353

354+
---
355+
354356
## M0-4 · Cancellation probe
355357

356-
Pending.
358+
Run 2026-08-21 against the same three harnesses, with a throwaway probe that sends
359+
`session/cancel` itself and then **waits** — deliberately not `run.mjs`, whose SIGINT path
360+
(`run.mjs:81-88`) kills the subprocess 2 s after cancelling. Because `#failAll` rejects
361+
every pending request on `close` (`acp.mjs:62-64`), a cancel slower than that grace would
362+
have surfaced as `agent "x" exited with code N` and the stop reason would have been lost.
363+
The probe is spike tooling, kept on branch `spike/m0-4-cancel-probe` (never merged) the way
364+
M0-3's stubs were; the numbers are the deliverable.
365+
366+
Three runs per harness, each in its own workspace (a copy of this repo's five `.mjs` files),
367+
`--mode auto`, prompt *"read every file in this directory one at a time and summarise it"*:
368+
369+
1. **observe** — nobody interferes. The control: without it, "the turn ended" cannot be
370+
distinguished from "the turn was ending anyway".
371+
2. **cancel** — `session/cancel` on the **first `tool_call`** (a protocol event, not a
372+
wall-clock delay, so the three are cut at a comparable point despite very different turn
373+
lengths), then wait up to 60 s, then send a second prompt to test whether the session
374+
survived.
375+
3. **double** — a second `session/prompt` while the first turn is in flight, awaited to
376+
completion, carrying a marker word so "served" can be told from "swallowed".
377+
378+
### Question 1 — does the turn end promptly, and with what `stopReason`?
379+
380+
Promptly on all three — and one of them lies about why.
381+
382+
| | opencode 1.18.18 | claude-agent-acp 0.49.0 | copilot 1.0.80 |
383+
|---|---|---|---|
384+
| `stopReason` on cancel | `cancelled` | `cancelled` | **`end_turn`** |
385+
| latency, 2 samples | 64 ms · 45 ms | 5 ms · 10 ms | 26 ms · 28 ms |
386+
| events in the cancelled turn | 20 · 16 | 14 · 14 | 11 · 11 |
387+
| events in an uninterrupted turn | **443** | **180** | **69** |
388+
| uninterrupted turn length | 68.9 s | 51.2 s | 36.3 s |
389+
| `session/update` after the response | 0 | 0 | 0 |
390+
| session usable afterwards | yes (`end_turn`) | yes | yes |
391+
392+
**Latency is not the interesting number.** Every sample landed inside 64 ms, two orders of
393+
magnitude below any plausible TTL, and nothing arrived on the wire after the response. **No
394+
harness came near the 60 s wait bound**, so there is no "did not end within N s" row to
395+
record. The interesting number is the stop reason.
396+
397+
Note also what is *not* wrong here: every literal observed is inside M1b-2's set
398+
(`docs/STORIES.md:334-343`) — nobody invented a value. The mismatch is semantic, and that is
399+
worse, because a vocabulary error would have been caught by a type.
400+
401+
**Copilot reports `end_turn` for a turn it cancelled**, in both samples. The baseline row is
402+
what settles it: 11 events and ~5.6 s against 69 events and 36.3 s, with the `tool_call`
403+
that triggered the cancel never completing. It obeyed immediately and then described the
404+
result as a clean finish. This contradicts the protocol's own MUST — *"the Agent **MUST**
405+
respond to the original `session/prompt` request with the `cancelled` stop reason"*
406+
([ACP, prompt turn](https://agentclientprotocol.com/protocol/prompt-turn)) — and it is
407+
**undocumented**: GitHub's [ACP server reference](https://docs.github.com/en/copilot/reference/copilot-cli-reference/acp-server)
408+
names `stopReason` only inside an example (`!== "end_turn"`) and never enumerates the
409+
values, and [Canceling a Copilot CLI operation](https://docs.github.com/en/copilot/concepts/agents/copilot-cli/cancel-and-roll-back)
410+
covers only Esc and Ctrl-C in the TUI. No issue on `github/copilot-cli` reported it either,
411+
so this spike filed one:
412+
[github/copilot-cli#4561](https://github.com/github/copilot-cli/issues/4561), with a
413+
standalone repro and the control run above as the evidence the turn was truncated.
414+
415+
Worth knowing that the public record here is inverted: this exact defect was filed against
416+
**opencode** ([anomalyco/opencode#25899](https://github.com/anomalyco/opencode/issues/25899),
417+
*"reporting end_turn for a user-cancel makes a stopped turn look like a clean completion"*)
418+
and closed as not planned — yet opencode 1.18.18 answers `cancelled` here. Neither the
419+
issue tracker nor the vendor's docs predict what a given build does. **Pin the version next
420+
to every row above; that is the only durable part of this table.**
421+
422+
### Question 2 — does a second `session/prompt` in flight get rejected?
423+
424+
**No. The expectation in the story is wrong for all three, in three different ways, and the
425+
failure is silent every time.**
426+
427+
| | opencode | claude-agent-acp | copilot |
428+
|---|---|---|---|
429+
| second prompt | accepted | accepted | accepted |
430+
| answered after | 7.0 s (with the first, ~8 ms apart) | 52.1 s | **10 ms** |
431+
| its `stopReason` | `end_turn` | `end_turn` | `end_turn` |
432+
| effect on the running turn | aborted at 14.3 s (vs 68.9 s) | ended at 5.5 s (vs 51.2 s) | aborted at 11.0 s (vs 36.3 s) |
433+
| where the text landed | merged into the running turn | served as a fresh turn | merged into the running turn |
434+
435+
Nobody errors, nobody queues cleanly, and **the in-flight turn dies in every case** — with
436+
`stopReason: end_turn`, again indistinguishable from success. Copilot's answer is the
437+
strangest: the JSON-RPC response came back 10 ms after the request, *before* the prompt had
438+
been acted on and 5.8 s before the first turn's own response. For a concurrent prompt, that
439+
response is not a completion signal at all.
440+
441+
The marker earns its place: rather than being executed, the second prompt shows up in the
442+
agents' *reasoning* about the first task — copilot: *"looks like it could be a prompt
443+
injection asking me to just reply BANANA"*; claude-agent-acp: *"…conflicts with the task
444+
you…"*. So on opencode and copilot a concurrent prompt is not a message to the agent, it is
445+
**an injection into the running turn's context**. Only claude-agent-acp treats it as a turn
446+
of its own (ending the first at 5.5 s, then spending 52 s on the second).
447+
448+
Copilot's half of this is independently reported, with the mechanism:
449+
[github/copilot-cli#4555](https://github.com/github/copilot-cli/issues/4555) (open) —
450+
*"the `session/prompt` handler calls `session.abort()` unconditionally as its **first
451+
action**"*, tearing down background work *"even when no prompt turn is in flight … and the
452+
client never sent `session/cancel`"*, while the `cancel` handler guards on `pendingPrompt`
453+
and `prompt` does not. That extends our finding: on copilot it is not concurrency that is
454+
unsafe, it is **any** second prompt, including one sent to a session we believe is idle.
455+
Adjacent, same area: [#4336](https://github.com/github/copilot-cli/issues/4336),
456+
[#4127](https://github.com/github/copilot-cli/issues/4127).
457+
458+
### Question 3 — what state is the session left in?
459+
460+
Clean, on all three. Zero `session/update` notifications arrived after the cancelled
461+
`session/prompt` response, and a follow-up prompt was accepted and completed (`end_turn`) in
462+
every cancel run. **A cancelled task does not need its subprocess killed**, and the client
463+
may `stop()` immediately after the response without racing a trailing notification.
464+
465+
### Consequences
466+
467+
- **M1c-1's grace window: 2 s.** Thirty times the slowest observed cancel and identical to
468+
the grace `run.mjs:81-88` already uses, so the watchdog inherits a number rather than
469+
inventing one. Kill only if that expires — expect never to reach it.
470+
- **M1b-2's stop-reason table cannot key on `stopReason` alone.** `cancelled` may never
471+
arrive (copilot), and `end_turn` is now known to mean *finished*, *I was cancelled*, or
472+
*someone interrupted me*, depending on harness. The status must be derived from the
473+
triple (`stopReason`, did **we** send a cancel, did the watchdog fire) — the supervisor's
474+
own record of intent is the authoritative half. The table stands; its input does not.
475+
- **One prompt per session per task, and never a "nudge".** Given #4555, a supervisor that
476+
sends a second prompt — even to a session it believes idle — can silently destroy work
477+
and get `end_turn` for it. This is a second, empirical argument for M1c-1's deliberate
478+
absence of a third rung: there is no safe way to steer a turn in flight, and on two of
479+
three harnesses the attempt lands in the model's context as a suspected prompt injection.
480+
- **A cancelled turn's worktree is worth keeping.** Cancellation truncates mid-tool-call
481+
(copilot's triggering `tool_call` never completed), so the same "stopped mid-edit" hazard
482+
M1b-2 records for `max_tokens` applies to `timed_out` tasks.
483+
484+
### Coverage
485+
486+
Two samples per harness for cancellation, one for each of the other two questions, one
487+
workspace shape, one read-only prompt. `omp`, `pi` and `gemini` remain uninstalled here —
488+
**unknown by absence, not by test**. Nothing was written to any workspace during these
489+
runs, so the interaction between cancellation and a half-finished *edit* is untested and is
490+
the obvious next probe if `timed_out` tasks ever start losing work.
491+
492+
---
357493

358494
## M0-5 · Usage reporting survey
359495

‎docs/PROJECT-PLAN.md‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ policy across all of them and returning machine-readable results.
3434
| Result contract | A `<task-result>` block in the final message. `git status` cross-checks it; disagreement raises a task **warning**, never a failure. MCP `report_result` deferred. |
3535
| Worktrees | Supervisor owns fully — creates off a base ref, removes on success, **keeps worktree and branch on failure or timeout**, reaps orphans only when the run is terminal and past the retention window. |
3636
| Merge-back | Explicit `agent-team merge <runId>`; never automatic. `--squash` (default) lands changes **unstaged**; `--rebase` preserves per-stage commits. Aborts wholesale on overlap. |
37-
| Liveness | Per-task **stall → warn** (idle, or the same tool repeating) and **TTL → `session/cancel`**. Each disabled by setting `0`. |
37+
| Liveness | Per-task **stall → warn** (idle, or the same tool repeating) and **TTL → `session/cancel`**. Each disabled by setting `0`. **Measured (M0-4): cancellation works and is fast — every harness ended its turn within 64 ms and stopped emitting, and the session stayed usable, so the TTL rung needs a 2 s grace and no kill. But the stop reason cannot classify the outcome: copilot reports `end_turn` for a turn it cancelled, so the supervisor's own record of *did I cancel* is the authoritative half.** |
3838
| Crash recovery | Startup sweep plus transactional spawn. A partially-created task leaves nothing behind. |
3939
| Run definition | A `.ts` module default-exporting the run object; `.json` also accepted; the daemon's POST body is the same shape. One validator behind all three. |
4040
| Handoff | Same worktree, sequential. Successor sees predecessor's changes on disk plus its result summary in-prompt. |
@@ -205,8 +205,9 @@ internally — so for that harness we judge more than a title string even withou
205205
Agenda item #1 in the brief. Costs almost nothing and de-risks every design choice after.
206206

207207
> **Status:** item 1 is done — see `M0-FINDINGS.md`. Items 2 and 7 are done — M0-2, which
208-
> also retired the `claude-code` registry entry. Items 3-6 and 8 are still open, and 3/5/6
209-
> now start from the baseline that survey recorded rather than from nothing.
208+
> also retired the `claude-code` registry entry. Items 3 and 4 are done — M0-3. Item 5 is
209+
> done — M0-4. Items 6 and 8 are still open, and 6 now starts from the baseline those
210+
> surveys recorded rather than from nothing.
210211
211212
1. ~~Run today's MVP against real **opencode** and **claude-agent-acp**.~~ **Done, and
212213
widened to all three targets.** All spawn, all complete a turn at protocol v1, none
@@ -232,10 +233,17 @@ Agenda item #1 in the brief. Costs almost nothing and de-risks every design choi
232233
(`terminal | notification | permission | none`). opencode shows us nothing and cannot be
233234
gated by argv at all; claude-agent-acp's argv arrives late, on `tool_call_update`.
234235
**M3-6 therefore survives without M3-5.**
235-
5. **Cancellation probe.** Confirm the expected result — that no harness accepts a second
236-
`session/prompt` while a turn is in flight — and record what each one does with
237-
`session/cancel` mid-turn: does the turn end promptly, and with what `stopReason`?
238-
M1's TTL rung is only as good as this answer.
236+
5. ~~**Cancellation probe.**~~ **Done by M0-4, and the expectation was wrong**
237+
(`docs/M0-FINDINGS.md` §M0-4). Cancellation itself is the easy half: all three end the
238+
turn within 64 ms, emit nothing afterwards, and leave the session reusable — so the TTL
239+
rung gets a 2 s grace and never needs to kill. Two results this plan did not anticipate.
240+
**Copilot answers `end_turn` for a turn it cancelled**, contrary to ACP's MUST, so
241+
`stopReason` alone cannot classify an outcome and the supervisor must carry its own
242+
cancel intent. And **every harness accepts a second `session/prompt` mid-turn** — none
243+
rejects it, the in-flight turn dies, both requests answer `end_turn`, and on opencode and
244+
copilot the text lands in the running turn's context rather than being served
245+
(`github/copilot-cli#4555` reports the mechanism: `session/prompt` aborts the session
246+
unconditionally, even when idle). There is no safe way to nudge a turn in flight.
239247
6. **Usage reporting.** Record, per harness, whether `usage_update` arrives at all and what
240248
its fields actually mean. `{used, size}` is context occupancy, not spend — check whether
241249
anything reports cumulative or billable counts, or names the model it selected. This is
@@ -353,8 +361,12 @@ driven from a run file, fully verifiable offline.
353361
reuses the `toolCallId → {title, kind}` map already maintained as a protocol invariant, so
354362
it needs only a per-task "last title + consecutive count". Both signals emit a warning and
355363
the task keeps running; the separate, longer `ttlMs` calls `session/cancel`, preserves the
356-
branch and marks the task `timed_out`. Every knob disabled at `0`. Best-effort throughout —
357-
one task's failure must never halt the sweep. Two notes: there is no third rung, because
364+
branch and marks the task `timed_out`. **M0-4 supplies the missing number: wait 2 s after
365+
the cancel before force-killing** — thirty times the slowest cancel measured, and the same
366+
grace `run.mjs` already uses — and expect never to reach it. `timed_out` is set from the
367+
watchdog's own record, not from the stop reason, which may come back `end_turn`. Every
368+
knob disabled at `0`. Best-effort throughout — one task's failure must never halt the
369+
sweep. Two notes: there is no third rung, because
358370
ACP gives us no way to nudge a turn in flight, so detection reports rather than corrects;
359371
and a token-based signal was considered and dropped, because `usage_update` reports context
360372
occupancy in units that are not comparable between vendors.

‎docs/STORIES.md‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,15 @@ with what `stopReason`. Confirm the expectation that none accepts a second `sess
115115
while a turn is in flight.
116116
*Done when* the note states the observed stop reason and latency per harness — M1c-1's TTL
117117
rung is only as good as this answer.
118+
**Done 2026-08-21** (`docs/M0-FINDINGS.md` §M0-4). Three corrections to this text as
119+
written. The probe could not be a `run.mjs` run: its SIGINT path kills at 2 s
120+
(`run.mjs:81-88`) and `#failAll` (`acp.mjs:62-64`) would have turned any slower cancel into
121+
an exit error, so the answer needed tooling that cancels and waits — plus an **uninterrupted
122+
control run**, without which "the turn ended" cannot be told from "the turn was ending
123+
anyway". The expectation about a second `session/prompt` is **false on all three**: none
124+
rejects it, the in-flight turn dies, and both requests answer `end_turn`. And latency turned
125+
out not to be the interesting number — everything landed inside 64 ms — while
126+
**copilot reports `end_turn` for a turn it cancelled**, which is what actually moves M1b-2.
118127

119128
**M0-5 · Usage reporting survey · S · needs M0-1**
120129
Per harness: does `usage_update` arrive, and what do its fields mean? `{used, size}` is
@@ -148,7 +157,7 @@ default, and M3-5's terminal scope.
148157
M0-3/4/5 sections are stubbed with the partial input that survey already produced (no
149158
`terminal/*` traffic under `terminal: false`; copilot supplying real argv on the permission
150159
request; `usage_update` shapes, including `cost` on opencode alone).
151-
**M0-3 filled 2026-08-21.** Two sections remain: M0-4 and M0-5.
160+
**M0-3 and M0-4 filled 2026-08-21.** One section remains: M0-5.
152161

153162
---
154163

@@ -344,6 +353,15 @@ gates on this. **Decision — one table, in this module:**
344353
| `refusal` | `failed` | agent declined; `summary` carries its text |
345354
| `cancelled` | `timed_out` if the watchdog fired, else `cancelled` | the supervisor knows which |
346355

356+
**Revised by M0-4** — the table stands; its input does not. `cancelled` may never arrive:
357+
copilot answers `end_turn` for a turn it cancelled (both samples, against a control run
358+
proving the turn was cut short), contrary to ACP's MUST. And `end_turn` is now known to mean
359+
*finished*, *I was cancelled*, or *someone sent a second prompt and killed my turn*.
360+
**Key the status on the triple (`stopReason`, did we send a cancel, did the watchdog fire)**,
361+
with the supervisor's own record authoritative and `stopReason` consulted only where the two
362+
agree. A cancelled turn is also truncated mid-tool-call, so the "stopped mid-edit" hazard
363+
recorded above for `max_tokens` applies to `timed_out` too.
364+
347365
**Gap 2** — the plan gives no rule for what happens to dependents when a dependency fails,
348366
no rule for rolling task statuses into a run status, and its lifecycle
349367
(`pending → running → done | failed | timed_out | cancelled`) has no state for "never
@@ -446,9 +464,19 @@ already maintained as a protocol invariant, so it needs only a per-task last-tit
446464
count. Both signals warn and the task keeps running; the separate, longer `ttlMs` calls
447465
`session/cancel`, preserves the branch and marks the task `timed_out`. Every knob disabled
448466
at `0`. Best-effort throughout — one task's failure must never halt the sweep.
467+
**M0-4 fills in the TTL rung** (`docs/M0-FINDINGS.md` §M0-4): every harness ended its turn
468+
within 64 ms of `session/cancel`, sent nothing afterwards, and accepted a further prompt — so
469+
**wait 2 s after cancelling, then force-kill**, and expect never to reach it. Mark
470+
`timed_out` from the watchdog's own record: copilot's cancelled turns come back `end_turn`.
449471
*Done when* `node:test` covers the repeat counter's reset behaviour and one task throwing
450472
inside the sweep does not stop the others being checked. There is deliberately no third
451473
rung: ACP gives no way to nudge a turn in flight, so detection reports rather than corrects.
474+
**M0-4 turned that into a hard rule.** A second `session/prompt` is accepted by all three
475+
harnesses and kills the running turn; on opencode and copilot its text is merged into that
476+
turn's context, where both models read it as a possible prompt injection. Copilot aborts the
477+
session on *any* prompt, even one sent while idle
478+
([github/copilot-cli#4555](https://github.com/github/copilot-cli/issues/4555)). One prompt
479+
per session per task.
452480
A token-based signal was considered and dropped — `usage_update` reports context occupancy
453481
in units that are not comparable between vendors.
454482

0 commit comments

Comments
 (0)