Skip to content

M0-4 · Cancellation probe findings - #56

Merged
EdwardLiuyc merged 3 commits into
masterfrom
m0-4-cancellation-probe
Aug 22, 2026
Merged

EdwardLiuyc merged 3 commits into
masterfrom
m0-4-cancellation-probe

Conversation

@EdwardLiuyc

Copy link
Copy Markdown
Owner

Closes #5.

Fills docs/M0-FINDINGS.md §M0-4 and folds the results into PROJECT-PLAN.md and STORIES.md. Probe tooling is on spike/m0-4-cancel-probe (never merged), as M0-3's stubs were.

Method

Three runs per harness, each in its own workspace, --mode auto, prompt "read every file one at a time and summarise it" — cancel (or the second prompt) fired on the first tool_call, a protocol event rather than a wall clock, so the three are cut at a comparable point:

  1. observe — uninterrupted control. Without it, "the turn ended" cannot be told from "the turn was ending anyway", and the central finding below is unprovable.
  2. cancel — session/cancel, then wait (not run.mjs, whose SIGINT path kills at 2 s and would have lost the stop reason), then a follow-up prompt to test session survival. Two samples each.
  3. double — a second session/prompt in flight, awaited to completion, marker-tagged.

Cancellation works, and is not the interesting part

opencode 1.18.18 claude-agent-acp 0.49.0 copilot 1.0.80
stopReason cancelled cancelled end_turn
latency (2 samples) 64 / 45 ms 5 / 10 ms 26 / 28 ms
events, cancelled vs control 20 · 16 vs 443 14 · 14 vs 180 11 · 11 vs 69
events after the response 0 0 0
session usable afterwards yes yes yes

M1c-1 gets its missing number: a 2 s grace after session/cancel, then force-kill — 30× the slowest sample, the same grace run.mjs already uses, and expected never to be reached.

Two results the plan did not anticipate

1. Copilot reports end_turn for a turn it cancelled (both samples). The control settles it: 11 events and ~5.6 s against 69 and 36.3 s, with the triggering tool_call never completing. It obeyed and mislabelled the result. Contrary to ACP's MUST ("the Agent MUST respond … with the cancelled stop reason"), and undocumented — GitHub's ACP server reference names stopReason only inside an example, and no github/copilot-cli issue reports it. The same defect was filed against opencode (anomalyco/opencode#25899, closed as not planned), yet opencode answers cancelled here — so versions, not trackers, decide. Every row above is version-pinned for that reason.

2. Every harness accepts a second session/prompt mid-turn. The story's expectation is false three different ways, and the failure is silent each time: nobody rejects it, the in-flight turn dies, and both requests answer end_turn. On opencode and copilot the text is merged into the running turn's context — both models read it as a possible prompt injection — rather than served. github/copilot-cli#4555 has the mechanism: session/prompt calls session.abort() unconditionally, even when idle.

So end_turn now means finished, I was cancelled, or someone interrupted me, depending on harness and circumstance.

Fold-backs

  • STORIES.md M1b-2 — the stop-reason table stands; its input does not. Key status on the triple (stopReason, did we cancel, did the watchdog fire), supervisor's record authoritative. Cancelled turns truncate mid-tool-call, so max_tokens's "stopped mid-edit" hazard applies to timed_out.
  • STORIES.md M1c-1 — 2 s grace; timed_out set from the watchdog's own record; "no third rung" upgraded from a design preference to a measured rule: one prompt per session per task.
  • PROJECT-PLAN.md — Liveness row in the decisions table, agenda item 5 struck, watchdog module bullet carries the grace.
  • STORIES.md M0-4 / M0-7 — marked done; M0-5 is the last stub.

Not covered

Two samples per harness for cancellation, one apiece for the other questions, one prompt shape, nothing written to disk during the runs — so cancelling mid-edit is untested and named in the note as the next probe. omp, pi, gemini remain unknown by absence.

EdwardLiuyc and others added 2 commits August 21, 2026 23:32
Fills docs/M0-FINDINGS.md §M0-4 and folds the results back into the plan and
the stories. Method: cancel (and a second in-flight prompt) fired on the first
tool_call, two samples per harness, each against an uninterrupted control run —
without the control, "the turn ended" cannot be told from "the turn was ending
anyway". opencode 1.18.18, claude-agent-acp 0.49.0, copilot 1.0.80.

Cancellation is the easy half: every harness ended its turn within 64 ms, sent
nothing afterwards, and accepted a further prompt. M1c-1 therefore gets a 2 s
grace and never needs to kill.

Two results the plan did not anticipate:

- copilot answers `end_turn` for a turn it cancelled, contrary to ACP's MUST
  and undocumented by GitHub. `stopReason` alone cannot classify an outcome;
  M1b-2 must key on (stopReason, did we cancel, did the watchdog fire).
- every harness accepts a second `session/prompt` mid-turn. None rejects it,
  the in-flight turn dies, both requests answer `end_turn`, and on opencode and
  copilot the text is merged into the running turn's context, where both models
  read it as a possible prompt injection. github/copilot-cli#4555 has the
  mechanism. One prompt per session per task; there is no safe nudge.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_013ZVZAP4MdJ69xjyDHg6RVx
…cabulary is intact

The two acceptance criteria on #5 asked for both explicitly: a recorded
'did not end within N s' where a harness hit the bound, and a plain statement of
whether the observed literals fall inside M1b-2's set. They do — the failure is
semantic, not a vocabulary error.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_013ZVZAP4MdJ69xjyDHg6RVx
@EdwardLiuyc
EdwardLiuyc merged commit 7f79a37 into master Aug 22, 2026
@EdwardLiuyc
EdwardLiuyc deleted the m0-4-cancellation-probe branch August 22, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

M0-4 · Cancellation probe

1 participant