Repository navigation
M0-4 · Cancellation probe findings - #56
Merged
Merged
Conversation
Fills docs/M0-FINDINGS.md §M0-4 and folds the results back into the plan and the stories. Method: cancel (and a second in-flight prompt) fired on the first tool_call, two samples per harness, each against an uninterrupted control run — without the control, "the turn ended" cannot be told from "the turn was ending anyway". opencode 1.18.18, claude-agent-acp 0.49.0, copilot 1.0.80. Cancellation is the easy half: every harness ended its turn within 64 ms, sent nothing afterwards, and accepted a further prompt. M1c-1 therefore gets a 2 s grace and never needs to kill. Two results the plan did not anticipate: - copilot answers `end_turn` for a turn it cancelled, contrary to ACP's MUST and undocumented by GitHub. `stopReason` alone cannot classify an outcome; M1b-2 must key on (stopReason, did we cancel, did the watchdog fire). - every harness accepts a second `session/prompt` mid-turn. None rejects it, the in-flight turn dies, both requests answer `end_turn`, and on opencode and copilot the text is merged into the running turn's context, where both models read it as a possible prompt injection. github/copilot-cli#4555 has the mechanism. One prompt per session per task; there is no safe nudge. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_013ZVZAP4MdJ69xjyDHg6RVx
…cabulary is intact The two acceptance criteria on #5 asked for both explicitly: a recorded 'did not end within N s' where a harness hit the bound, and a plain statement of whether the observed literals fall inside M1b-2's set. They do — the failure is semantic, not a vocabulary error. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_013ZVZAP4MdJ69xjyDHg6RVx
Closed
github/copilot-cli#4561, with a zero-dependency repro and the control run as evidence the turn really was cut short. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_013ZVZAP4MdJ69xjyDHg6RVx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5.
Fills
docs/M0-FINDINGS.md§M0-4 and folds the results intoPROJECT-PLAN.mdandSTORIES.md. Probe tooling is onspike/m0-4-cancel-probe(never merged), as M0-3's stubs were.Method
Three runs per harness, each in its own workspace,
--mode auto, prompt "read every file one at a time and summarise it" — cancel (or the second prompt) fired on the firsttool_call, a protocol event rather than a wall clock, so the three are cut at a comparable point:session/cancel, then wait (notrun.mjs, whose SIGINT path kills at 2 s and would have lost the stop reason), then a follow-up prompt to test session survival. Two samples each.session/promptin flight, awaited to completion, marker-tagged.Cancellation works, and is not the interesting part
stopReasoncancelledcancelledend_turnM1c-1 gets its missing number: a 2 s grace after
session/cancel, then force-kill — 30× the slowest sample, the same gracerun.mjsalready uses, and expected never to be reached.Two results the plan did not anticipate
1. Copilot reports
end_turnfor a turn it cancelled (both samples). The control settles it: 11 events and ~5.6 s against 69 and 36.3 s, with the triggeringtool_callnever completing. It obeyed and mislabelled the result. Contrary to ACP's MUST ("the Agent MUST respond … with thecancelledstop reason"), and undocumented — GitHub's ACP server reference namesstopReasononly inside an example, and nogithub/copilot-cliissue reports it. The same defect was filed against opencode (anomalyco/opencode#25899, closed as not planned), yet opencode answerscancelledhere — so versions, not trackers, decide. Every row above is version-pinned for that reason.2. Every harness accepts a second
session/promptmid-turn. The story's expectation is false three different ways, and the failure is silent each time: nobody rejects it, the in-flight turn dies, and both requests answerend_turn. On opencode and copilot the text is merged into the running turn's context — both models read it as a possible prompt injection — rather than served. github/copilot-cli#4555 has the mechanism:session/promptcallssession.abort()unconditionally, even when idle.So
end_turnnow means finished, I was cancelled, or someone interrupted me, depending on harness and circumstance.Fold-backs
STORIES.mdM1b-2 — the stop-reason table stands; its input does not. Key status on the triple (stopReason, did we cancel, did the watchdog fire), supervisor's record authoritative. Cancelled turns truncate mid-tool-call, somax_tokens's "stopped mid-edit" hazard applies totimed_out.STORIES.mdM1c-1 — 2 s grace;timed_outset from the watchdog's own record; "no third rung" upgraded from a design preference to a measured rule: one prompt per session per task.PROJECT-PLAN.md— Liveness row in the decisions table, agenda item 5 struck, watchdog module bullet carries the grace.STORIES.mdM0-4 / M0-7 — marked done; M0-5 is the last stub.Not covered
Two samples per harness for cancellation, one apiece for the other questions, one prompt shape, nothing written to disk during the runs — so cancelling mid-edit is untested and named in the note as the next probe.
omp,pi,geminiremain unknown by absence.