1313
1414What this means for the auto-yes mechanic the x-x skills rely on:
1515
16- The x-x and x-plan skills are designed to pause and ask the user to "Reply
17- yes to proceed" at gate points. Claude's driver handles that with a
18- JSON-Lines auto-yes loop on a single long-lived process. Copilot CLI ships
19- the equivalent as a single command-line flag: `--no-ask-user` prevents the
20- agent from pausing for clarification, and `--allow-all-tools` waives tool
21- permission prompts. With both flags set, Copilot proceeds through the
22- skill without pausing — replicating the "always reply yes" behavior in one
23- invocation instead of a turn loop.
24-
25- Flags exercised here (all documented in the reference above):
16+ The x-x and x-plan skills pause at "Reply `yes` to proceed" gates. We
17+ originally hoped `--no-ask-user` would replace the auto-yes loop, but
18+ empirical evidence from the first manual eval run shows it's a hint,
19+ not a hard constraint: the agent decides per-turn whether to honor the
20+ gate. Run 26432641287 had three of four turns proceed autonomously and
21+ one stop dead at "Reply yes" — workspace state goes wrong, downstream
22+ turns see an empty `.x-plans/`.
23+
24+ The fix is a continuation loop modeled on the Claude driver: when one
25+ turn ends with the "Reply yes" prompt in its captured stdout, run
26+ `copilot --continue -p "yes"` to resume the same session and let the
27+ agent move past the gate. `--continue` picks up the most recently
28+ closed local session for the user (per the Copilot CLI reference) —
29+ safe here because the eval suite runs sequentially.
30+
31+ Flags exercised (all documented in the reference above):
2632 -p <prompt> Run the prompt non-interactively and exit.
33+ --continue Resume the most recently closed local session.
2734 --allow-all-tools Skip per-tool permission prompts.
28- --no-ask-user Prevent the agent from asking clarifying questions.
35+ --no-ask-user Prevent the agent from asking clarifying questions
36+ (best-effort — see note above).
2937 --add-dir <dir> Add a directory to the allowed-paths list.
3038 --model <id> Pick the model (we route to deepseek-v4-pro via the
3139 Anthropic-compatible BYOK env vars below).
3442flags — COPILOT_PROVIDER_TYPE=anthropic, COPILOT_PROVIDER_BASE_URL=
3543https://api.deepseek.com/anthropic, COPILOT_PROVIDER_API_KEY, COPILOT_MODEL.
3644
37- Known gaps to discover in CI (do not paper over — fail loudly per the
38- agent-eval recipe in docs/internal/adding-agent-eval-backend.md):
39-
40- 1. Skill loading. `x-x init --agents` does not yet accept `copilot` (the
41- `agentTargets` registry in constants.go ships claude + codex only).
42- Tests in this suite call `x-x init --agents claude`, which writes skills
43- to `.claude/skills/`. Whether Copilot CLI discovers slash commands
44- under that path, under its own convention, or not at all, is the first
45- thing the workflow run will reveal. The smoke test does not depend on
46- skills and is the canary.
47-
48- 2. Wire format. There is no public `--output-format stream-json` analogue.
49- We capture stdout as the transcript (plain text) and log every line.
50- If a future Copilot version adds structured output, swap the line
51- reader for a JSON parser.
52-
53- 3. Multi-turn. Each skill invocation is its own process. The workspace
54- filesystem (plan files, produced artifacts) is the only state that
55- carries between turns. That matches the x-x flow — `/x-plan` writes
56- files; `/x-x` reads them — but it does mean we cannot pass intermediate
57- conversation context between calls.
45+ Provider type MUST be `anthropic` (not `openai`) — DeepSeek requires
46+ reasoning_content echo-back on subsequent requests, which Copilot CLI's
47+ OpenAI integration does not support. The Anthropic Messages wire avoids
48+ the issue.
5849
5950Logging policy (non-negotiable, same as the Claude driver): every state
6051transition, every external call, every line of the agent's stderr lands
6556
6657import os
6758import queue
59+ import re
6860import shutil
6961import subprocess
7062import threading
7567
7668from ._logging import log
7769
78- DEFAULT_MAX_TURNS = 1
70+ CONFIRMATION_PATTERN = re .compile (r"reply\s+`?yes`?" , re .IGNORECASE )
71+
72+ DEFAULT_MAX_TURNS = 10
7973DEFAULT_PER_TURN_TIMEOUT_S = 600.0
8074
8175# Echoed at driver startup so CI logs show exactly which backend Copilot
10094class SkillRun :
10195 """Mirror of claude_driver.SkillRun for cross-driver test parity.
10296
103- Field semantics differ slightly because the wire is different :
104- - `turns` is always 0 or 1 — Copilot CLI is single-shot per invocation .
105- - `yes_replies` is always 0 — `--no-ask-user` replaces the auto- yes
106- loop with a single flag .
107- - `events_received` counts stdout lines (plain text), not JSON events .
97+ Field semantics:
98+ - `turns` counts Copilot invocations (initial + each `--continue` reply) .
99+ - `yes_replies` counts how often the driver had to feed " yes" to a
100+ gate the agent emitted despite `--no-ask-user` .
101+ - `events_received` counts stdout lines across all turns .
108102 - `transcript` holds raw stdout lines wrapped in `{"line": "..."}`
109- dicts so callers that walk the transcript get the same shape as
110- Claude's JSONL stream.
103+ dicts. Turn boundaries are marked with sentinel
104+ `{"line": "--- turn N ---"}` entries so the post-mortem can tell
105+ which output came from which invocation.
111106 """
112107
113108 workspace : Path
@@ -137,29 +132,121 @@ def drive_skill(
137132 per_turn_timeout : float = DEFAULT_PER_TURN_TIMEOUT_S ,
138133 transcript_path : Path | None = None ,
139134) -> SkillRun :
140- """Run one skill in `workspace` via a single `copilot -p` invocation .
135+ """Run one skill in `workspace`, auto-resuming on "Reply yes" gates .
141136
142- `max_turns` is accepted for signature parity with claude_driver and
143- ignored beyond a sanity assertion — Copilot CLI is single-shot.
137+ Turn 1: `copilot -p <initial_prompt> --allow-all-tools --no-ask-user`.
138+ Turns 2..max_turns: `copilot --continue -p yes --allow-all-tools
139+ --no-ask-user`, but only when the previous turn's stdout matched
140+ the "Reply yes" gate pattern. Otherwise the loop exits.
144141 """
145142 if max_turns < 1 :
146143 raise ValueError (f"max_turns must be >= 1, got { max_turns } " )
147144
148145 run = SkillRun (workspace = workspace , initial_prompt = initial_prompt )
149- _log_startup (workspace , initial_prompt , per_turn_timeout )
146+ _log_startup (workspace , initial_prompt , per_turn_timeout , max_turns )
147+
148+ loop_start = time .time ()
149+ turn_prompt = initial_prompt
150+ use_continue = False
151+
152+ while run .turns < max_turns :
153+ run .transcript .append ({"line" : f"--- turn { run .turns + 1 } ---" })
154+ turn_output , exit_code , stderr_tail , timed_out = _run_one_turn (
155+ workspace ,
156+ turn_prompt ,
157+ per_turn_timeout = per_turn_timeout ,
158+ use_continue = use_continue ,
159+ )
160+ for line in turn_output :
161+ run .transcript .append ({"line" : line })
162+ run .events_received += len (turn_output )
163+ run .turns += 1
164+ run .exit_code = exit_code
165+ run .stderr_tail = stderr_tail
166+ if timed_out :
167+ run .timed_out = True
168+ log (
169+ "driver" ,
170+ f"turn { run .turns } TIMED OUT — abandoning continuation loop" ,
171+ )
172+ break
173+ if exit_code != 0 :
174+ log (
175+ "driver" ,
176+ f"turn { run .turns } exited { exit_code } — abandoning continuation loop" ,
177+ )
178+ break
179+
180+ last_text = "\n " .join (turn_output [- 40 :])
181+ if not _asks_for_confirmation (last_text ):
182+ log (
183+ "driver" ,
184+ f"turn { run .turns } ended without 'Reply yes' gate — session done" ,
185+ )
186+ run .completed = True
187+ break
150188
151- cmd = [
152- "copilot" ,
153- "-p" , initial_prompt ,
189+ log (
190+ "driver" ,
191+ f"turn { run .turns } ended at 'Reply yes' gate — resuming with --continue" ,
192+ )
193+ run .yes_replies += 1
194+ turn_prompt = "yes"
195+ use_continue = True
196+ else :
197+ # while-else fires when the loop's condition (turns < max_turns)
198+ # becomes false without a break. We hit the turn cap while the agent
199+ # was still gating — surface that loudly, not silently as "completed".
200+ log (
201+ "driver" ,
202+ f"hit max_turns={ max_turns } with the gate still firing — stopping" ,
203+ )
204+
205+ log (
206+ "driver" ,
207+ f"drive_skill done: turns={ run .turns } yes_replies={ run .yes_replies } "
208+ f"lines={ run .events_received } exit_code={ run .exit_code } "
209+ f"completed={ run .completed } timed_out={ run .timed_out } "
210+ f"elapsed={ time .time () - loop_start :.1f} s" ,
211+ )
212+
213+ if transcript_path is not None :
214+ run .save_transcript (transcript_path )
215+ log ("driver" , f"transcript written to { transcript_path } " )
216+
217+ return run
218+
219+
220+ def _run_one_turn (
221+ workspace : Path ,
222+ prompt : str ,
223+ * ,
224+ per_turn_timeout : float ,
225+ use_continue : bool ,
226+ ) -> tuple [list [str ], int | None , str , bool ]:
227+ """Spawn one `copilot` invocation and return (stdout_lines, exit_code,
228+ stderr_tail, timed_out).
229+
230+ Each call is a fresh subprocess — Copilot CLI is single-shot per
231+ invocation. `use_continue=True` adds `--continue` so the process
232+ resumes the previous session's conversation history instead of
233+ starting fresh.
234+ """
235+ cmd = ["copilot" ]
236+ if use_continue :
237+ cmd .append ("--continue" )
238+ cmd .extend ([
239+ "-p" , prompt ,
154240 "--allow-all-tools" ,
155241 "--no-ask-user" ,
156242 "--add-dir" , str (workspace ),
157- ]
243+ ])
158244 model = os .environ .get ("COPILOT_MODEL" )
159245 if model :
160246 cmd .extend (["--model" , model ])
161247
162- log ("driver" , f"spawn: { ' ' .join (cmd )} " )
248+ log ("driver" , f"spawn: { ' ' .join (cmd [:3 ])} ... (--continue={ use_continue } )" )
249+ log ("driver" , f"prompt: { _brief (prompt , 200 )} " )
163250 log ("driver" , f"cwd: { workspace } " )
164251
165252 proc = subprocess .Popen (
@@ -182,68 +269,63 @@ def drive_skill(
182269 target = _pump_stderr_live , args = (proc .stderr , err_lines ), daemon = True
183270 ).start ()
184271
185- loop_start = time .time ()
272+ stdout_lines : list [str ] = []
273+ turn_start = time .time ()
274+ timed_out = False
186275
187276 try :
188277 while True :
189278 try :
190279 line = out_q .get (timeout = per_turn_timeout )
191280 except queue .Empty :
192- run . timed_out = True
281+ timed_out = True
193282 log (
194283 "driver" ,
195284 f"TIMEOUT after { per_turn_timeout } s waiting for next stdout line "
196- f"(lines seen : { run . events_received } , "
197- f"elapsed since spawn : { time .time () - loop_start :.1f} s)" ,
285+ f"(lines this turn : { len ( stdout_lines ) } , "
286+ f"elapsed: { time .time () - turn_start :.1f} s)" ,
198287 )
199288 break
200289
201290 if line is None :
202291 log (
203292 "driver" ,
204- f"stdout EOF (copilot exited); lines seen : { run . events_received } " ,
293+ f"stdout EOF (copilot exited); lines this turn : { len ( stdout_lines ) } " ,
205294 )
206- run .completed = True
207- run .turns = 1
208295 break
209296
210- run .transcript .append ({"line" : line })
211- run .events_received += 1
297+ stdout_lines .append (line )
212298 if line .strip ():
213- log ("driver" , f"stdout #{ run .events_received } : { _brief (line , 200 )} " )
214-
299+ log ("driver" , f"stdout #{ len (stdout_lines )} : { _brief (line , 200 )} " )
215300 finally :
216301 try :
217302 proc .wait (timeout = 15 )
218303 except subprocess .TimeoutExpired :
219304 log ("driver" , "copilot did not exit within 15s; killing" )
220305 proc .kill ()
221306 proc .wait (timeout = 5 )
222- run .exit_code = proc .returncode
223- run .stderr_tail = "\n " .join (err_lines [- 40 :])
224- log (
225- "driver" ,
226- f"copilot exited code={ run .exit_code } "
227- f"lines={ run .events_received } "
228- f"completed={ run .completed } timed_out={ run .timed_out } "
229- f"elapsed={ time .time () - loop_start :.1f} s" ,
230- )
231-
232- if transcript_path is not None :
233- run .save_transcript (transcript_path )
234- log ("driver" , f"transcript written to { transcript_path } " )
235307
236- return run
308+ stderr_tail = "\n " .join (err_lines [- 40 :])
309+ log (
310+ "driver" ,
311+ f"turn done: exit={ proc .returncode } lines={ len (stdout_lines )} "
312+ f"timed_out={ timed_out } elapsed={ time .time () - turn_start :.1f} s" ,
313+ )
314+ return stdout_lines , proc .returncode , stderr_tail , timed_out
237315
238316
239317def _log_startup (
240318 workspace : Path ,
241319 initial_prompt : str ,
242320 per_turn_timeout : float ,
321+ max_turns : int ,
243322) -> None :
244323 log ("driver" , f"drive_skill called: workspace={ workspace } " )
245324 log ("driver" , f"initial_prompt: { _brief (initial_prompt , 200 )} " )
246- log ("driver" , f"per_turn_timeout={ per_turn_timeout } s" )
325+ log (
326+ "driver" ,
327+ f"per_turn_timeout={ per_turn_timeout } s max_turns={ max_turns } " ,
328+ )
247329
248330 copilot_path = shutil .which ("copilot" )
249331 log ("driver" , f"copilot on PATH: { copilot_path } " )
@@ -283,6 +365,10 @@ def _pump_stderr_live(stream: IO[str], sink: list[str]) -> None:
283365 log ("driver" , f"stderr: { s } " )
284366
285367
368+ def _asks_for_confirmation (text : str ) -> bool :
369+ return bool (CONFIRMATION_PATTERN .search (text or "" ))
370+
371+
286372def _brief (text : str , limit : int = 140 ) -> str :
287373 collapsed = " " .join ((text or "" ).split ())
288374 return collapsed if len (collapsed ) <= limit else collapsed [: limit - 1 ] + "…"
0 commit comments