Skip to content

Commit ad34581

Browse files
committed
fix: add auto-yes continuation loop to copilot driver
The first manual eval run (workflow 26432641287) failed on the supersedes scenario because the agent honored the skill's "Reply yes to proceed" gate on some turns and bypassed it on others — `--no-ask-user` is a hint, not a hard constraint. Three of four test_copilot_reminders_supersedes_todo turns stopped dead at the gate; the workspace ended up with one plan instead of two, and the supersede mechanic never fired. Fix: model the loop on claude_driver. - Initial turn runs `copilot -p <prompt> --allow-all-tools --no-ask-user`. - If the captured stdout matches the "Reply yes" pattern, the next turn runs `copilot --continue -p yes --allow-all-tools --no-ask-user` to resume the same session with "yes" as the user's reply. - Loop bounded by DEFAULT_MAX_TURNS=10 (was 1); exits early when a turn ends without the gate or when copilot returns non-zero. - `--continue` is safe because the eval suite runs sequentially — per the official reference, it resumes the most recently closed local session for the user. Transcript entries get `--- turn N ---` boundary markers so the post-mortem can tell which output came from which invocation. Smoke test still completes in 1 turn (no skill, no gate); scenario tests now have headroom for multi-turn auto-yes recovery. Signed-off-by: Tanzim Saqib <[email protected]>
1 parent ef7a35e commit ad34581

1 file changed

Lines changed: 161 additions & 75 deletions

File tree

‎skills-evals/src/skills_evals/copilot_driver.py‎

Lines changed: 161 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -13,19 +13,27 @@
1313
1414
What this means for the auto-yes mechanic the x-x skills rely on:
1515
16-
The x-x and x-plan skills are designed to pause and ask the user to "Reply
17-
yes to proceed" at gate points. Claude's driver handles that with a
18-
JSON-Lines auto-yes loop on a single long-lived process. Copilot CLI ships
19-
the equivalent as a single command-line flag: `--no-ask-user` prevents the
20-
agent from pausing for clarification, and `--allow-all-tools` waives tool
21-
permission prompts. With both flags set, Copilot proceeds through the
22-
skill without pausing — replicating the "always reply yes" behavior in one
23-
invocation instead of a turn loop.
24-
25-
Flags exercised here (all documented in the reference above):
16+
The x-x and x-plan skills pause at "Reply `yes` to proceed" gates. We
17+
originally hoped `--no-ask-user` would replace the auto-yes loop, but
18+
empirical evidence from the first manual eval run shows it's a hint,
19+
not a hard constraint: the agent decides per-turn whether to honor the
20+
gate. Run 26432641287 had three of four turns proceed autonomously and
21+
one stop dead at "Reply yes" — workspace state goes wrong, downstream
22+
turns see an empty `.x-plans/`.
23+
24+
The fix is a continuation loop modeled on the Claude driver: when one
25+
turn ends with the "Reply yes" prompt in its captured stdout, run
26+
`copilot --continue -p "yes"` to resume the same session and let the
27+
agent move past the gate. `--continue` picks up the most recently
28+
closed local session for the user (per the Copilot CLI reference) —
29+
safe here because the eval suite runs sequentially.
30+
31+
Flags exercised (all documented in the reference above):
2632
-p <prompt> Run the prompt non-interactively and exit.
33+
--continue Resume the most recently closed local session.
2734
--allow-all-tools Skip per-tool permission prompts.
28-
--no-ask-user Prevent the agent from asking clarifying questions.
35+
--no-ask-user Prevent the agent from asking clarifying questions
36+
(best-effort — see note above).
2937
--add-dir <dir> Add a directory to the allowed-paths list.
3038
--model <id> Pick the model (we route to deepseek-v4-pro via the
3139
Anthropic-compatible BYOK env vars below).
@@ -34,27 +42,10 @@
3442
flags — COPILOT_PROVIDER_TYPE=anthropic, COPILOT_PROVIDER_BASE_URL=
3543
https://api.deepseek.com/anthropic, COPILOT_PROVIDER_API_KEY, COPILOT_MODEL.
3644
37-
Known gaps to discover in CI (do not paper over — fail loudly per the
38-
agent-eval recipe in docs/internal/adding-agent-eval-backend.md):
39-
40-
1. Skill loading. `x-x init --agents` does not yet accept `copilot` (the
41-
`agentTargets` registry in constants.go ships claude + codex only).
42-
Tests in this suite call `x-x init --agents claude`, which writes skills
43-
to `.claude/skills/`. Whether Copilot CLI discovers slash commands
44-
under that path, under its own convention, or not at all, is the first
45-
thing the workflow run will reveal. The smoke test does not depend on
46-
skills and is the canary.
47-
48-
2. Wire format. There is no public `--output-format stream-json` analogue.
49-
We capture stdout as the transcript (plain text) and log every line.
50-
If a future Copilot version adds structured output, swap the line
51-
reader for a JSON parser.
52-
53-
3. Multi-turn. Each skill invocation is its own process. The workspace
54-
filesystem (plan files, produced artifacts) is the only state that
55-
carries between turns. That matches the x-x flow — `/x-plan` writes
56-
files; `/x-x` reads them — but it does mean we cannot pass intermediate
57-
conversation context between calls.
45+
Provider type MUST be `anthropic` (not `openai`) — DeepSeek requires
46+
reasoning_content echo-back on subsequent requests, which Copilot CLI's
47+
OpenAI integration does not support. The Anthropic Messages wire avoids
48+
the issue.
5849
5950
Logging policy (non-negotiable, same as the Claude driver): every state
6051
transition, every external call, every line of the agent's stderr lands
@@ -65,6 +56,7 @@
6556

6657
import os
6758
import queue
59+
import re
6860
import shutil
6961
import subprocess
7062
import threading
@@ -75,7 +67,9 @@
7567

7668
from ._logging import log
7769

78-
DEFAULT_MAX_TURNS = 1
70+
CONFIRMATION_PATTERN = re.compile(r"reply\s+`?yes`?", re.IGNORECASE)
71+
72+
DEFAULT_MAX_TURNS = 10
7973
DEFAULT_PER_TURN_TIMEOUT_S = 600.0
8074

8175
# Echoed at driver startup so CI logs show exactly which backend Copilot
@@ -100,14 +94,15 @@
10094
class SkillRun:
10195
"""Mirror of claude_driver.SkillRun for cross-driver test parity.
10296
103-
Field semantics differ slightly because the wire is different:
104-
- `turns` is always 0 or 1 — Copilot CLI is single-shot per invocation.
105-
- `yes_replies` is always 0 — `--no-ask-user` replaces the auto-yes
106-
loop with a single flag.
107-
- `events_received` counts stdout lines (plain text), not JSON events.
97+
Field semantics:
98+
- `turns` counts Copilot invocations (initial + each `--continue` reply).
99+
- `yes_replies` counts how often the driver had to feed "yes" to a
100+
gate the agent emitted despite `--no-ask-user`.
101+
- `events_received` counts stdout lines across all turns.
108102
- `transcript` holds raw stdout lines wrapped in `{"line": "..."}`
109-
dicts so callers that walk the transcript get the same shape as
110-
Claude's JSONL stream.
103+
dicts. Turn boundaries are marked with sentinel
104+
`{"line": "--- turn N ---"}` entries so the post-mortem can tell
105+
which output came from which invocation.
111106
"""
112107

113108
workspace: Path
@@ -137,29 +132,121 @@ def drive_skill(
137132
per_turn_timeout: float = DEFAULT_PER_TURN_TIMEOUT_S,
138133
transcript_path: Path | None = None,
139134
) -> SkillRun:
140-
"""Run one skill in `workspace` via a single `copilot -p` invocation.
135+
"""Run one skill in `workspace`, auto-resuming on "Reply yes" gates.
141136
142-
`max_turns` is accepted for signature parity with claude_driver and
143-
ignored beyond a sanity assertion — Copilot CLI is single-shot.
137+
Turn 1: `copilot -p <initial_prompt> --allow-all-tools --no-ask-user`.
138+
Turns 2..max_turns: `copilot --continue -p yes --allow-all-tools
139+
--no-ask-user`, but only when the previous turn's stdout matched
140+
the "Reply yes" gate pattern. Otherwise the loop exits.
144141
"""
145142
if max_turns < 1:
146143
raise ValueError(f"max_turns must be >= 1, got {max_turns}")
147144

148145
run = SkillRun(workspace=workspace, initial_prompt=initial_prompt)
149-
_log_startup(workspace, initial_prompt, per_turn_timeout)
146+
_log_startup(workspace, initial_prompt, per_turn_timeout, max_turns)
147+
148+
loop_start = time.time()
149+
turn_prompt = initial_prompt
150+
use_continue = False
151+
152+
while run.turns < max_turns:
153+
run.transcript.append({"line": f"--- turn {run.turns + 1} ---"})
154+
turn_output, exit_code, stderr_tail, timed_out = _run_one_turn(
155+
workspace,
156+
turn_prompt,
157+
per_turn_timeout=per_turn_timeout,
158+
use_continue=use_continue,
159+
)
160+
for line in turn_output:
161+
run.transcript.append({"line": line})
162+
run.events_received += len(turn_output)
163+
run.turns += 1
164+
run.exit_code = exit_code
165+
run.stderr_tail = stderr_tail
166+
if timed_out:
167+
run.timed_out = True
168+
log(
169+
"driver",
170+
f"turn {run.turns} TIMED OUT — abandoning continuation loop",
171+
)
172+
break
173+
if exit_code != 0:
174+
log(
175+
"driver",
176+
f"turn {run.turns} exited {exit_code} — abandoning continuation loop",
177+
)
178+
break
179+
180+
last_text = "\n".join(turn_output[-40:])
181+
if not _asks_for_confirmation(last_text):
182+
log(
183+
"driver",
184+
f"turn {run.turns} ended without 'Reply yes' gate — session done",
185+
)
186+
run.completed = True
187+
break
150188

151-
cmd = [
152-
"copilot",
153-
"-p", initial_prompt,
189+
log(
190+
"driver",
191+
f"turn {run.turns} ended at 'Reply yes' gate — resuming with --continue",
192+
)
193+
run.yes_replies += 1
194+
turn_prompt = "yes"
195+
use_continue = True
196+
else:
197+
# while-else fires when the loop's condition (turns < max_turns)
198+
# becomes false without a break. We hit the turn cap while the agent
199+
# was still gating — surface that loudly, not silently as "completed".
200+
log(
201+
"driver",
202+
f"hit max_turns={max_turns} with the gate still firing — stopping",
203+
)
204+
205+
log(
206+
"driver",
207+
f"drive_skill done: turns={run.turns} yes_replies={run.yes_replies} "
208+
f"lines={run.events_received} exit_code={run.exit_code} "
209+
f"completed={run.completed} timed_out={run.timed_out} "
210+
f"elapsed={time.time() - loop_start:.1f}s",
211+
)
212+
213+
if transcript_path is not None:
214+
run.save_transcript(transcript_path)
215+
log("driver", f"transcript written to {transcript_path}")
216+
217+
return run
218+
219+
220+
def _run_one_turn(
221+
workspace: Path,
222+
prompt: str,
223+
*,
224+
per_turn_timeout: float,
225+
use_continue: bool,
226+
) -> tuple[list[str], int | None, str, bool]:
227+
"""Spawn one `copilot` invocation and return (stdout_lines, exit_code,
228+
stderr_tail, timed_out).
229+
230+
Each call is a fresh subprocess — Copilot CLI is single-shot per
231+
invocation. `use_continue=True` adds `--continue` so the process
232+
resumes the previous session's conversation history instead of
233+
starting fresh.
234+
"""
235+
cmd = ["copilot"]
236+
if use_continue:
237+
cmd.append("--continue")
238+
cmd.extend([
239+
"-p", prompt,
154240
"--allow-all-tools",
155241
"--no-ask-user",
156242
"--add-dir", str(workspace),
157-
]
243+
])
158244
model = os.environ.get("COPILOT_MODEL")
159245
if model:
160246
cmd.extend(["--model", model])
161247

162-
log("driver", f"spawn: {' '.join(cmd)}")
248+
log("driver", f"spawn: {' '.join(cmd[:3])} ... (--continue={use_continue})")
249+
log("driver", f"prompt: {_brief(prompt, 200)}")
163250
log("driver", f"cwd: {workspace}")
164251

165252
proc = subprocess.Popen(
@@ -182,68 +269,63 @@ def drive_skill(
182269
target=_pump_stderr_live, args=(proc.stderr, err_lines), daemon=True
183270
).start()
184271

185-
loop_start = time.time()
272+
stdout_lines: list[str] = []
273+
turn_start = time.time()
274+
timed_out = False
186275

187276
try:
188277
while True:
189278
try:
190279
line = out_q.get(timeout=per_turn_timeout)
191280
except queue.Empty:
192-
run.timed_out = True
281+
timed_out = True
193282
log(
194283
"driver",
195284
f"TIMEOUT after {per_turn_timeout}s waiting for next stdout line "
196-
f"(lines seen: {run.events_received}, "
197-
f"elapsed since spawn: {time.time() - loop_start:.1f}s)",
285+
f"(lines this turn: {len(stdout_lines)}, "
286+
f"elapsed: {time.time() - turn_start:.1f}s)",
198287
)
199288
break
200289

201290
if line is None:
202291
log(
203292
"driver",
204-
f"stdout EOF (copilot exited); lines seen: {run.events_received}",
293+
f"stdout EOF (copilot exited); lines this turn: {len(stdout_lines)}",
205294
)
206-
run.completed = True
207-
run.turns = 1
208295
break
209296

210-
run.transcript.append({"line": line})
211-
run.events_received += 1
297+
stdout_lines.append(line)
212298
if line.strip():
213-
log("driver", f"stdout #{run.events_received}: {_brief(line, 200)}")
214-
299+
log("driver", f"stdout #{len(stdout_lines)}: {_brief(line, 200)}")
215300
finally:
216301
try:
217302
proc.wait(timeout=15)
218303
except subprocess.TimeoutExpired:
219304
log("driver", "copilot did not exit within 15s; killing")
220305
proc.kill()
221306
proc.wait(timeout=5)
222-
run.exit_code = proc.returncode
223-
run.stderr_tail = "\n".join(err_lines[-40:])
224-
log(
225-
"driver",
226-
f"copilot exited code={run.exit_code} "
227-
f"lines={run.events_received} "
228-
f"completed={run.completed} timed_out={run.timed_out} "
229-
f"elapsed={time.time() - loop_start:.1f}s",
230-
)
231-
232-
if transcript_path is not None:
233-
run.save_transcript(transcript_path)
234-
log("driver", f"transcript written to {transcript_path}")
235307

236-
return run
308+
stderr_tail = "\n".join(err_lines[-40:])
309+
log(
310+
"driver",
311+
f"turn done: exit={proc.returncode} lines={len(stdout_lines)} "
312+
f"timed_out={timed_out} elapsed={time.time() - turn_start:.1f}s",
313+
)
314+
return stdout_lines, proc.returncode, stderr_tail, timed_out
237315

238316

239317
def _log_startup(
240318
workspace: Path,
241319
initial_prompt: str,
242320
per_turn_timeout: float,
321+
max_turns: int,
243322
) -> None:
244323
log("driver", f"drive_skill called: workspace={workspace}")
245324
log("driver", f"initial_prompt: {_brief(initial_prompt, 200)}")
246-
log("driver", f"per_turn_timeout={per_turn_timeout}s")
325+
log(
326+
"driver",
327+
f"per_turn_timeout={per_turn_timeout}s max_turns={max_turns}",
328+
)
247329

248330
copilot_path = shutil.which("copilot")
249331
log("driver", f"copilot on PATH: {copilot_path}")
@@ -283,6 +365,10 @@ def _pump_stderr_live(stream: IO[str], sink: list[str]) -> None:
283365
log("driver", f"stderr: {s}")
284366

285367

368+
def _asks_for_confirmation(text: str) -> bool:
369+
return bool(CONFIRMATION_PATTERN.search(text or ""))
370+
371+
286372
def _brief(text: str, limit: int = 140) -> str:
287373
collapsed = " ".join((text or "").split())
288374
return collapsed if len(collapsed) <= limit else collapsed[: limit - 1] + "…"

0 commit comments

Comments
 (0)