Skip to content

Commit 5497a31

Browse files
committed
fix(model): consolidate Copilot review cleanup
Clarify null and cloud-service documentation, document the tool-call limitation, harden and deduplicate JSONL parsing, centralize child-environment sanitization, normalize legacy backend selection, and replace source-text CLI checks with behavioral assertions. Co-authored-by: Copilot <[email protected]> Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
1 parent 4b4d9c5 commit 5497a31

9 files changed

Lines changed: 92 additions & 74 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,11 @@ All notable changes to SkillOpt are documented here. This project adheres to
1010
- **GitHub Copilot CLI backend**, in two forms: `copilot_chat` (usable as both
1111
optimizer and target) and `copilot_exec` (target-only execution harness).
1212
Because the Copilot CLI carries its own sign-in, `--backend copilot` selects
13-
`copilot_chat` for both roles and runs a complete train/eval loop with **no
14-
cloud API key**. Calls disable built-in MCP servers and custom instructions,
15-
and never pass `--allow-all-tools`; `copilot_exec` requires an explicit
13+
`copilot_chat` for both roles and runs a complete train/eval loop with no
14+
separate provider API key; inference still uses the GitHub Copilot cloud
15+
service. Chat calls disable all built-in tools, built-in MCP servers, and
16+
custom instructions, strip inherited `COPILOT_ALLOW_ALL`, and never pass
17+
`--allow-all-tools`; `copilot_exec` requires an explicit
1618
`copilot_exec_allow_all_tools` opt-in before granting unattended tool use.
1719
The CLI reports no token counts, so usage totals are zero for these backends.
1820
- A non-destructive Devin installer and SessionEnd activity marker, preserving

‎configs/_base_/default.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,8 @@ model:
3030
# using CLI authentication rather than a separate provider API key.
3131
copilot_exec_path: "" # blank uses COPILOT_EXEC_PATH or copilot
3232
copilot_exec_home: "" # blank leaves COPILOT_HOME untouched
33-
copilot_exec_allow_all_tools: null # blank uses COPILOT_EXEC_ALLOW_ALL_TOOLS (default off);
34-
# copilot_exec only, required for file-edit rollouts
33+
# null preserves COPILOT_EXEC_ALLOW_ALL_TOOLS (default off); copilot_exec only
34+
copilot_exec_allow_all_tools: null # required for file-edit rollouts
3535
copilot_chat_optimizer_model: "" # blank lets the CLI pick its default model
3636
copilot_chat_target_model: ""
3737
copilot_chat_timeout: 0 # 0 uses COPILOT_CHAT_TIMEOUT or the built-in default

‎docs/guide/configuration.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ cloud service. Sign in once with `copilot` (GitHub Copilot CLI) beforehand.
5656
Expect roughly 20-40 s per call: the CLI is an agent, not a completions
5757
endpoint, so a full-size run is far slower than a hosted backend. It also
5858
reports no token counts, so usage totals are zero.
59+
It does not support caller-supplied tools or structured tool calls; environments
60+
that require those features must use another chat backend.
5961

6062
The current MiniMax adapter has one shared deployment. Set
6163
`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot
@@ -230,8 +232,8 @@ Because the CLI carries its own sign-in, selecting it for both roles
230232
(`--backend copilot`) avoids separate provider API-key configuration:
231233

232234
```bash
233-
copilot # sign in once, then exit
234-
skillopt-train --config configs/train/default.yaml --backend copilot
235+
copilot login
236+
skillopt-train --config configs/searchqa/default.yaml --backend copilot
235237
```
236238

237239
Chat calls use an empty tool allowlist and disable built-in MCP servers and

‎docs/reference/config.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`.
7171
| `model.cursor_exec_sandbox` | Cursor sandbox mode: `enabled` (default) or `disabled`; file-edit rollouts require `enabled` |
7272
| `model.copilot_exec_path` | GitHub Copilot CLI executable path; default `copilot` |
7373
| `model.copilot_exec_home` | Optional `COPILOT_HOME` override isolating CLI config |
74-
| `model.copilot_exec_allow_all_tools` | Opt in to `--allow-all-tools`; `false` by default, required for file-edit rollouts |
74+
| `model.copilot_exec_allow_all_tools` | Optional opt-in to `--allow-all-tools`; unset by default so `COPILOT_EXEC_ALLOW_ALL_TOOLS` remains authoritative |
7575
| `model.copilot_chat_optimizer_model` / `model.copilot_chat_target_model` | Optional per-role `--model` IDs for `copilot_chat` |
7676
| `model.copilot_chat_timeout` | Per-call timeout in seconds for `copilot_chat` |
7777

‎skillopt/engine/trainer.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -464,7 +464,7 @@ def _resolve_role_backends(
464464
something else always wins.
465465
"""
466466
backend = normalize_backend_name(backend)
467-
if backend in {"claude", "claude_chat"}:
467+
if backend == "claude_chat":
468468
# A chat backend fills BOTH roles, so -- like copilot -- a role pinned
469469
# to a default (including the base config's truthy openai_chat) must be
470470
# overridden. `x = x or ...` would leave openai_chat in place.
@@ -481,11 +481,11 @@ def _resolve_role_backends(
481481
optimizer_backend = optimizer_backend or "openai_chat"
482482
if target_backend in _ROLE_BACKEND_DEFAULTS:
483483
target_backend = "claude_code_exec"
484-
elif backend in {"cursor", "cursor_exec"}:
484+
elif backend == "cursor_exec":
485485
optimizer_backend = optimizer_backend or "openai_chat"
486486
if target_backend in _ROLE_BACKEND_DEFAULTS:
487487
target_backend = "cursor_exec"
488-
elif backend in {"copilot", "copilot_chat"}:
488+
elif backend == "copilot_chat":
489489
# Both roles use the locally installed, CLI-authenticated backend.
490490
if optimizer_backend in _ROLE_BACKEND_DEFAULTS:
491491
optimizer_backend = "copilot_chat"
@@ -495,7 +495,7 @@ def _resolve_role_backends(
495495
optimizer_backend = optimizer_backend or "openai_chat"
496496
if target_backend in _ROLE_BACKEND_DEFAULTS:
497497
target_backend = "copilot_exec"
498-
elif backend in {"qwen", "qwen_chat"}:
498+
elif backend == "qwen_chat":
499499
optimizer_backend = optimizer_backend or "openai_chat"
500500
if target_backend in _ROLE_BACKEND_DEFAULTS:
501501
target_backend = "qwen_chat"

‎skillopt/model/__init__.py‎

Lines changed: 10 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
set_optimizer_backend,
3131
set_target_backend,
3232
)
33+
from skillopt.model.common import normalize_backend_name
3334

3435

3536
def set_backend(name: str | None) -> str:
@@ -39,32 +40,28 @@ def set_backend(name: str | None) -> str:
3940
target. Keep that entry point so older scripts continue to work, while
4041
mapping it onto the split optimizer/target backend model.
4142
"""
42-
normalized = str(name or "azure_openai").strip().lower()
43-
if normalized in {"azure_openai", "openai_chat", "azure", "azure-openai"}:
43+
normalized = normalize_backend_name(name)
44+
if normalized in {"azure_openai", "openai_chat"}:
4445
set_optimizer_backend("openai_chat")
4546
set_target_backend("openai_chat")
4647
return "azure_openai"
47-
if normalized in {"claude", "claude_chat", "anthropic"}:
48+
if normalized == "claude_chat":
4849
set_optimizer_backend("claude_chat")
4950
set_target_backend("claude_chat")
5051
return "claude_chat"
51-
if normalized == "codex":
52-
set_optimizer_backend("codex_exec")
53-
set_target_backend("codex_exec")
54-
return "codex"
55-
if normalized == "codex_exec":
52+
if normalized in {"codex", "codex_exec"}:
5653
set_optimizer_backend("codex_exec")
5754
set_target_backend("codex_exec")
5855
return normalized
5956
if normalized == "claude_code_exec":
6057
set_optimizer_backend("openai_chat")
6158
set_target_backend(normalized)
6259
return normalized
63-
if normalized in {"cursor", "cursor_agent", "cursor_exec"}:
60+
if normalized == "cursor_exec":
6461
set_optimizer_backend("openai_chat")
6562
set_target_backend("cursor_exec")
6663
return "cursor_exec"
67-
if normalized in {"copilot", "copilot_cli", "github_copilot", "copilot_chat"}:
64+
if normalized == "copilot_chat":
6865
# The CLI-authenticated backend drives both roles without a separate
6966
# provider API key; inference still uses the Copilot cloud service.
7067
set_optimizer_backend("copilot_chat")
@@ -74,15 +71,15 @@ def set_backend(name: str | None) -> str:
7471
set_optimizer_backend("openai_chat")
7572
set_target_backend("copilot_exec")
7673
return "copilot_exec"
77-
if normalized in {"qwen", "qwen_chat"}:
74+
if normalized == "qwen_chat":
7875
set_optimizer_backend("openai_chat")
7976
set_target_backend("qwen_chat")
8077
return "qwen_chat"
81-
if normalized in {"minimax", "minimax_chat"}:
78+
if normalized == "minimax_chat":
8279
set_optimizer_backend("openai_chat")
8380
set_target_backend("minimax_chat")
8481
return "minimax_chat"
85-
if normalized in {"openai_compatible", "openai_compatible_chat", "openai-compatible", "compat"}:
82+
if normalized == "openai_compatible":
8683
set_optimizer_backend("openai_compatible")
8784
set_target_backend("openai_compatible")
8885
return "openai_compatible"

‎skillopt/model/codex_harness.py‎

Lines changed: 6 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@
1818
get_cursor_exec_config,
1919
get_target_backend,
2020
)
21+
from skillopt.model.copilot_backend import (
22+
build_copilot_subprocess_env,
23+
parse_copilot_jsonl,
24+
)
2125

2226
ANSWER_SCHEMA: dict[str, Any] = {
2327
"type": "object",
@@ -1360,13 +1364,8 @@ def run_copilot_exec(
13601364
for path in add_dirs:
13611365
cmd.extend(["--add-dir", path])
13621366

1363-
env = os.environ.copy()
1364-
# Configuration is authoritative: an inherited CLI-wide opt-in must
1365-
# not bypass SkillOpt's allow_all_tools and allow_file_edits gates.
1366-
env.pop("COPILOT_ALLOW_ALL", None)
13671367
home = str(config.get("home") or "")
1368-
if home:
1369-
env["COPILOT_HOME"] = home
1368+
env = build_copilot_subprocess_env(home)
13701369

13711370
try:
13721371
proc = subprocess.run(
@@ -1400,7 +1399,7 @@ def run_copilot_exec(
14001399
f"Copilot CLI failed with exit code {proc.returncode}: {detail}"
14011400
)
14021401

1403-
response = _parse_copilot_jsonl(stdout)
1402+
response = parse_copilot_jsonl(stdout)
14041403
if response:
14051404
return response, combined
14061405

@@ -1412,24 +1411,6 @@ def run_copilot_exec(
14121411
raise RuntimeError(f"{last_error}\n{detail}" if detail else last_error)
14131412

14141413

1415-
def _parse_copilot_jsonl(raw: str) -> str:
1416-
"""Concatenate ``assistant.message`` content from a Copilot JSONL stream."""
1417-
parts: list[str] = []
1418-
for line in (raw or "").splitlines():
1419-
line = line.strip()
1420-
if not line or not line.startswith("{"):
1421-
continue
1422-
try:
1423-
obj = json.loads(line)
1424-
except Exception:
1425-
continue
1426-
if obj.get("type") == "assistant.message":
1427-
content = (obj.get("data") or {}).get("content")
1428-
if isinstance(content, str) and content:
1429-
parts.append(content)
1430-
return "\n".join(parts).strip()
1431-
1432-
14331414
def run_target_exec(
14341415
*,
14351416
work_dir: str,

‎skillopt/model/copilot_backend.py‎

Lines changed: 20 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,15 @@
3232
_ZERO_USAGE = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
3333

3434

35+
def build_copilot_subprocess_env(home: str = "") -> dict[str, str]:
36+
"""Build a child environment without inherited unattended-tool approval."""
37+
env = os.environ.copy()
38+
env.pop("COPILOT_ALLOW_ALL", None)
39+
if home:
40+
env["COPILOT_HOME"] = home
41+
return env
42+
43+
3544
def _compose_prompt(system: str, user: str) -> str:
3645
system = (system or "").strip()
3746
user = (user or "").strip()
@@ -56,18 +65,24 @@ def _messages_to_prompt(messages: list[dict[str, Any]]) -> str:
5665
return "\n\n---\n\n".join(parts)
5766

5867

59-
def _parse_jsonl_response(raw: str) -> str:
68+
def parse_copilot_jsonl(raw: str) -> str:
69+
"""Concatenate assistant text from a Copilot JSONL event stream."""
6070
parts: list[str] = []
6171
for line in (raw or "").splitlines():
6272
line = line.strip()
6373
if not line or not line.startswith("{"):
6474
continue
6575
try:
6676
obj = json.loads(line)
67-
except Exception:
77+
except json.JSONDecodeError:
78+
continue
79+
if not isinstance(obj, dict):
6880
continue
6981
if obj.get("type") == "assistant.message":
70-
content = (obj.get("data") or {}).get("content")
82+
data = obj.get("data")
83+
if not isinstance(data, dict):
84+
continue
85+
content = data.get("content")
7186
if isinstance(content, str) and content:
7287
parts.append(content)
7388
return "\n".join(parts).strip()
@@ -96,13 +111,8 @@ def _invoke(prompt: str, *, model: str, timeout: float | None) -> str:
96111
if chosen:
97112
cmd.extend(["--model", chosen])
98113

99-
env = os.environ.copy()
100-
# The parent may use this for its own Copilot session. It must not silently
101-
# turn a SkillOpt child into an unrestricted agent.
102-
env.pop("COPILOT_ALLOW_ALL", None)
103114
home = str(config.get("home") or "")
104-
if home:
105-
env["COPILOT_HOME"] = home
115+
env = build_copilot_subprocess_env(home)
106116

107117
proc = subprocess.run(
108118
cmd,
@@ -116,7 +126,7 @@ def _invoke(prompt: str, *, model: str, timeout: float | None) -> str:
116126
if proc.returncode != 0:
117127
detail = ((proc.stderr or proc.stdout) or "").strip()[:4000]
118128
raise RuntimeError(f"Copilot CLI failed with exit code {proc.returncode}: {detail}")
119-
return _parse_jsonl_response(proc.stdout or "")
129+
return parse_copilot_jsonl(proc.stdout or "")
120130

121131

122132
def _chat_impl(

‎tests/test_copilot_exec_backend.py‎

Lines changed: 40 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
from __future__ import annotations
22

3+
import ast
34
import json
45
import os
56
import subprocess
7+
import sys
68
from collections.abc import Iterator
79
from pathlib import Path
810

@@ -65,10 +67,13 @@ def test_aliases_normalize_to_copilot_exec(alias: str) -> None:
6567

6668

6769
@pytest.mark.parametrize("alias", ["copilot", "copilot_cli", "github_copilot"])
68-
def test_bare_copilot_aliases_normalize_to_the_fully_local_chat_backend(
70+
def test_bare_copilot_aliases_normalize_to_the_cli_authenticated_chat_backend(
6971
alias: str,
7072
) -> None:
7173
assert normalize_backend_name(alias) == "copilot_chat"
74+
assert model.set_backend(alias) == "copilot_chat"
75+
assert backend_config.get_optimizer_backend() == "copilot_chat"
76+
assert backend_config.get_target_backend() == "copilot_chat"
7277

7378

7479
def test_set_backend_routes_target_to_copilot_and_keeps_chat_optimizer() -> None:
@@ -119,16 +124,19 @@ def test_parse_jsonl_concatenates_assistant_messages_and_ignores_noise() -> None
119124
json.dumps({"type": "tool.call", "data": {"content": "ignored"}}),
120125
json.dumps({"type": "assistant.message", "data": {"content": "first"}}),
121126
"{ broken json",
127+
"[]",
128+
"null",
129+
json.dumps({"type": "assistant.message", "data": "invalid"}),
122130
json.dumps({"type": "assistant.message", "data": {"content": "second"}}),
123131
json.dumps({"type": "assistant.message", "data": {}}),
124132
]
125133
)
126-
assert harness._parse_copilot_jsonl(raw) == "first\nsecond"
134+
assert copilot_backend.parse_copilot_jsonl(raw) == "first\nsecond"
127135

128136

129137
def test_parse_jsonl_returns_empty_for_no_assistant_messages() -> None:
130-
assert harness._parse_copilot_jsonl('{"type":"tool.call","data":{}}') == ""
131-
assert harness._parse_copilot_jsonl("") == ""
138+
assert copilot_backend.parse_copilot_jsonl('{"type":"tool.call","data":{}}') == ""
139+
assert copilot_backend.parse_copilot_jsonl("") == ""
132140

133141

134142
def _fake_run(captured: dict, stdout: str, returncode: int = 0):
@@ -216,7 +224,8 @@ def test_copilot_backends_keep_a_real_deployment_fallback() -> None:
216224
assert default_model_for_backend("copilot_chat") == "gpt-4o"
217225

218226

219-
def test_no_response_error_includes_cli_output(monkeypatch, tmp_path) -> None: # copilot_exec persists no artifacts, so a bare "returned no response"
227+
def test_no_response_error_includes_cli_output(monkeypatch, tmp_path) -> None:
228+
# copilot_exec persists no artifacts, so a bare "returned no response"
220229
# would leave an empty/invalid JSONL stream undebuggable.
221230
model.set_backend("copilot")
222231
model.configure_copilot_exec(path="copilot", home="", allow_all_tools=False)
@@ -459,19 +468,36 @@ def test_copilot_config_keys_survive_flattening() -> None:
459468
@pytest.mark.parametrize("script", ["train", "eval_only"])
460469
def test_cli_exposes_copilot_backend_and_flags(script: str) -> None:
461470
root = Path(__file__).resolve().parents[1]
462-
text = (root / "scripts" / f"{script}.py").read_text(encoding="utf-8")
463-
assert "copilot_chat" in text
464-
assert "copilot_exec" in text
471+
script_path = root / "scripts" / f"{script}.py"
472+
result = subprocess.run(
473+
[sys.executable, str(script_path), "--help"],
474+
capture_output=True,
475+
text=True,
476+
timeout=30,
477+
)
478+
assert result.returncode == 0, result.stderr
479+
assert "copilot_chat" in result.stdout
480+
assert "copilot_exec" in result.stdout
465481
for flag in (
466482
"--copilot_exec_path",
483+
"--copilot_exec_home",
484+
"--copilot_exec_allow_all_tools",
485+
"--copilot_chat_optimizer_model",
467486
"--copilot_chat_target_model",
468487
"--copilot_chat_timeout",
469488
):
470-
assert flag in text
489+
assert flag in result.stdout
471490
# train.py delegates backend configuration to the trainer; eval_only wires
472-
# it directly. Assert whichever applies so the wiring can't silently drop.
473-
applier = (
474-
text if script == "eval_only" else (root / "skillopt" / "engine" / "trainer.py").read_text(encoding="utf-8")
491+
# it directly. Inspect call nodes rather than source spelling/formatting.
492+
applier_path = (
493+
script_path
494+
if script == "eval_only"
495+
else root / "skillopt" / "engine" / "trainer.py"
475496
)
476-
assert "configure_copilot_chat" in applier
477-
assert "configure_copilot_exec" in applier
497+
tree = ast.parse(applier_path.read_text(encoding="utf-8"))
498+
calls = {
499+
node.func.id
500+
for node in ast.walk(tree)
501+
if isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
502+
}
503+
assert {"configure_copilot_chat", "configure_copilot_exec"} <= calls

0 commit comments

Comments
 (0)