Skip to content

fix(skills-evals): pin Claude 2.1.150, inline SKILL.md for copilot, rename copilot fixture - #47

Merged
tsaqib merged 3 commits into
mainfrom
fix/skills-evals-workflows
May 29, 2026
Merged

tsaqib merged 3 commits into
mainfrom
fix/skills-evals-workflows

Conversation

@tsaqib

@tsaqib tsaqib commented May 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three commits address every reproducible skills-eval workflow failure on main.

  • claude × 2 (project + user scope) — Claude Code 2.1.156 (current @latest) emits a system-role message at messages[1] of the wire payload; DeepSeek's Anthropic-compat endpoint rejects it with 400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system'. Pinning the install step to @2.1.150 (the most recent verified-clean release before the regression) restores both claude workflows. Reproduces against post-fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows #44 main on runs 26618513746 / 26618519248.
  • copilot × 2 (project + user scope) — fixture collection error — three of the four copilot scenario tests referenced a copilot_workspace pytest fixture that does not exist in conftest.py; only the generic workspace fixture is provided. Renaming the references unblocks collection. Pre-existing bug introduced in test: add github copilot cli backend to skills-evals #10, masked since by earlier failures (fix(skills-evals): finish the plans → work-items rename #43 syntax / import errors, fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows #44 STAX_AGENT_KEY miss).
  • copilot × 2 — scenario failure exposed by the fixture fix — once collection succeeds, the scenario tests fail with expected exactly N work-item files, got 0. Copilot CLI's -p <prompt> does NOT resolve leading slash commands the way Claude Code does: copilot -p "/scope build a single HTML todo list app" is passed verbatim and the model just builds the app, skipping the scope planning step entirely. The driver now inlines the corresponding SKILL.md body when the prompt starts with /scope or /ship, mirroring cline_driver.compose_skill_prompt (which itself dates back to the opencode driver's pre---command era). Non-slash prompts (the smoke test) pass through unchanged.

Verified

All 12 skills-eval workflows on this branch — green:

Workflow Result
claude code ✅ pass (first run)
claude code (user scope) ✅ pass (passed on rerun — test_claude_reminders_supersedes_todo flaked once on the supersedes-status assertion, a model-behavior race; clean rerun confirmed it's flake, not regression)
cline / cline (user scope) ✅ pass
github copilot cli ✅ pass (first run — /scope and /ship engaged with the inlined SKILL.md as designed)
github copilot cli (user scope) ✅ pass (passed on rerun — same supersedes-LLM-flake shape as the claude one)
omp / omp (user scope) ✅ pass
opencode / opencode (user scope) ✅ pass
pi / pi (user scope) ✅ pass

Plus pre-push hook green locally (vet 1.7s · govulncheck 2.5s · lint 4.0s · test 4.1s · e2e 9.3s, e2e 625/0).

The two reruns are disclosed explicitly above. Both failed on test_*_reminders_supersedes_todo, the test most sensitive to LLM-judge non-determinism (it asks a model to delete a superseded predecessor artifact and a model-judge to score whether it did). Same code path passed at project scope in the same wall-clock minute; this PR does NOT silently retry — it's a known LLM-eval flake category, separately worth tracking via pytest-rerunfailures config in a follow-up, but out of scope for the structural fixes here.

Test plan

  • Local pre-push from clean state passes
  • CI claude / claude-user-scope workflows green
  • CI copilot / copilot-user-scope workflows green (first time these tests have ever passed in CI — they had been failing since test: add github copilot cli backend to skills-evals #10 introduced them)
  • CI cline / omp / opencode / pi workflows remain green

tsaqib added 3 commits May 29, 2026 00:58
The 3 copilot scenario tests (test_copilot_todo,
test_copilot_work_item_extends, test_copilot_reminders_supersedes_todo)
referenced a `copilot_workspace` pytest fixture that does not exist in
conftest.py. The conftest only provides `workspace`. Every recent
copilot eval errored at fixture-setup time with:

  fixture 'copilot_workspace' not found
  available fixtures: ..., workspace

The copilot smoke test passes because it builds its own workspace
directly without the fixture. The three scenario tests must use the
same conftest-provided `workspace` fixture every other backend's tests
use.

Verified locally with --collect-only against the 3 files: 3 tests
collected, no errors.

Pre-existing bug, surfaces only now because earlier failures
(STAX_AGENT_KEY missing in #44, syntax/import errors in #43) masked
the collection-time error.

Signed-off-by: Tanzim Saqib <[email protected]>
2.1.156 (current @latest) emits a `system`-role message at
messages[1] of the wire payload. DeepSeek's Anthropic-compat
endpoint rejects it with `400 Failed to deserialize the JSON
body into the target type: messages[1].role: unknown variant
'system', expected 'user' or 'assistant'`.

Reproduces on both claude workflows against the post-#44 main:

  https://github.com/stackific/stax/actions/runs/26618513746
  https://github.com/stackific/stax/actions/runs/26618519248

2.1.150 is the most recent release I can verify doesn't carry
the regression (chosen by working back from 2.1.156; the gap
between 2.1.150 and 2.1.156 is six patch releases). Pinning
rather than @latest matches the policy the install-step comment
has always claimed.

Bump intentionally after verifying the smoke test still passes.

Signed-off-by: Tanzim Saqib <[email protected]>
Copilot CLI's `-p <prompt>` does NOT resolve leading slash commands the
way Claude Code does. Running `copilot -p "/scope build a single HTML
todo list app"` passes the prompt verbatim and the model treats it as
a literal request to build the app — skipping the scope planning step,
never creating any work-item plan files. Direct evidence from run
26618829987 (copilot user-scope, fix/copilot-workspace-fixture):

  [driver] initial_prompt: /scope build a single HTML and localStorage-
    based todo list app
  [driver] spawn: copilot -p /scope build a single HTML ...
  [driver] stdout #5: Done. Created `todo.html` — a single-file ...

The fixture-rename commit (b254fa8) made the collection-time error go
away and exposed this deeper, pre-existing pathology that has kept all
three copilot scenario tests (todo / work_item_extends /
reminders_supersedes_todo) failing since they were introduced in #10.
None have ever passed in CI; the prior fixture bug masked it as a
collection error rather than a scenario error.

Fix mirrors the cline driver's `compose_skill_prompt` strategy (which
itself dates back to opencode's pre-`--command` era): detect a
`/scope` or `/ship` prefix in the prompt, read the corresponding
SKILL.md off disk, prepend its body, append the same CI directive
every other inlining driver uses, and hand the composed text to
`copilot -p`. The model then sees the actual skill instructions and
follows them instead of treating the slash as ornamental.

Discovery paths probed (project then user) mirror copilot's
documented skill lookup list (constants.go: `.agents/skills/`,
`.claude/skills/`, `.github/skills/` at project; `~/.copilot/skills/`,
`~/.agents/skills/` at user) plus `~/.claude/skills/` because copilot
tests install via `stax init --agents claude` (the transitional shape
in conftest.AGENT_INIT_VALUE_FOR_KEY) which lands skills under
`.claude/skills/`.

Non-slash prompts (the smoke test's `Respond with the single word:
ok`) pass through unchanged so the no-skill code path keeps working;
`run.initial_prompt` retains the original slash form so logs and
transcripts read the way the test author wrote them.

CI_DIRECTIVE is copied verbatim from cline_driver — kept verbatim, not
imported, because each driver's docstring asserts cross-driver parity
is by-value and a `from cline_driver import CI_DIRECTIVE` would couple
the modules in a way that hides accidental drift in code review.

Signed-off-by: Tanzim Saqib <[email protected]>
@tsaqib
tsaqib merged commit 5dcc957 into main May 29, 2026
18 of 21 checks passed
@tsaqib
tsaqib deleted the fix/skills-evals-workflows branch May 29, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant