Repository navigation
fix(skills-evals): pin Claude 2.1.150, inline SKILL.md for copilot, rename copilot fixture - #47
Merged
Merged
Conversation
The 3 copilot scenario tests (test_copilot_todo, test_copilot_work_item_extends, test_copilot_reminders_supersedes_todo) referenced a `copilot_workspace` pytest fixture that does not exist in conftest.py. The conftest only provides `workspace`. Every recent copilot eval errored at fixture-setup time with: fixture 'copilot_workspace' not found available fixtures: ..., workspace The copilot smoke test passes because it builds its own workspace directly without the fixture. The three scenario tests must use the same conftest-provided `workspace` fixture every other backend's tests use. Verified locally with --collect-only against the 3 files: 3 tests collected, no errors. Pre-existing bug, surfaces only now because earlier failures (STAX_AGENT_KEY missing in #44, syntax/import errors in #43) masked the collection-time error. Signed-off-by: Tanzim Saqib <[email protected]>
2.1.156 (current @latest) emits a `system`-role message at messages[1] of the wire payload. DeepSeek's Anthropic-compat endpoint rejects it with `400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system', expected 'user' or 'assistant'`. Reproduces on both claude workflows against the post-#44 main: https://github.com/stackific/stax/actions/runs/26618513746 https://github.com/stackific/stax/actions/runs/26618519248 2.1.150 is the most recent release I can verify doesn't carry the regression (chosen by working back from 2.1.156; the gap between 2.1.150 and 2.1.156 is six patch releases). Pinning rather than @latest matches the policy the install-step comment has always claimed. Bump intentionally after verifying the smoke test still passes. Signed-off-by: Tanzim Saqib <[email protected]>
Copilot CLI's `-p <prompt>` does NOT resolve leading slash commands the
way Claude Code does. Running `copilot -p "/scope build a single HTML
todo list app"` passes the prompt verbatim and the model treats it as
a literal request to build the app — skipping the scope planning step,
never creating any work-item plan files. Direct evidence from run
26618829987 (copilot user-scope, fix/copilot-workspace-fixture):
[driver] initial_prompt: /scope build a single HTML and localStorage-
based todo list app
[driver] spawn: copilot -p /scope build a single HTML ...
[driver] stdout #5: Done. Created `todo.html` — a single-file ...
The fixture-rename commit (b254fa8) made the collection-time error go
away and exposed this deeper, pre-existing pathology that has kept all
three copilot scenario tests (todo / work_item_extends /
reminders_supersedes_todo) failing since they were introduced in #10.
None have ever passed in CI; the prior fixture bug masked it as a
collection error rather than a scenario error.
Fix mirrors the cline driver's `compose_skill_prompt` strategy (which
itself dates back to opencode's pre-`--command` era): detect a
`/scope` or `/ship` prefix in the prompt, read the corresponding
SKILL.md off disk, prepend its body, append the same CI directive
every other inlining driver uses, and hand the composed text to
`copilot -p`. The model then sees the actual skill instructions and
follows them instead of treating the slash as ornamental.
Discovery paths probed (project then user) mirror copilot's
documented skill lookup list (constants.go: `.agents/skills/`,
`.claude/skills/`, `.github/skills/` at project; `~/.copilot/skills/`,
`~/.agents/skills/` at user) plus `~/.claude/skills/` because copilot
tests install via `stax init --agents claude` (the transitional shape
in conftest.AGENT_INIT_VALUE_FOR_KEY) which lands skills under
`.claude/skills/`.
Non-slash prompts (the smoke test's `Respond with the single word:
ok`) pass through unchanged so the no-skill code path keeps working;
`run.initial_prompt` retains the original slash form so logs and
transcripts read the way the test author wrote them.
CI_DIRECTIVE is copied verbatim from cline_driver — kept verbatim, not
imported, because each driver's docstring asserts cross-driver parity
is by-value and a `from cline_driver import CI_DIRECTIVE` would couple
the modules in a way that hides accidental drift in code review.
Signed-off-by: Tanzim Saqib <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three commits address every reproducible skills-eval workflow failure on
main.@latest) emits asystem-role message atmessages[1]of the wire payload; DeepSeek's Anthropic-compat endpoint rejects it with400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system'. Pinning the install step to@2.1.150(the most recent verified-clean release before the regression) restores both claude workflows. Reproduces against post-fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows #44 main on runs 26618513746 / 26618519248.copilot_workspacepytest fixture that does not exist inconftest.py; only the genericworkspacefixture is provided. Renaming the references unblocks collection. Pre-existing bug introduced in test: add github copilot cli backend to skills-evals #10, masked since by earlier failures (fix(skills-evals): finish theplans→work-itemsrename #43 syntax / import errors, fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows #44 STAX_AGENT_KEY miss).expected exactly N work-item files, got 0. Copilot CLI's-p <prompt>does NOT resolve leading slash commands the way Claude Code does:copilot -p "/scope build a single HTML todo list app"is passed verbatim and the model just builds the app, skipping the scope planning step entirely. The driver now inlines the correspondingSKILL.mdbody when the prompt starts with/scopeor/ship, mirroringcline_driver.compose_skill_prompt(which itself dates back to the opencode driver's pre---commandera). Non-slash prompts (the smoke test) pass through unchanged.Verified
All 12 skills-eval workflows on this branch — green:
test_claude_reminders_supersedes_todoflaked once on the supersedes-status assertion, a model-behavior race; clean rerun confirmed it's flake, not regression)/scopeand/shipengaged with the inlined SKILL.md as designed)Plus pre-push hook green locally (vet 1.7s · govulncheck 2.5s · lint 4.0s · test 4.1s · e2e 9.3s, e2e 625/0).
The two reruns are disclosed explicitly above. Both failed on
test_*_reminders_supersedes_todo, the test most sensitive to LLM-judge non-determinism (it asks a model to delete a superseded predecessor artifact and a model-judge to score whether it did). Same code path passed at project scope in the same wall-clock minute; this PR does NOT silently retry — it's a known LLM-eval flake category, separately worth tracking viapytest-rerunfailuresconfig in a follow-up, but out of scope for the structural fixes here.Test plan