Repository navigation
fix: windows-only e2e in the GH workflow - #5
Merged
Merged
Conversation
Signed-off-by: Tanzim Saqib <[email protected]>
tsaqib
added a commit
that referenced
this pull request
May 29, 2026
…ename copilot fixture (#47) * fix(skills-evals): rename copilot_workspace fixture to workspace The 3 copilot scenario tests (test_copilot_todo, test_copilot_work_item_extends, test_copilot_reminders_supersedes_todo) referenced a `copilot_workspace` pytest fixture that does not exist in conftest.py. The conftest only provides `workspace`. Every recent copilot eval errored at fixture-setup time with: fixture 'copilot_workspace' not found available fixtures: ..., workspace The copilot smoke test passes because it builds its own workspace directly without the fixture. The three scenario tests must use the same conftest-provided `workspace` fixture every other backend's tests use. Verified locally with --collect-only against the 3 files: 3 tests collected, no errors. Pre-existing bug, surfaces only now because earlier failures (STAX_AGENT_KEY missing in #44, syntax/import errors in #43) masked the collection-time error. Signed-off-by: Tanzim Saqib <[email protected]> * ci(skills-eval): pin Claude Code to 2.1.150 2.1.156 (current @latest) emits a `system`-role message at messages[1] of the wire payload. DeepSeek's Anthropic-compat endpoint rejects it with `400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system', expected 'user' or 'assistant'`. Reproduces on both claude workflows against the post-#44 main: https://github.com/stackific/stax/actions/runs/26618513746 https://github.com/stackific/stax/actions/runs/26618519248 2.1.150 is the most recent release I can verify doesn't carry the regression (chosen by working back from 2.1.156; the gap between 2.1.150 and 2.1.156 is six patch releases). Pinning rather than @latest matches the policy the install-step comment has always claimed. Bump intentionally after verifying the smoke test still passes. Signed-off-by: Tanzim Saqib <[email protected]> * fix(copilot-driver): inline SKILL.md to actually invoke /scope and /ship Copilot CLI's `-p <prompt>` does NOT resolve leading slash commands the way Claude Code does. Running `copilot -p "/scope build a single HTML todo list app"` passes the prompt verbatim and the model treats it as a literal request to build the app — skipping the scope planning step, never creating any work-item plan files. Direct evidence from run 26618829987 (copilot user-scope, fix/copilot-workspace-fixture): [driver] initial_prompt: /scope build a single HTML and localStorage- based todo list app [driver] spawn: copilot -p /scope build a single HTML ... [driver] stdout #5: Done. Created `todo.html` — a single-file ... The fixture-rename commit (b254fa8) made the collection-time error go away and exposed this deeper, pre-existing pathology that has kept all three copilot scenario tests (todo / work_item_extends / reminders_supersedes_todo) failing since they were introduced in #10. None have ever passed in CI; the prior fixture bug masked it as a collection error rather than a scenario error. Fix mirrors the cline driver's `compose_skill_prompt` strategy (which itself dates back to opencode's pre-`--command` era): detect a `/scope` or `/ship` prefix in the prompt, read the corresponding SKILL.md off disk, prepend its body, append the same CI directive every other inlining driver uses, and hand the composed text to `copilot -p`. The model then sees the actual skill instructions and follows them instead of treating the slash as ornamental. Discovery paths probed (project then user) mirror copilot's documented skill lookup list (constants.go: `.agents/skills/`, `.claude/skills/`, `.github/skills/` at project; `~/.copilot/skills/`, `~/.agents/skills/` at user) plus `~/.claude/skills/` because copilot tests install via `stax init --agents claude` (the transitional shape in conftest.AGENT_INIT_VALUE_FOR_KEY) which lands skills under `.claude/skills/`. Non-slash prompts (the smoke test's `Respond with the single word: ok`) pass through unchanged so the no-skill code path keeps working; `run.initial_prompt` retains the original slash form so logs and transcripts read the way the test author wrote them. CI_DIRECTIVE is copied verbatim from cline_driver — kept verbatim, not imported, because each driver's docstring asserts cross-driver parity is by-value and a `from cline_driver import CI_DIRECTIVE` would couple the modules in a way that hides accidental drift in code review. Signed-off-by: Tanzim Saqib <[email protected]> --------- Signed-off-by: Tanzim Saqib <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Checklist
git rebase --signoff --exec 'git commit --amend --no-edit -S' $(git merge-base HEAD origin/main) && git push --force-with-lease