Repository navigation
fix(skills-evals): rename copilot_workspace fixture to workspace - #46
Merged
Merged
Conversation
The 3 copilot scenario tests (test_copilot_todo, test_copilot_work_item_extends, test_copilot_reminders_supersedes_todo) referenced a `copilot_workspace` pytest fixture that does not exist in conftest.py. The conftest only provides `workspace`. Every recent copilot eval errored at fixture-setup time with: fixture 'copilot_workspace' not found available fixtures: ..., workspace The copilot smoke test passes because it builds its own workspace directly without the fixture. The three scenario tests must use the same conftest-provided `workspace` fixture every other backend's tests use. Verified locally with --collect-only against the 3 files: 3 tests collected, no errors. Pre-existing bug, surfaces only now because earlier failures (STAX_AGENT_KEY missing in #44, syntax/import errors in #43) masked the collection-time error. Signed-off-by: Tanzim Saqib <[email protected]>
2.1.156 (current @latest) emits a `system`-role message at messages[1] of the wire payload. DeepSeek's Anthropic-compat endpoint rejects it with `400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system', expected 'user' or 'assistant'`. Reproduces on both claude workflows against the post-#44 main: https://github.com/stackific/stax/actions/runs/26618513746 https://github.com/stackific/stax/actions/runs/26618519248 2.1.150 is the most recent release I can verify doesn't carry the regression (chosen by working back from 2.1.156; the gap between 2.1.150 and 2.1.156 is six patch releases). Pinning rather than @latest matches the policy the install-step comment has always claimed. Bump intentionally after verifying the smoke test still passes. Signed-off-by: Tanzim Saqib <[email protected]>
Signed-off-by: Tanzim Saqib <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 3 copilot scenario tests reference a
copilot_workspacepytest fixture that does not exist. conftest.py only providesworkspace. Every recent copilot eval errors at fixture-setup time before any agent code runs:Reproduces in https://github.com/stackific/stax/actions/runs/26618529351 (the first run after #44 unblocked copilot collection): copilot smoke test PASSES (it builds its own workspace, doesn't use the fixture), then all 3 scenario tests ERROR with the missing-fixture message.
Every other backend's tests (
test_claude_*,test_opencode_*,test_pi_*,test_cline_*,test_omp_*) useworkspace. Renaming the copilot tests to match unblocks them — the conftestworkspacefixture is generic across backends and readsSTAX_AGENT_KEYto dispatch any agent-specific setup.Pre-existing bug; only surfaces now because PR #43 (collection-time SyntaxError/ImportError) and PR #44 (copilot STAX_AGENT_KEY default) masked it earlier.
Verification
Local:
STAX_AGENT_KEY=copilot uv run pytest tests/test_copilot_todo.py tests/test_copilot_work_item_extends.py tests/test_copilot_reminders_supersedes_todo.py --collect-only→ 3 tests collected, 0 errors.Pre-push: vet, lint, govulncheck, test, e2e (624 passed) all green.
Test plan
workflow_dispatchre-run ofskills eval / github copilot cliagainst this branch — 3 scenario tests reach the agent driver instead of erroring at fixture setupskills eval / github copilot cli (user scope)