Repository navigation
fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows - #44
Merged
Merged
Conversation
skills-evals/tests/conftest.py defaults the agent filter to "claude"
when STAX_AGENT_KEY is unset (per the comment in
skills-eval-opencode.yml's env block: "Default is 'claude'"). The two
copilot workflows never set the var, so every Copilot CLI run was
collected, then immediately deselected:
[conftest] deselected 4 tests not matching agent='claude':
['tests/test_copilot_stream_smoke.py::...', ...]
============================ no tests ran in 0.12s =============================
##[error]Process completed with exit code 5.
Every other backend (cline / opencode / omp / pi, both scopes) sets
STAX_AGENT_KEY correctly; copilot was the lone outlier. Adds the
matching line to both copilot env blocks.
Out of scope: claude runs against this branch are now reporting an
upstream DeepSeek wire incompatibility (`API Error: 400 ... messages[1].role:
unknown variant 'system', expected 'user' or 'assistant'`) — Claude
Code 2.1.156 installed via @latest is emitting a system-role message
mid-stream that DeepSeek's Anthropic-compat endpoint rejects. That's
a Claude Code / DeepSeek version interaction, not a workflow bug, and
a separate change (pinning to a known-good Claude Code version) will
need to address it.
Signed-off-by: Tanzim Saqib <[email protected]>
Followup to the line-by-line audit. None of these change behavior on a
green path; they remove inconsistencies that were silent failure
vectors when the workflows drifted.
Behavioral changes:
- Drop the redundant per-agent `setup-node@v6` block from 10 of 12
workflows (everything except omp/omp-user-scope, which install via
curl and never had a second block). The first setup-node at the
top of each job already configures node 22 with the frontend's
package-lock cache; calling it again with no cache settings was a
no-op. PR #41 kept it deliberately to anchor the per-agent comment,
but the comments now sit on the npm install step where they
belong.
- Change `skills-eval-cline.yml`, `skills-eval-cline-user-scope.yml`,
`skills-eval-copilot.yml`, `skills-eval-copilot-user-scope.yml`
from the explicit 4-file `pytest tests/test_<agent>_*.py` selector
to bare `uv run pytest`. The conftest agent filter
(`STAX_AGENT_KEY=<key>`) already picks the right tests for every
other backend; the explicit form would silently skip any new test
file added to the matching backend. omp's redundant `export PATH=...`
(only affects the current step) is dropped; the GITHUB_PATH line
propagates correctly on its own.
Text fixes (no behavior change):
- Unify the `timeout-minutes: 60` comment across all 12 to one
sentence that doesn't claim a specific scenario count. claude.yml
said "One scenario today (TODO app)" and copilot.yml said "smoke
+ three scenario tests" for equally-shaped 4-5 test workloads.
- Unify the `run pytest` comment around `conftest.pytest_collection_modifyitems`
instead of the false "alphabetical pytest order" claim. Fix
`# test_stream_json_smoke.py runs first` in claude.yml /
claude-user-scope.yml — the actual filename is
`test_claude_stream_json_smoke.py`.
- Drop the "pinning is preferable to @latest" line from
claude.yml / claude-user-scope.yml. The workflows install
@latest, so the policy contradicted the implementation. Replace
with a "fragile but kept here" framing matching cline / copilot /
opencode / pi.
- Strip stale header-docblock claims:
- cline.yml "Mirrors skills-eval-opencode.yml" — it doesn't
(opencode uses bare pytest, cline used an explicit selector).
- cline.yml `Run uv run pytest tests/test_cline_*` glob hint —
the actual command was an explicit file list, not a glob.
- omp.yml "Drivers, judges, and test files live in a follow-up
PR" — the follow-up has shipped; tests live under
skills-evals/tests/.
- pi.yml `stax init --agents codex` — pi has its own
first-class agentTargets entry now, so the doc was describing
a transitional shape that no longer applies.
- Strip user-scope variants down to a minimal docblock
(`# Identical to skills-eval-<agent>.yml EXCEPT for
STAX_INSTALL_SCOPE.`) and drop the duplicated env-block /
per-step comments. claude-user-scope.yml previously carried the
full claude.yml comments; the other 5 user-scope variants
stripped them. Pick the stripped pattern uniformly.
Verified locally: `STAX_AGENT_KEY=<key> uv run pytest --collect-only`
for every key in {claude, opencode, copilot, pi, cline, omp} still
collects 25 tests with the conftest filter selecting the active
backend's set. All 12 YAML files parse clean.
Out of scope: workflow env blocks for claude / claude-user-scope and
copilot / copilot-user-scope still duplicate conftest's
CLAUDE_ENV_DEFAULTS / COPILOT_ENV_DEFAULTS. Dropping the duplicates
would risk the install-step `<agent> --version` smoke check losing
config that runs BEFORE pytest loads conftest. Leaving for a separate
change after the install-step semantics are confirmed.
Signed-off-by: Tanzim Saqib <[email protected]>
3 tasks done
tsaqib
added a commit
that referenced
this pull request
May 29, 2026
2.1.156 (current @latest) emits a `system`-role message at messages[1] of the wire payload. DeepSeek's Anthropic-compat endpoint rejects it with `400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system', expected 'user' or 'assistant'`. Reproduces on both claude workflows against the post-#44 main: https://github.com/stackific/stax/actions/runs/26618513746 https://github.com/stackific/stax/actions/runs/26618519248 2.1.150 is the most recent release I can verify doesn't carry the regression (chosen by working back from 2.1.156; the gap between 2.1.150 and 2.1.156 is six patch releases). Pinning rather than @latest matches the policy the install-step comment has always claimed. Bump intentionally after verifying the smoke test still passes. Signed-off-by: Tanzim Saqib <[email protected]>
Merged
4 tasks done
tsaqib
added a commit
that referenced
this pull request
May 29, 2026
* fix(skills-evals): rename copilot_workspace fixture to workspace The 3 copilot scenario tests (test_copilot_todo, test_copilot_work_item_extends, test_copilot_reminders_supersedes_todo) referenced a `copilot_workspace` pytest fixture that does not exist in conftest.py. The conftest only provides `workspace`. Every recent copilot eval errored at fixture-setup time with: fixture 'copilot_workspace' not found available fixtures: ..., workspace The copilot smoke test passes because it builds its own workspace directly without the fixture. The three scenario tests must use the same conftest-provided `workspace` fixture every other backend's tests use. Verified locally with --collect-only against the 3 files: 3 tests collected, no errors. Pre-existing bug, surfaces only now because earlier failures (STAX_AGENT_KEY missing in #44, syntax/import errors in #43) masked the collection-time error. Signed-off-by: Tanzim Saqib <[email protected]> * ci(skills-eval): pin Claude Code to 2.1.150 2.1.156 (current @latest) emits a `system`-role message at messages[1] of the wire payload. DeepSeek's Anthropic-compat endpoint rejects it with `400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system', expected 'user' or 'assistant'`. Reproduces on both claude workflows against the post-#44 main: https://github.com/stackific/stax/actions/runs/26618513746 https://github.com/stackific/stax/actions/runs/26618519248 2.1.150 is the most recent release I can verify doesn't carry the regression (chosen by working back from 2.1.156; the gap between 2.1.150 and 2.1.156 is six patch releases). Pinning rather than @latest matches the policy the install-step comment has always claimed. Bump intentionally after verifying the smoke test still passes. Signed-off-by: Tanzim Saqib <[email protected]> * fix(skills-evals): switch copilot to its own agent target Signed-off-by: Tanzim Saqib <[email protected]> --------- Signed-off-by: Tanzim Saqib <[email protected]>
tsaqib
added a commit
that referenced
this pull request
May 29, 2026
…ename copilot fixture (#47) * fix(skills-evals): rename copilot_workspace fixture to workspace The 3 copilot scenario tests (test_copilot_todo, test_copilot_work_item_extends, test_copilot_reminders_supersedes_todo) referenced a `copilot_workspace` pytest fixture that does not exist in conftest.py. The conftest only provides `workspace`. Every recent copilot eval errored at fixture-setup time with: fixture 'copilot_workspace' not found available fixtures: ..., workspace The copilot smoke test passes because it builds its own workspace directly without the fixture. The three scenario tests must use the same conftest-provided `workspace` fixture every other backend's tests use. Verified locally with --collect-only against the 3 files: 3 tests collected, no errors. Pre-existing bug, surfaces only now because earlier failures (STAX_AGENT_KEY missing in #44, syntax/import errors in #43) masked the collection-time error. Signed-off-by: Tanzim Saqib <[email protected]> * ci(skills-eval): pin Claude Code to 2.1.150 2.1.156 (current @latest) emits a `system`-role message at messages[1] of the wire payload. DeepSeek's Anthropic-compat endpoint rejects it with `400 Failed to deserialize the JSON body into the target type: messages[1].role: unknown variant 'system', expected 'user' or 'assistant'`. Reproduces on both claude workflows against the post-#44 main: https://github.com/stackific/stax/actions/runs/26618513746 https://github.com/stackific/stax/actions/runs/26618519248 2.1.150 is the most recent release I can verify doesn't carry the regression (chosen by working back from 2.1.156; the gap between 2.1.150 and 2.1.156 is six patch releases). Pinning rather than @latest matches the policy the install-step comment has always claimed. Bump intentionally after verifying the smoke test still passes. Signed-off-by: Tanzim Saqib <[email protected]> * fix(copilot-driver): inline SKILL.md to actually invoke /scope and /ship Copilot CLI's `-p <prompt>` does NOT resolve leading slash commands the way Claude Code does. Running `copilot -p "/scope build a single HTML todo list app"` passes the prompt verbatim and the model treats it as a literal request to build the app — skipping the scope planning step, never creating any work-item plan files. Direct evidence from run 26618829987 (copilot user-scope, fix/copilot-workspace-fixture): [driver] initial_prompt: /scope build a single HTML and localStorage- based todo list app [driver] spawn: copilot -p /scope build a single HTML ... [driver] stdout #5: Done. Created `todo.html` — a single-file ... The fixture-rename commit (b254fa8) made the collection-time error go away and exposed this deeper, pre-existing pathology that has kept all three copilot scenario tests (todo / work_item_extends / reminders_supersedes_todo) failing since they were introduced in #10. None have ever passed in CI; the prior fixture bug masked it as a collection error rather than a scenario error. Fix mirrors the cline driver's `compose_skill_prompt` strategy (which itself dates back to opencode's pre-`--command` era): detect a `/scope` or `/ship` prefix in the prompt, read the corresponding SKILL.md off disk, prepend its body, append the same CI directive every other inlining driver uses, and hand the composed text to `copilot -p`. The model then sees the actual skill instructions and follows them instead of treating the slash as ornamental. Discovery paths probed (project then user) mirror copilot's documented skill lookup list (constants.go: `.agents/skills/`, `.claude/skills/`, `.github/skills/` at project; `~/.copilot/skills/`, `~/.agents/skills/` at user) plus `~/.claude/skills/` because copilot tests install via `stax init --agents claude` (the transitional shape in conftest.AGENT_INIT_VALUE_FOR_KEY) which lands skills under `.claude/skills/`. Non-slash prompts (the smoke test's `Respond with the single word: ok`) pass through unchanged so the no-skill code path keeps working; `run.initial_prompt` retains the original slash form so logs and transcripts read the way the test author wrote them. CI_DIRECTIVE is copied verbatim from cline_driver — kept verbatim, not imported, because each driver's docstring asserts cross-driver parity is by-value and a `from cline_driver import CI_DIRECTIVE` would couple the modules in a way that hides accidental drift in code review. Signed-off-by: Tanzim Saqib <[email protected]> --------- Signed-off-by: Tanzim Saqib <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Followup to #43. The two copilot workflows were the lone outlier — every other backend's workflow sets
STAX_AGENT_KEYin its env block (the conftest defaults toclaudewhen unset, per the comment inskills-eval-opencode.yml). Without it, both copilot runs collect their four explicit tests and then deselect every one as not-matching-claude, leaving pytest with no tests to run:Sample failing run (workflow_dispatch against the #43 branch): https://github.com/stackific/stax/actions/runs/26617667665/job/78436696879
I missed this in the audit while preparing #43 — apologies.
Out of scope
The claude workflows are currently failing with a different, upstream issue: Claude Code
2.1.156(installed via@anthropic-ai/claude-code@latest) emits asystem-role message at position 1 of the wire messages array, which DeepSeek's Anthropic-compat endpoint rejects with400 ... messages[1].role: unknown variant 'system', expected 'user' or 'assistant'. That's a Claude Code / DeepSeek-shim version interaction and needs a separate change (pinning Claude Code to a known-good version).The opencode-user-scope run that's currently failing (https://github.com/stackific/stax/actions/runs/26617404908) is the eval doing its job —
ArtifactJudge failed for reminders task: score=0.50 reason=...retained the old todo.html without removing or marking it as superseded. That's a real agent-quality verdict from the judge, not a CI infrastructure bug.Test plan
workflow_dispatchre-run ofskills eval / github copilot cliagainst this branch — pytest collects 4 copilot tests and runs themworkflow_dispatchre-run ofskills eval / github copilot cli (user scope)against this branch — same