Repository navigation
ci: build frontend before go install in every skills-eval workflow - #41
Merged
Merged
Conversation
server.go embeds `frontend/dist/` via `//go:embed all:frontend/dist` (commit 049a75b). The directory is gitignored, so any Go compile fails with `pattern all:frontend/dist: no matching files found` if the Vite build has not run first. `test.yml`, `release.yml`, and `windows-cli.yml` were updated to build the frontend before invoking Go. The twelve manually-triggered `skills-eval-*.yml` workflows were not, so every one of them has been broken since 049a75b — failing at the `install stax to PATH` step before the eval body ever runs. Mirror the build order from `test.yml:48-64`: hoist `setup-node` (with npm cache keyed on `frontend/package-lock.json`) ahead of the Go install step and run `npm ci` + `npm run build` in `frontend/`. The existing per-agent `setup-node` block later in each workflow is left in place — a second call against the same node version is a no-op and keeps the agent-specific comments next to the npm install step they document. The omp variants previously had no `setup-node` block at all (the omp installer is curl-based); they get one here purely for the frontend build. Signed-off-by: Tanzim Saqib <[email protected]>
tsaqib
added a commit
that referenced
this pull request
May 29, 2026
Followup to the line-by-line audit. None of these change behavior on a
green path; they remove inconsistencies that were silent failure
vectors when the workflows drifted.
Behavioral changes:
- Drop the redundant per-agent `setup-node@v6` block from 10 of 12
workflows (everything except omp/omp-user-scope, which install via
curl and never had a second block). The first setup-node at the
top of each job already configures node 22 with the frontend's
package-lock cache; calling it again with no cache settings was a
no-op. PR #41 kept it deliberately to anchor the per-agent comment,
but the comments now sit on the npm install step where they
belong.
- Change `skills-eval-cline.yml`, `skills-eval-cline-user-scope.yml`,
`skills-eval-copilot.yml`, `skills-eval-copilot-user-scope.yml`
from the explicit 4-file `pytest tests/test_<agent>_*.py` selector
to bare `uv run pytest`. The conftest agent filter
(`STAX_AGENT_KEY=<key>`) already picks the right tests for every
other backend; the explicit form would silently skip any new test
file added to the matching backend. omp's redundant `export PATH=...`
(only affects the current step) is dropped; the GITHUB_PATH line
propagates correctly on its own.
Text fixes (no behavior change):
- Unify the `timeout-minutes: 60` comment across all 12 to one
sentence that doesn't claim a specific scenario count. claude.yml
said "One scenario today (TODO app)" and copilot.yml said "smoke
+ three scenario tests" for equally-shaped 4-5 test workloads.
- Unify the `run pytest` comment around `conftest.pytest_collection_modifyitems`
instead of the false "alphabetical pytest order" claim. Fix
`# test_stream_json_smoke.py runs first` in claude.yml /
claude-user-scope.yml — the actual filename is
`test_claude_stream_json_smoke.py`.
- Drop the "pinning is preferable to @latest" line from
claude.yml / claude-user-scope.yml. The workflows install
@latest, so the policy contradicted the implementation. Replace
with a "fragile but kept here" framing matching cline / copilot /
opencode / pi.
- Strip stale header-docblock claims:
- cline.yml "Mirrors skills-eval-opencode.yml" — it doesn't
(opencode uses bare pytest, cline used an explicit selector).
- cline.yml `Run uv run pytest tests/test_cline_*` glob hint —
the actual command was an explicit file list, not a glob.
- omp.yml "Drivers, judges, and test files live in a follow-up
PR" — the follow-up has shipped; tests live under
skills-evals/tests/.
- pi.yml `stax init --agents codex` — pi has its own
first-class agentTargets entry now, so the doc was describing
a transitional shape that no longer applies.
- Strip user-scope variants down to a minimal docblock
(`# Identical to skills-eval-<agent>.yml EXCEPT for
STAX_INSTALL_SCOPE.`) and drop the duplicated env-block /
per-step comments. claude-user-scope.yml previously carried the
full claude.yml comments; the other 5 user-scope variants
stripped them. Pick the stripped pattern uniformly.
Verified locally: `STAX_AGENT_KEY=<key> uv run pytest --collect-only`
for every key in {claude, opencode, copilot, pi, cline, omp} still
collects 25 tests with the conftest filter selecting the active
backend's set. All 12 YAML files parse clean.
Out of scope: workflow env blocks for claude / claude-user-scope and
copilot / copilot-user-scope still duplicate conftest's
CLAUDE_ENV_DEFAULTS / COPILOT_ENV_DEFAULTS. Dropping the duplicates
would risk the install-step `<agent> --version` smoke check losing
config that runs BEFORE pytest loads conftest. Leaving for a separate
change after the install-step semantics are confirmed.
Signed-off-by: Tanzim Saqib <[email protected]>
tsaqib
added a commit
that referenced
this pull request
May 29, 2026
* fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows
skills-evals/tests/conftest.py defaults the agent filter to "claude"
when STAX_AGENT_KEY is unset (per the comment in
skills-eval-opencode.yml's env block: "Default is 'claude'"). The two
copilot workflows never set the var, so every Copilot CLI run was
collected, then immediately deselected:
[conftest] deselected 4 tests not matching agent='claude':
['tests/test_copilot_stream_smoke.py::...', ...]
============================ no tests ran in 0.12s =============================
##[error]Process completed with exit code 5.
Every other backend (cline / opencode / omp / pi, both scopes) sets
STAX_AGENT_KEY correctly; copilot was the lone outlier. Adds the
matching line to both copilot env blocks.
Out of scope: claude runs against this branch are now reporting an
upstream DeepSeek wire incompatibility (`API Error: 400 ... messages[1].role:
unknown variant 'system', expected 'user' or 'assistant'`) — Claude
Code 2.1.156 installed via @latest is emitting a system-role message
mid-stream that DeepSeek's Anthropic-compat endpoint rejects. That's
a Claude Code / DeepSeek version interaction, not a workflow bug, and
a separate change (pinning to a known-good Claude Code version) will
need to address it.
Signed-off-by: Tanzim Saqib <[email protected]>
* ci(skills-eval): unify 12 workflows, drop stale comments and dead steps
Followup to the line-by-line audit. None of these change behavior on a
green path; they remove inconsistencies that were silent failure
vectors when the workflows drifted.
Behavioral changes:
- Drop the redundant per-agent `setup-node@v6` block from 10 of 12
workflows (everything except omp/omp-user-scope, which install via
curl and never had a second block). The first setup-node at the
top of each job already configures node 22 with the frontend's
package-lock cache; calling it again with no cache settings was a
no-op. PR #41 kept it deliberately to anchor the per-agent comment,
but the comments now sit on the npm install step where they
belong.
- Change `skills-eval-cline.yml`, `skills-eval-cline-user-scope.yml`,
`skills-eval-copilot.yml`, `skills-eval-copilot-user-scope.yml`
from the explicit 4-file `pytest tests/test_<agent>_*.py` selector
to bare `uv run pytest`. The conftest agent filter
(`STAX_AGENT_KEY=<key>`) already picks the right tests for every
other backend; the explicit form would silently skip any new test
file added to the matching backend. omp's redundant `export PATH=...`
(only affects the current step) is dropped; the GITHUB_PATH line
propagates correctly on its own.
Text fixes (no behavior change):
- Unify the `timeout-minutes: 60` comment across all 12 to one
sentence that doesn't claim a specific scenario count. claude.yml
said "One scenario today (TODO app)" and copilot.yml said "smoke
+ three scenario tests" for equally-shaped 4-5 test workloads.
- Unify the `run pytest` comment around `conftest.pytest_collection_modifyitems`
instead of the false "alphabetical pytest order" claim. Fix
`# test_stream_json_smoke.py runs first` in claude.yml /
claude-user-scope.yml — the actual filename is
`test_claude_stream_json_smoke.py`.
- Drop the "pinning is preferable to @latest" line from
claude.yml / claude-user-scope.yml. The workflows install
@latest, so the policy contradicted the implementation. Replace
with a "fragile but kept here" framing matching cline / copilot /
opencode / pi.
- Strip stale header-docblock claims:
- cline.yml "Mirrors skills-eval-opencode.yml" — it doesn't
(opencode uses bare pytest, cline used an explicit selector).
- cline.yml `Run uv run pytest tests/test_cline_*` glob hint —
the actual command was an explicit file list, not a glob.
- omp.yml "Drivers, judges, and test files live in a follow-up
PR" — the follow-up has shipped; tests live under
skills-evals/tests/.
- pi.yml `stax init --agents codex` — pi has its own
first-class agentTargets entry now, so the doc was describing
a transitional shape that no longer applies.
- Strip user-scope variants down to a minimal docblock
(`# Identical to skills-eval-<agent>.yml EXCEPT for
STAX_INSTALL_SCOPE.`) and drop the duplicated env-block /
per-step comments. claude-user-scope.yml previously carried the
full claude.yml comments; the other 5 user-scope variants
stripped them. Pick the stripped pattern uniformly.
Verified locally: `STAX_AGENT_KEY=<key> uv run pytest --collect-only`
for every key in {claude, opencode, copilot, pi, cline, omp} still
collects 25 tests with the conftest filter selecting the active
backend's set. All 12 YAML files parse clean.
Out of scope: workflow env blocks for claude / claude-user-scope and
copilot / copilot-user-scope still duplicate conftest's
CLAUDE_ENV_DEFAULTS / COPILOT_ENV_DEFAULTS. Dropping the duplicates
would risk the install-step `<agent> --version` smoke check losing
config that runs BEFORE pytest loads conftest. Leaving for a separate
change after the install-step semantics are confirmed.
Signed-off-by: Tanzim Saqib <[email protected]>
---------
Signed-off-by: Tanzim Saqib <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
skills-eval-*.ymlworkflow has been broken since049a75b(Vite MPA +//go:embed all:frontend/distinserver.go). Theinstall stax to PATHstep runsgo install .against a missingfrontend/dist/, so the compile aborts withpattern all:frontend/dist: no matching files foundbefore the eval body ever runs.test.yml,release.yml, andwindows-cli.ymlwere updated alongside the embed directive; the 12 manually-triggered skills-eval workflows were not. This PR closes that gap.test.yml:48-64: hoistssetup-node(with npm cache keyed onfrontend/package-lock.json) ahead of the Go install step and runsnpm ci+npm run buildinfrontend/. The per-agentsetup-nodeblock later in each workflow is left in place — a second call against the same node version is a no-op and keeps the agent-specific comments next to the npm install step they document.setup-nodeblock at all (the installer is curl-based); they get one here purely for the frontend build.Affected runs:
Test plan
workflow_dispatchre-run ofskills eval / claude codeagainst this branch reaches at least theinstall claude codestepworkflow_dispatchre-run ofskills eval / opencodeagainst this branch reaches at least theinstall opencodestepworkflow_dispatchre-run ofskills eval / opencode (user scope)against this branch reaches at least theinstall opencodestepvet,govulncheck,lint,test,e2e) green