Skip to content

ci: build frontend before go install in every skills-eval workflow - #41

Merged
tsaqib merged 1 commit into
mainfrom
fix/skills-eval-frontend-build
May 29, 2026
Merged

tsaqib merged 1 commit into
mainfrom
fix/skills-eval-frontend-build

Conversation

@tsaqib

@tsaqib tsaqib commented May 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Every skills-eval-*.yml workflow has been broken since 049a75b (Vite MPA + //go:embed all:frontend/dist in server.go). The install stax to PATH step runs go install . against a missing frontend/dist/, so the compile aborts with pattern all:frontend/dist: no matching files found before the eval body ever runs.
  • test.yml, release.yml, and windows-cli.yml were updated alongside the embed directive; the 12 manually-triggered skills-eval workflows were not. This PR closes that gap.
  • Mirrors the build order from test.yml:48-64: hoists setup-node (with npm cache keyed on frontend/package-lock.json) ahead of the Go install step and runs npm ci + npm run build in frontend/. The per-agent setup-node block later in each workflow is left in place — a second call against the same node version is a no-op and keeps the agent-specific comments next to the npm install step they document.
  • omp variants previously had no setup-node block at all (the installer is curl-based); they get one here purely for the frontend build.

Affected runs:

Test plan

  • workflow_dispatch re-run of skills eval / claude code against this branch reaches at least the install claude code step
  • workflow_dispatch re-run of skills eval / opencode against this branch reaches at least the install opencode step
  • workflow_dispatch re-run of skills eval / opencode (user scope) against this branch reaches at least the install opencode step
  • Local pre-push (vet, govulncheck, lint, test, e2e) green

server.go embeds `frontend/dist/` via `//go:embed all:frontend/dist`
(commit 049a75b). The directory is gitignored, so any Go compile
fails with `pattern all:frontend/dist: no matching files found` if
the Vite build has not run first.

`test.yml`, `release.yml`, and `windows-cli.yml` were updated to
build the frontend before invoking Go. The twelve manually-triggered
`skills-eval-*.yml` workflows were not, so every one of them has
been broken since 049a75b — failing at the `install stax to PATH`
step before the eval body ever runs.

Mirror the build order from `test.yml:48-64`: hoist `setup-node`
(with npm cache keyed on `frontend/package-lock.json`) ahead of the
Go install step and run `npm ci` + `npm run build` in `frontend/`.
The existing per-agent `setup-node` block later in each workflow is
left in place — a second call against the same node version is a
no-op and keeps the agent-specific comments next to the npm install
step they document.

The omp variants previously had no `setup-node` block at all
(the omp installer is curl-based); they get one here purely for the
frontend build.

Signed-off-by: Tanzim Saqib <[email protected]>
@tsaqib
tsaqib merged commit 22bbf25 into main May 29, 2026
6 checks passed
@tsaqib
tsaqib deleted the fix/skills-eval-frontend-build branch May 29, 2026 03:45
tsaqib added a commit that referenced this pull request May 29, 2026
Followup to the line-by-line audit. None of these change behavior on a
green path; they remove inconsistencies that were silent failure
vectors when the workflows drifted.

Behavioral changes:
  - Drop the redundant per-agent `setup-node@v6` block from 10 of 12
    workflows (everything except omp/omp-user-scope, which install via
    curl and never had a second block). The first setup-node at the
    top of each job already configures node 22 with the frontend's
    package-lock cache; calling it again with no cache settings was a
    no-op. PR #41 kept it deliberately to anchor the per-agent comment,
    but the comments now sit on the npm install step where they
    belong.
  - Change `skills-eval-cline.yml`, `skills-eval-cline-user-scope.yml`,
    `skills-eval-copilot.yml`, `skills-eval-copilot-user-scope.yml`
    from the explicit 4-file `pytest tests/test_<agent>_*.py` selector
    to bare `uv run pytest`. The conftest agent filter
    (`STAX_AGENT_KEY=<key>`) already picks the right tests for every
    other backend; the explicit form would silently skip any new test
    file added to the matching backend. omp's redundant `export PATH=...`
    (only affects the current step) is dropped; the GITHUB_PATH line
    propagates correctly on its own.

Text fixes (no behavior change):
  - Unify the `timeout-minutes: 60` comment across all 12 to one
    sentence that doesn't claim a specific scenario count. claude.yml
    said "One scenario today (TODO app)" and copilot.yml said "smoke
    + three scenario tests" for equally-shaped 4-5 test workloads.
  - Unify the `run pytest` comment around `conftest.pytest_collection_modifyitems`
    instead of the false "alphabetical pytest order" claim. Fix
    `# test_stream_json_smoke.py runs first` in claude.yml /
    claude-user-scope.yml — the actual filename is
    `test_claude_stream_json_smoke.py`.
  - Drop the "pinning is preferable to @latest" line from
    claude.yml / claude-user-scope.yml. The workflows install
    @latest, so the policy contradicted the implementation. Replace
    with a "fragile but kept here" framing matching cline / copilot /
    opencode / pi.
  - Strip stale header-docblock claims:
      - cline.yml "Mirrors skills-eval-opencode.yml" — it doesn't
        (opencode uses bare pytest, cline used an explicit selector).
      - cline.yml `Run uv run pytest tests/test_cline_*` glob hint —
        the actual command was an explicit file list, not a glob.
      - omp.yml "Drivers, judges, and test files live in a follow-up
        PR" — the follow-up has shipped; tests live under
        skills-evals/tests/.
      - pi.yml `stax init --agents codex` — pi has its own
        first-class agentTargets entry now, so the doc was describing
        a transitional shape that no longer applies.
  - Strip user-scope variants down to a minimal docblock
    (`# Identical to skills-eval-<agent>.yml EXCEPT for
    STAX_INSTALL_SCOPE.`) and drop the duplicated env-block /
    per-step comments. claude-user-scope.yml previously carried the
    full claude.yml comments; the other 5 user-scope variants
    stripped them. Pick the stripped pattern uniformly.

Verified locally: `STAX_AGENT_KEY=<key> uv run pytest --collect-only`
for every key in {claude, opencode, copilot, pi, cline, omp} still
collects 25 tests with the conftest filter selecting the active
backend's set. All 12 YAML files parse clean.

Out of scope: workflow env blocks for claude / claude-user-scope and
copilot / copilot-user-scope still duplicate conftest's
CLAUDE_ENV_DEFAULTS / COPILOT_ENV_DEFAULTS. Dropping the duplicates
would risk the install-step `<agent> --version` smoke check losing
config that runs BEFORE pytest loads conftest. Leaving for a separate
change after the install-step semantics are confirmed.

Signed-off-by: Tanzim Saqib <[email protected]>
tsaqib added a commit that referenced this pull request May 29, 2026
* fix(ci): set STAX_AGENT_KEY=copilot on both copilot workflows

skills-evals/tests/conftest.py defaults the agent filter to "claude"
when STAX_AGENT_KEY is unset (per the comment in
skills-eval-opencode.yml's env block: "Default is 'claude'"). The two
copilot workflows never set the var, so every Copilot CLI run was
collected, then immediately deselected:

    [conftest] deselected 4 tests not matching agent='claude':
      ['tests/test_copilot_stream_smoke.py::...', ...]
    ============================ no tests ran in 0.12s =============================
    ##[error]Process completed with exit code 5.

Every other backend (cline / opencode / omp / pi, both scopes) sets
STAX_AGENT_KEY correctly; copilot was the lone outlier. Adds the
matching line to both copilot env blocks.

Out of scope: claude runs against this branch are now reporting an
upstream DeepSeek wire incompatibility (`API Error: 400 ... messages[1].role:
unknown variant 'system', expected 'user' or 'assistant'`) — Claude
Code 2.1.156 installed via @latest is emitting a system-role message
mid-stream that DeepSeek's Anthropic-compat endpoint rejects. That's
a Claude Code / DeepSeek version interaction, not a workflow bug, and
a separate change (pinning to a known-good Claude Code version) will
need to address it.

Signed-off-by: Tanzim Saqib <[email protected]>

* ci(skills-eval): unify 12 workflows, drop stale comments and dead steps

Followup to the line-by-line audit. None of these change behavior on a
green path; they remove inconsistencies that were silent failure
vectors when the workflows drifted.

Behavioral changes:
  - Drop the redundant per-agent `setup-node@v6` block from 10 of 12
    workflows (everything except omp/omp-user-scope, which install via
    curl and never had a second block). The first setup-node at the
    top of each job already configures node 22 with the frontend's
    package-lock cache; calling it again with no cache settings was a
    no-op. PR #41 kept it deliberately to anchor the per-agent comment,
    but the comments now sit on the npm install step where they
    belong.
  - Change `skills-eval-cline.yml`, `skills-eval-cline-user-scope.yml`,
    `skills-eval-copilot.yml`, `skills-eval-copilot-user-scope.yml`
    from the explicit 4-file `pytest tests/test_<agent>_*.py` selector
    to bare `uv run pytest`. The conftest agent filter
    (`STAX_AGENT_KEY=<key>`) already picks the right tests for every
    other backend; the explicit form would silently skip any new test
    file added to the matching backend. omp's redundant `export PATH=...`
    (only affects the current step) is dropped; the GITHUB_PATH line
    propagates correctly on its own.

Text fixes (no behavior change):
  - Unify the `timeout-minutes: 60` comment across all 12 to one
    sentence that doesn't claim a specific scenario count. claude.yml
    said "One scenario today (TODO app)" and copilot.yml said "smoke
    + three scenario tests" for equally-shaped 4-5 test workloads.
  - Unify the `run pytest` comment around `conftest.pytest_collection_modifyitems`
    instead of the false "alphabetical pytest order" claim. Fix
    `# test_stream_json_smoke.py runs first` in claude.yml /
    claude-user-scope.yml — the actual filename is
    `test_claude_stream_json_smoke.py`.
  - Drop the "pinning is preferable to @latest" line from
    claude.yml / claude-user-scope.yml. The workflows install
    @latest, so the policy contradicted the implementation. Replace
    with a "fragile but kept here" framing matching cline / copilot /
    opencode / pi.
  - Strip stale header-docblock claims:
      - cline.yml "Mirrors skills-eval-opencode.yml" — it doesn't
        (opencode uses bare pytest, cline used an explicit selector).
      - cline.yml `Run uv run pytest tests/test_cline_*` glob hint —
        the actual command was an explicit file list, not a glob.
      - omp.yml "Drivers, judges, and test files live in a follow-up
        PR" — the follow-up has shipped; tests live under
        skills-evals/tests/.
      - pi.yml `stax init --agents codex` — pi has its own
        first-class agentTargets entry now, so the doc was describing
        a transitional shape that no longer applies.
  - Strip user-scope variants down to a minimal docblock
    (`# Identical to skills-eval-<agent>.yml EXCEPT for
    STAX_INSTALL_SCOPE.`) and drop the duplicated env-block /
    per-step comments. claude-user-scope.yml previously carried the
    full claude.yml comments; the other 5 user-scope variants
    stripped them. Pick the stripped pattern uniformly.

Verified locally: `STAX_AGENT_KEY=<key> uv run pytest --collect-only`
for every key in {claude, opencode, copilot, pi, cline, omp} still
collects 25 tests with the conftest filter selecting the active
backend's set. All 12 YAML files parse clean.

Out of scope: workflow env blocks for claude / claude-user-scope and
copilot / copilot-user-scope still duplicate conftest's
CLAUDE_ENV_DEFAULTS / COPILOT_ENV_DEFAULTS. Dropping the duplicates
would risk the install-step `<agent> --version` smoke check losing
config that runs BEFORE pytest loads conftest. Leaving for a separate
change after the install-step semantics are confirmed.

Signed-off-by: Tanzim Saqib <[email protected]>

---------

Signed-off-by: Tanzim Saqib <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant