Skip to content

fix(cli): document --backend in qa setup usage and docs - #434

Merged
barbatdev merged 4 commits into
mainfrom
fix/336-qa-setup-usage
Nov 12, 2026
Merged

barbatdev merged 4 commits into
mainfrom
fix/336-qa-setup-usage

Conversation

@barbatdev

Copy link
Copy Markdown
Contributor

Pull Request

Linked Issue

Fixes #336

PR Type

  • type:feature Feature
  • type:bug Bug fix
  • type:docs Documentation
  • type:refactor Refactoring
  • type:chore Maintenance
  • type:breaking-change Breaking change

Summary

#335 added per-backend routes to qa setup, but qa --help, the root help and docs/operating.md still described setup without --backend and as pi-only. Help and docs now describe the real options from one shared usage line.

Changes

  • internal/cli/qasetup.go: qaSetupUsageLine constant, used by qaSetupUsage.
  • internal/cli/doctor.go: qaUsage uses the same constant, so qa --help, the root help and qa setup refusals print one line: usage: cortex qa setup --catalog <dir> --provider <p> --model <m> [--backend pi|claude|opencode] [--effort <e>].
  • internal/cli/qa_test.go: TestQAHelpDocumentsSetupBackendLikeSetupUsage.
  • docs/operating.md: setup walkthrough shows --backend and --effort defaults; removes "pi-backend scoped (Bug: qa route profiles are backend-scoped but qa setup can only configure pi routes #328)"; describes routes per role and backend and the route_incomplete refusal.

Focused Tests

Claim Test command Observed result Limit State Observed by
Help lacked --backend go test ./internal/cli -run TestQAHelpDocumentsSetupBackendLikeSetupUsage -count=1 before the fix Failed: the old usage line Verified Agent that made the change
Help and setup refusal print the same line Same test after the fix qa --help exit 0, empty stderr, exact line; qa setup with no arguments exits with usage, same line on stderr Verified Agent that made the change
Existing help limits hold TestQAHelp, TestQAHelpDocumentsEvidenceSidecarSchema Pass; root help stays under 2048 bytes Small margin left Verified Agent that made the change
Full suites on current main go vet ./...; go test ./... -count=1; python3 -m unittest discover -s tests Clean; 35 packages ok; OK Verified Parent re-run

Not tested: qa setup --backend claude or opencode against real runtimes; only help and docs changed.

Failed or not done: none.

Warnings: the root help is close to its 2048-byte limit; the next addition may need to shorten it.

Runtime Harness

N/A: help text and documentation.

Rollback

Revert the single fix commit.

Chain Dependencies / Out of Scope

Checklist

  • Linked issue is open and status:approved.
  • Exactly one matching type:* label is applied.
  • Focused tests and runtime harness evidence are recorded.
  • Rollback and out-of-scope boundaries are documented.

@barbatdev barbatdev added the type:bug Bug fix label Nov 12, 2026
@barbatdev

Copy link
Copy Markdown
Contributor Author

Fleet review (glmflash)

Lens: adversarial. This PR fixes both blocking findings of correction issue #336 (stale help line and stale pi-scoped docs) and I could not construct a concrete failure against it. Verified from the head tree: qaSetupUsageLine (internal/cli/qasetup.go) is now referenced by both qaSetupUsage and the qaUsage const in internal/cli/doctor.go, so qa --help, the root help, and the qa setup refusal all emit the identical line naming --backend; the added test pins both surfaces. The new docs claims in docs/operating.md match the actual setup behavior in the head tree: only the pi pair is probed against the runtime model table (backend=="pi" guard around qapi.AvailableModels), claude/opencode pairs are written unvalidated and refused at the run's preflight, and writeQAProfile merges per backend preserving routes for other backends — so an operator following the walkthrough on a pi-less machine now discovers --backend claude/opencode. Both acceptance criteria of #336 no longer reproduce. No blocking findings.

Suggestion (1)

  • Docs describe unvalidated claude/opencode pairs as refused only at preflight; consider naming the preflight stage explicitly (docs/operating.md:299): The new paragraph says an unavailable claude/opencode model 'is refused at the run's model_probe preflight'. The head code (internal/cli/qasetup.go, backend=="pi" guard) confirms setup writes such pairs unvalidated, so the claim is accurate; naming the exact preflight stage/error token an operator will see would make the failure self-diagnosing. Not required for merge.

Verdict: ready to merge

@barbatdev
barbatdev merged commit 9b67a95 into main Nov 12, 2026
2 checks passed
@barbatdev
barbatdev deleted the fix/336-qa-setup-usage branch November 12, 2026 20:51
@barbatdev

Copy link
Copy Markdown
Contributor Author

Fleet review (deepseek)

Lens: evidence + test design. Both blocking findings of the correction issue are closed at this head. qaSetupUsageLine is one constant in qasetup.go used by qaSetupUsage and by qaUsage in doctor.go, and because rootHelp ends with qaHelp, all three help/refusal surfaces now print the same --backend-bearing line, so the two mutually inconsistent usage lines are gone. docs/operating.md no longer calls setup pi-backend scoped: it shows [--backend pi|claude|opencode] [--effort <e>], says routes are keyed by role and then backend token, and states the route_incomplete refusal with its route_missing_backend=<b> line and qa setup ... --backend <b> remedy. I checked every documentation claim against the code at the reviewed head and they hold: backend defaults to pi and effort to high (qasetup.go:37-50), only the pi pair is probed against the model table while other backends are written unvalidated (qasetup.go:81-100), writeQAProfile merges the requested backend into the existing profile and keeps other backends' routes (qasetup.go:128-142), and the refusal line is doctor.go:417-418. The new test is real evidence for the help surface: qa --help must exit 0 with empty stderr and contain the exact line, and qa setup with no arguments must exit exitUsage with stdout empty and that same line as the stderr suffix. No blocking defect; one test-surface suggestion below.

Suggestion (1)

  • The test covers two of the three surfaces the PR claims print the shared line (internal/cli/qa_test.go:149): The PR states that qa --help, the root help and qa setup refusals print one line, but TestQAHelpDocumentsSetupBackendLikeSetupUsage asserts only qa --help and the qa setup refusal. Root help is covered transitively today because rootHelp concatenates qaHelp, so nothing is broken; the assertion is what would catch a future edit that gives the root help its own setup line, which is exactly the drift this correction was opened for. One extra assertion (Run(ctx, []string{"--help"}, ...) contains qaSetupUsageLine) pins the third surface at no cost to the existing 2048-byte limit check, and the PR already flags that the root help has little margin left, so a regression there is likely to arrive as an edit to that surface.

Verdict: ready to merge

@barbatdev

Copy link
Copy Markdown
Contributor Author

Fleet review (mimo)

Lens: requirements. Both acceptance criteria of correction issue #336 are satisfied at head af7b7e1. (1) qa --help / root help: the setup usage is now one shared constant, qaSetupUsageLine (internal/cli/qasetup.go:105), consumed by qaUsage (internal/cli/doctor.go:243), so every surface the issue named (qa --help, root help, qa run refusals, qa setup refusal) prints the line that includes [--backend pi|claude|opencode]; the old stale literal is gone. The new test TestQAHelpDocumentsSetupBackendLikeSetupUsage (internal/cli/qa_test.go:345) pins qa --help and the qa setup refusal against that exact line. (2) docs/operating.md: the walkthrough no longer says 'pi-backend scoped (#328)', the step-2 command now shows [--backend pi|claude|opencode] [--effort ] with explicit pi-less guidance, and the new paragraph after the profile example explains backend-keyed routes. I verified every load-bearing doc claim against the code at head: writeQAProfile merges and preserves other backends' routes per role, only the pi pair is probed at setup (AvailableModels guarded by backend == "pi"), effort defaults to high, route_incomplete refusals print route_missing_backend= with the cortex qa setup --backend remedy (internal/cli/doctor.go:418), and model_probe is a real preflight stage name. A repo-wide sweep (README.md, docs/*, AGENTS.md) shows no other surface mentioning qa setup, so no stale copy remains. One testability suggestion only; no blocking findings.

Suggestion (1)

  • New test does not assert the root-help surface named in the issue (internal/cli/qa_test.go:345): Issue Corrección: fix(qa): configure per-backend QA routes in qa setup (#335) #336 criterion 1 covers qa --help, the root help, and the qa run invalid-argument refusals (the qaUsage consumers). The new test asserts only qa --help (contains) and the qa setup refusal (suffix). The root help and qa run refusals are correct today only because they embed the same qaSetupUsageLine constant through qaHelp/qaUsage; if a later edit composes any of those surfaces from a different string, the exact drift this correction issue blocked returns and no test fails. Asserting that root help also contains the shared line would pin the surface the issue explicitly named.

Verdict: ready to merge

@barbatdev

Copy link
Copy Markdown
Contributor Author

Post-merge QA (glm)

The merge closes #336's two acceptance criteria: the single qaSetupUsageLine constant is now shared by qaSetupUsage (setup refusals) and qaUsage (qa --help / root help), so the binary can no longer print two mutually inconsistent setup usage lines, and docs/operating.md no longer describes setup as pi-backend scoped: the walkthrough names --backend, the 'pi-backend scoped (#328)' sentence is replaced by per-backend routing with route preservation, the JSON example's route keying is explained, and the claude/opencode model-probe asymmetry (checked only at run preflight) is documented. TestQAHelpDocumentsSetupBackendLikeSetupUsage pins the exact line on both surfaces (help exit 0 on stdout, bare qa setup exit usage with the same line on stderr). The PR's own warning stands: the root help is close to its 2048-byte cap, so the next addition may need to shorten it.

No findings.

Verdict: clean

@barbatdev

Copy link
Copy Markdown
Contributor Author

Post-merge QA (deepseek)

Lens: evidence + test design. Both acceptance criteria are met on read: qa --help and the qa setup refusal now share qaSetupUsageLine (internal/cli/qasetup.go), which carries [--backend pi|claude|opencode] [--effort <e>], and the operator walkthrough no longer says routes are 'pi-backend scoped (#328)' — it documents --backend, the per-backend keying, route_incomplete and the --effort default, and the tokens it cites do exist (route_incomplete in internal/qaroute/resolve.go, route_missing_backend in internal/cli/doctor.go). The added test goes through Run for both qa --help and a bare qa setup, asserting exit codes, empty stdout on the refusal and exact line equality, so the two surfaces cannot drift again. Remaining gaps are evidence-only: the third surface the PR names (root help) is not asserted, and two behaviours the new prose promises (preserving other backends' routes, pi-only model-table checking) have no test in this merge. No blocking defect found.

Question (2)

  • The PR names three surfaces; the test asserts two (internal/cli/qa_test.go:345): The PR body says qa --help, the root help and qa setup refusals print one line, but TestQAHelpDocumentsSetupBackendLikeSetupUsage only exercises qa --help and qa setup. The root help path is the one the PR itself warns is close to its 2048-byte ceiling, so the surface most likely to change is the untested one. Adding the --help root invocation to the same test (exact same expected line) closes the claim without new machinery.
  • Two behaviours the new walkthrough promises have no merged evidence (docs/operating.md:305): The rewritten step 2 now states that qa setup "preserves routes already configured for other backends, so running it once per backend routes the whole fleet" and that "Only the pi pair is checked against the runtime's model table at setup; a claude or opencode pair is written as given". Neither statement is exercised by a test added here, and they are the operational path the issue's scenario depends on (running setup once per backend on a pi-less machine). If the preservation claim is false, the second qa setup --backend claude silently drops the pi routes and the doc is the only place an operator would look.

Verdict: clean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Corrección: fix(qa): configure per-backend QA routes in qa setup (#335)

1 participant