feat(coverage): generated CUJ/CLI coverage matrix (RQ3)#1316
Conversation
Add tools/coverage, a generator that emits docs/user/coverage-matrix.md — a structural matrix of which critical user journeys and CLI verbs are exercised by an in-repo test or demo, on what hardware class, and at what cadence. CLI verbs are derived from the live pkg/cli command registry (new exported cli.RootCommand()), so a new verb surfaces as a not-yet-covered row instead of being silently dropped. Azure UAT trees render as stubbed (DC6, #1280); live pass/fail is a link into the AICR TestGrid, never embedded. The page is fully generated (no splice markers), so it stays inside the tools/check-docs-mdx gate with zero exclusions. Adds make coverage-docs / coverage-check (opt-in, not wired into qualify) and a weekly Helm-free clone of bom-refresh. Registered in docs/index.yml and cross-linked from cli-reference; the reverse recipe-health cross-link is deferred to RQ1 (#1283) since that page ships with #1229. Closes #1282
Coverage Report ✅
Coverage BadgeMerging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. |
This comment was marked as resolved.
This comment was marked as resolved.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ngs.Contains - Add a top-level concurrency group to coverage-matrix-refresh so the scheduled and manual triggers cannot race the shared chore/coverage-refresh branch. - Add an inline rationale for the single-use `make coverage-docs` shell step. - Reword the cli-reference cross-link sentence for clarity. - Replace the hand-rolled contains() test helper with strings.Contains.
njhensley
left a comment
njhensley
left a comment
There was a problem hiding this comment.
Reviewed the latest push and reconciled against the existing CodeRabbit comments. CodeRabbit's workflow/grammar/test-helper items look addressed, so I focused this review on remaining coverage-matrix correctness issues.
…b expansion Respond to @njhensley's review on #1316: - P0: derive CUJ/UAT coverage from the scheduled workflow's wired runner + config intent, not tree presence. cuj2-inference now renders `stubbed` (assets present, no scheduled inference config) instead of falsely claiming live nightly H100. - P1: scan the extensionless UAT runner scripts and match the shell argv-array invocation form, so `evidence verify` (run via `args=(evidence verify ...)`) is correctly attributed to UAT. - P1: include subcommands of action-bearing parents, so `recipe list` / `recipe verify-catalog` appear as rows instead of being collapsed into `recipe`. - P1: render only the AICR versions actually exercised (main); the multi-version axis is deferred to DC5. - P2: reword the legend so coverage reflects CUJ/CLI *journey* signals (chainsaw/KWOK/UAT/demo); Go unit-test coverage is a separate gate. Adds tools/coverage/wiring.go (workflow-entrypoint parsing) and updates tests.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/user/coverage-matrix.md (1)
5-18:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winClarify whether demo-backed rows count as covered.
The intro says the matrix tracks journeys exercised by an in-repo test or demo, but the legend defines
coveredonly in terms of chainsaw/KWOK tests and wired UAT runners. That makes the demo-backed rows ambiguous, andcuj2-inference-dynamois especially confusing because it is listed asdemo-exercised but still markedstubbed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/user/coverage-matrix.md` around lines 5 - 18, The doc is ambiguous about whether demos count as "covered" (intro says "test or demo" but Legend's `covered` mentions only chainsaw/KWOK and wired UAT); update the intro and Legend to explicitly state how demos are treated (e.g., define a new status like `demo-only` or state that demos count as `covered` only when accompanied by scheduled UAT runs), then reconcile the example row `cuj2-inference-dynamo` (either change its Status from `stubbed` to `demo-only` or add an explanatory Note) so the table and legend consistently reflect the intended semantics for demo-backed rows and remove the contradiction between the intro, Legend (`covered`, `stubbed`), and the sample CUJ rows.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/user/coverage-matrix.md`:
- Around line 5-18: The doc is ambiguous about whether demos count as "covered"
(intro says "test or demo" but Legend's `covered` mentions only chainsaw/KWOK
and wired UAT); update the intro and Legend to explicitly state how demos are
treated (e.g., define a new status like `demo-only` or state that demos count as
`covered` only when accompanied by scheduled UAT runs), then reconcile the
example row `cuj2-inference-dynamo` (either change its Status from `stubbed` to
`demo-only` or add an explanatory Note) so the table and legend consistently
reflect the intended semantics for demo-backed rows and remove the contradiction
between the intro, Legend (`covered`, `stubbed`), and the sample CUJ rows.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 9fddee93-6bca-43e3-9d37-69e633a0e1c0
📒 Files selected for processing (8)
docs/user/coverage-matrix.mdtools/coverage/build.gotools/coverage/build_test.gotools/coverage/render.gotools/coverage/scan.gotools/coverage/verbs.gotools/coverage/verbs_test.gotools/coverage/wiring.go
njhensley
left a comment
njhensley
left a comment
There was a problem hiding this comment.
Approved. The coverage-matrix correctness issues from my previous pass look addressed at the latest head.
Non-blocking follow-up: consider filtering hidden commands out of the user-facing matrix. tools/coverage/verbs.go skips built-ins but not cmd.Hidden, so recipe sign-catalog appears in docs/user/coverage-matrix.md even though pkg/cli/recipe_sign_catalog.go marks it hidden and CI-only. A small follow-up could skip hidden commands in expandVerb while keeping visible nested verbs like recipe list and recipe verify-catalog.
recipe sign-catalog is marked Hidden in pkg/cli/recipe_sign_catalog.go (CI-only, not user-facing) but was appearing in docs/user/coverage-matrix.md. Skip cmd.Hidden in expandVerb alongside the existing builtins filter. Visible nested verbs (recipe list, recipe verify-catalog) are unaffected.
2 participants
Summary
Add
tools/coverage, a generator fordocs/user/coverage-matrix.md— a structural matrix of which CUJs and CLI verbs are exercised by an in-repo test or demo, on what hardware class, and at what cadence (RQ3 of epic #1265).Motivation / Context
Promotes CUJ/CLI coverage from demo+chainsaw-only into a versioned, generated, public matrix that runs on
mainweekly (never a merge gate), cross-linked to the other recipe-quality surfaces. CLI verbs are derived from the livepkg/clicommand registry so a new verb surfaces automatically as anot-yet-coveredrow instead of being silently dropped. Live pass/fail stays a link into the AICR TestGrid; this page is structural only.Fixes: #1282
Related: #1265 (RQ epic), #1283 (RQ1 adds the reverse recipe-health cross-link), #1280 (DC6 owns the Azure revive-or-retire)
Type of Change
Component(s) Affected
cmd/aicr,pkg/cli) — new exportedcli.RootCommand()(behavior-neutral)docs/)tools/coveragegenerator +coverage-matrix-refreshworkflowImplementation Notes
cli.RootCommand()(extracted from the existing builder;Execute()unchanged) so the generator walks the live tree. Pure command groups (Action-less, e.g.evidence) expand to subcommands; action commands stay single.coverage-matrix.mdis regenerated, so it passestools/check-docs-mdxwith zero exclusions — avoiding the single-fileEXCLUDEcarve-out the BOM doc needs (the issue's suggested{/* */}marker would itself fail the gate's bare-{check; HTML-comment markers fail Check 3).query/verify/trust update/skillare exercised; onlydiff/mirror list/evidence digestare genuinelynot-yet-covered. Azure UAT trees render asstubbed(link to DC6/DC7 — Azure UAT revive-or-retire #1280). Deterministic output (byte-stable across runs).docs/user/recipe-health.mddoes not exist onmainyet (ships with tools/health generator + make targets + public recipe-health matrix #1229), so the page references it in prose without a hyperlink to avoid a broken-link (lychee) failure; the reverse link is RQ1/RQ1 — recipe-health ⇄ TestGrid Evidence deep-link #1283's job.make coverage-docs/coverage-check(opt-in, not inqualify/lint/merge gate); weeklycoverage-matrix-refresh.yaml(Helm-free clone ofbom-refresh, cron offset 30m);docs/index.ymlregistration;cli-reference.mdcross-link;labeler.yml+CODEOWNERSfortools/coverage/**.Testing
tools/coverage: new package, 83.5% statement coverage.pkg/cli: addedTestRootCommandso the new exported func is not at 0%.make qualifygreen (tests, MDX gate on the new page, lint, e2e, scan).Risk Assessment
cli.RootCommand()) is a behavior-neutral wrapper over the existing builder. The matrix never gates merges.Rollout notes: The matrix fills in as the dynamic-clusters epic (DC3/DC4/DC5) lands more execution; today it honestly reports
not-yet-covered/stubbedwhere no executable signal exists.Checklist
make testwith-race)make lint)tools/bom/bom-refresh)git commit -S)