Skip to content

Epic: Recipe health tracking (ADR-009 V1) — enumeration, structural soundness, public matrix #1224

Description

@njhensley

Summary

Implement ADR-009: Recipe Health Tracking V1.

AICR ships ~50 leaf recipes resolved from ~79 overlays, but there is no command that enumerates them and no durable, communicated record of per-recipe health. This epic closes both gaps using only hermetic, offline, deterministic signals — reproducible from a checkout, no GPU, no network, no wall-clock on the computed path.

V1 ships exactly three things (ADR-009 §1):

  1. aicr recipe list — enumerate resolvable criteria combinations. ✅ Delivered by Add aicr recipe list to enumerate curated overlays as JSON (with criteria filters) #1207 / feat(recipe): add aicr recipe list subcommand for catalog enumeration #1208 (merged): the command plus the MetadataStore.ListCatalog enumeration seam.
  2. pkg/health.Compute() — per-recipe structural soundness from three graded signals (resolves, chart_pinned, constraints_wellformed) plus one descriptor (declared_coverage), rolled up to pass | warn | fail (unknown is held, never rolled up).
  3. One generated public matrix (docs/user/recipe-health.md), refreshed weekly by a drift-only bot PR — a faithful clone of the BOM generate-and-PR loop.

Children

Acceptance criteria

  1. aicr recipe list enumerates the curated maximal-leaf combinations (JSON/YAML/table, criteria + curated/source, filters); every row round-trips to a working aicr recipe --service … --accelerator …, proven by a test feeding each enumerated Criteria back through BuildFromCriteria. A4 adds the structural-status + coverage columns.
  2. pkg/health.Compute() is deterministic on the non-error path: the serialized Report is byte-identical across runs from the same checkout, no time.Now() on the computed path.
  3. A broken/unpinned/malformed-constraint recipe → fail/warn; a clean recipe → pass; a transient resolver error → unknown (held). One broken combo never blanks the matrix.
  4. make recipe-health-docs regenerates the marked region of docs/user/recipe-health.md; make recipe-health-check reports staleness and is not wired into make qualify.
  5. health-refresh.yaml opens a chore/health-refresh PR only on drift, weekly (cron-offset from bom-refresh), with the built-in GITHUB_TOKEN.
  6. Docs registered (docs/index.yml), aicr recipe list documented (cli-reference.md), cross-linked from component-catalog.md, CODEOWNERS + a pkg/health labeler.yml glob added.
  7. A measured compute-budget gate (A5) holds the generator under the ADR-009 sub-minute target.
  8. make qualify green; pkg/health and tools/health clear the 75% coverage floor (.settings.yaml).

Non-goals (deferred — ADR-009 What V1 does not ship)

Validation-posture / evidence-freshness axis; snapshot-dependent constraint replay (non-hermetic); committed recipe-health.json; committed internal detail doc; aicr recipe health ad-hoc subcommand; PR-time sticky-comment apparatus; kwok_scheduling and chart_drift signals; /v1/health REST endpoint; Supported/Preview/Experimental vocabulary; A–F letter grades; blocking merge gate; external dashboard/badges. Each has a pull-trigger in the ADR.

Design notes

  • Hermetic & advisory. Every V1 signal is offline and deterministic; the matrix is advisory (like make bom-check), never a merge gate.
  • No false confidence. The matrix's Evidence column is a literal pending (no evidence axis in V1); chart_pinned is ADR-006 layer-1 (chart version) only — not image digests.
  • The detailed, meta-reviewed implementation plan lives in docs/design/009-recipe-health-tracking-implementation-plan.md (pending commit).

References

ADR-009 (docs/design/009-recipe-health-tracking.md); enumeration command/seam #1207 / #1208 (merged).

Metadata

Metadata

Assignees

Type

Fields

No fields configured for Epic.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions