Skip to content

Recipe validation parity: coverage, performance, validator capability #1041

Description

@mchmarny

Goal

Every recipe AICR ships must be validated end-to-end: it must resolve from criteria, ship a runtime performance gate anchored in empirical thresholds, and execute against the validator capability its workload kind requires. A recipe must not be passing because it silently skipped a gate.

This is a multi-quarter strategic outcome — the first AICR Initiative. It does not have a single landing PR. It closes when each Epic underneath it closes.

Why this is an Initiative, not an Epic

The "Recipe validation parity" outcome cleanly factors into three independent deliverable streams:

  1. Overlay coverage — authoring-blocked work on recipes/overlays/.
  2. Performance phase — testbed-blocked work on empirical thresholds.
  3. Validator capability — engineering-blocked work on validators/*.

Each stream has its own owner shape, blocking dependency, and "done" definition. Grouping all leaf issues directly under a single tracker conflated those streams; the Epic tier separates them so each can be sequenced and reviewed independently.

This Initiative also sets the organizational precedent for Initiative → Epic → Issue in this project:

  • Initiative — strategic outcome, no single PR, lives until the outcome is real
  • Epic — deliverable chunk with concrete success criteria, may span multiple PRs
  • Issue / Enhancement / Bug — single tactical work unit, typically one PR

Success criteria

The Initiative is done when:

  1. recipes/overlays_validation_floor_test.go passes with AICR_VALIDATION_FLOOR_STRICT=1 and zero exemptions tracked to open issues.
  2. For every accelerator in pkg/recipe/criteria.go, aicr recipe --accelerator <x> --service <y> --intent <z> resolves a recipe end-to-end.
  3. Every validator referenced by an overlay's validation.phases.*.checks actually runs against the workload kind that overlay ships (no silent skipped paths).

Sub-issues (Epics)

New work that fits the scope of one of these Epics should be filed as a standalone issue and attached as a sub-issue of the appropriate Epic. New work that doesn't fit any of the three (e.g., a new gating axis surfaced by a future audit) is filed as a new Epic and attached here.

Explicitly out of scope

These belong to other initiatives, not this one:

  • Validation framework / runtime changes (validators package architecture, agent wiring).
  • Recipe data model changes (overlay schema, mixin shape).
  • Conformance certification work (downstream consumer of validated recipes).

Tracking sources of truth

  • Strict-floor: recipes/overlays_validation_floor_test.go (AICR_VALIDATION_FLOOR_STRICT=1).
  • Coverage: pkg/recipe/criteria.go cross-referenced with recipes/overlays/.
  • Validator capability: per-check validators/<phase>/*.go skip / pass logic.

Metadata

Metadata

Assignees

No one assigned

    Fields

    No fields configured for Initiative.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions