Skip to content

[Epic]: Validator capability: performance / conformance checks must cover every shipped workload variant #1044

Description

@mchmarny

Goal

Every performance- or conformance-phase validator must actually run against the workload variant the recipe deploys. When a recipe ships a workload the validator currently skippeds on, the gap is closed here — either by extending the validator to cover the new workload kind, or by adding a sibling check that does. A "passing" recipe must not be passing because the gate silently skipped.

Parent initiative: #1041.

Success criteria

  1. For every recipe in recipes/overlays/, the validators its validation.phases.*.checks reference do not return status: "skipped - <reason>" on the intended workload — they either pass or fail with a real finding.
  2. New workload kinds added under an existing intent (e.g., NIM under inference, alongside the existing Dynamo coverage) ship with first-day validator coverage; if the existing validator cannot be extended, a new check is filed and added to the relevant overlay.
  3. Contributor docs (docs/contributor/validations.md) describe the supported workload kinds for each check, so future overlay authors know which check to reference.

Scope

In scope:

  • Extending an existing validator to recognize a new workload kind (e.g., NIM as a peer of Dynamo).
  • Adding a new check when the runtime contract is sufficiently different that extending in place would be awkward.
  • Documenting the supported workload kinds per check.

Out of scope (other epics under #1041):

  • Adding a constraint to a recipe that already references a working validator (performance Epic).
  • Authoring new overlays (overlay-coverage Epic).
  • Validator framework architecture (separate initiative).

New validator capability gaps should be filed as standalone issues and attached here.

Metadata

Metadata

Assignees

Fields

No fields configured for Enhancement.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions