Summary
The validator catalog (recipes/validators/catalog.yaml) is currently embedded in the binary and not overridable via the --data flag. This prevents external teams from adding custom validation checks without rebuilding the CLI.
Problem
The --data flag already supports layering external data over embedded recipe data:
registry.yaml — merged (external components override by name, new ones added)
- Overlays, values, mixins — external fully replaces embedded if present
However, catalog.Load() reads exclusively from the embedded filesystem (recipes.FS.ReadFile("validators/catalog.yaml")), bypassing the DataProvider interface entirely.
This means teams that need cluster-specific or platform-specific validation checks (e.g., ComputeDomain-aware DRA tests for GB200 clusters) must fork and rebuild the CLI to add them.
Proposed Solution
Extend the --data flag to also layer validators/catalog.yaml:
- Update
catalog.Load() to use the global DataProvider instead of reading directly from recipes.FS
- Use merge semantics similar to
registry.yaml: external validators override by name, new ones are added
- Validate external catalog entries (image, phase, timeout) the same way embedded ones are validated
Use Case
External teams maintaining custom validation suites (e.g., Dynamo cluster validators) can define additional checks in their own validators/catalog.yaml and run them alongside built-in checks using --data ./local-recipes without rebuilding aicr.
Summary
The validator catalog (
recipes/validators/catalog.yaml) is currently embedded in the binary and not overridable via the--dataflag. This prevents external teams from adding custom validation checks without rebuilding the CLI.Problem
The
--dataflag already supports layering external data over embedded recipe data:registry.yaml— merged (external components override by name, new ones added)However,
catalog.Load()reads exclusively from the embedded filesystem (recipes.FS.ReadFile("validators/catalog.yaml")), bypassing theDataProviderinterface entirely.This means teams that need cluster-specific or platform-specific validation checks (e.g., ComputeDomain-aware DRA tests for GB200 clusters) must fork and rebuild the CLI to add them.
Proposed Solution
Extend the
--dataflag to also layervalidators/catalog.yaml:catalog.Load()to use the globalDataProviderinstead of reading directly fromrecipes.FSregistry.yaml: external validators override by name, new ones are addedUse Case
External teams maintaining custom validation suites (e.g., Dynamo cluster validators) can define additional checks in their own
validators/catalog.yamland run them alongside built-in checks using--data ./local-recipeswithout rebuildingaicr.