Skip to content

[Feature]: extensible criteria values via catalog-driven runtime registry (unblocks --data overlays) #998

Description

@mchmarny

Feature Summary

Make criteria values (service, accelerator, OS, intent, platform) extensible at runtime via the --data overlay catalog, instead of requiring a Go-code change and AICR rebuild for every new value.

Problem / Use Case

Today the criteria value lists in pkg/recipe/criteria.go are hardcoded switch arms inside ParseCriteria{Service,Accelerator,Intent,OS,Platform}Type(). Adding a new criteria value — e.g., an undisclosed services, a proprietary platform, or still embedded GPU SKUs — requires:

  • A code change in pkg/recipe/criteria.go
  • A new AICR binary build
  • A public OSS release for values that may not be public

This forces either a fork or an upstream contribution for each internal value, neither of which scales. AICR is intended to be one binary distributed across all clusters, with extensions composed via --data.

The catalog side is already open: pkg/recipe/provider.go walks --data recursively with filepath.WalkDir and does not validate spec.criteria values at load time. The gap is at CLI/API input parsing: ParseCriteria*Type() runs at flag-parse time, before the overlay catalog is loaded, and rejects values its hardcoded switch doesn't recognize:

aicr recipe --service customer-x --data /opt/internal-overlays/
  ↓ ParseCriteriaServiceType("customer-x") rejects at flag-parse time
  ↓ catalog from --data is never consulted
  ✗ exits with "invalid service type"

Proposed Solution

Three coupled changes (single PR — see Compatibility for why):

1. Catalog-driven criteria registry — pkg/recipe/criteria.go, pkg/recipe/provider.go

  • As overlays are loaded (both embedded and --data), register each spec.criteria.{service,accelerator,os,intent,platform} value into a runtime registry.
  • ParseCriteria*Type() consults the registry on miss: existing canonical/aliased switch arms first (fast path), then catalog-registered values, then error.
  • GetCriteria*Types() returns the union (embedded + registered) so CLI help reflects the current catalog.

2. Defer criteria validation until after catalog load — pkg/cli/recipe.go, pkg/recipe/handler.go, pkg/recipe/handler_query.go

  • CLI: parse --service/--accelerator/etc. as raw strings; defer Criteria.Validate() until after LayeredDataProvider is initialized with --data.
  • REST handler: same ordering — load catalog (or use a request-scoped one if --data is request-supplied), then validate request criteria.

3. Strict mode for OSS CI gate — pkg/cli/recipe.go, pkg/recipe/criteria.go

  • New flag --criteria-strict (default off; CI sets it on) that requires every criteria value to resolve to an embedded-OSS overlay, ignoring registry entries contributed by --data.
  • Prevents the upstream catalog from accidentally depending on internal-only values during dev.

Success Criteria

  • Given an external overlay directory at /opt/internal-overlays/ containing an overlay with spec.criteria.service: customer-x, aicr recipe --service customer-x --data /opt/internal-overlays/ succeeds and returns the matching recipe.
  • Same scenario via REST API: a GET /v1/recipes?service=customer-x with the API server started with --data /opt/internal-overlays/ succeeds.
  • aicr recipe --help text for --service/--accelerator/--os/--intent/--platform reflects values from the loaded catalog (embedded + --data), not just embedded.
  • With --criteria-strict, the same call fails with a clear error naming the value as unrecognized in the embedded catalog.
  • All existing OSS criteria values (gke, eks, h100, gb200, training, inference, ubuntu, cos, etc.) continue to validate with no behavior change.
  • Unit tests for the registry; CLI integration tests for --data + --service unknown-value in both default and strict modes; API handler test for the REST equivalent.
  • make qualify passes; per-package coverage on pkg/recipe/ and pkg/cli/ does not regress > 0.5%.

Alternatives Considered

  1. Fork AICR for the internal binary. Rejected — composition over inheritance; perpetual merge burden against upstream.
  2. Rebuild AICR per new criteria value. Rejected — defeats the single-binary model; every NCP/customer/product addition becomes a release cycle.
  3. Make criteria fully open strings (no validation at all). Rejected — loses typo safety (--service eka would silently match no overlays). Catalog-driven validation preserves the safety while making the value set extensible.
  4. Per-cluster overlay files. Rejected — configuration-explosion antipattern. Per-cluster specifics belong in AICRConfig (--config), not in overlays.

Compatibility / Breaking Changes

Backward compatible for OSS users:

  • All existing criteria values continue to validate (canonical/aliased switch arms remain as the fast path).
  • GetCriteria*Types() callers see the same minimum set; the set only grows depending on the loaded catalog.
  • Default behavior (no --data, no --criteria-strict) is unchanged.

The three changes are tightly coupled and should land together:

  • Registry without reordering → registry never gets populated for --data overlays. No observable change.
  • Reordering without the registry → ParseCriteria*Type() still rejects unknown values. No observable change.
  • Strict mode without the others → flag has nothing to gate against.

Single PR resolves the whole scope.

Operational Considerations

  • Observability. Registry contents loggable at DEBUG. Consider exposing via aicr query for inspection (out of scope for this PR; track separately if needed).
  • CI. OSS make qualify should run with --criteria-strict (or an equivalent assertion) so the upstream catalog can't drift onto internal-only values.
  • Documentation. docs/integrator/recipe-development.md and docs/contributor/data.md need a section on the criteria-extension contract — how --data overlays contribute values, how strict mode interacts, what the precedence rules are between embedded and external catalogs.
  • Multi-tenancy. No change — --data is per-invocation; multiple users with different --data paths see different catalogs, which is the intended composition model.
  • Security. External overlay loading already enforces filepath.IsLocal and a max-file-size guard in provider.go; the registry inherits those. No new attack surface introduced by the validator change.

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions