Summary

Follow-up note: this issue is the broader all-components follow-up to #607 and its narrow implementation PR #611. It does not widen or reopen that GPU-scoped work.

Follow up to issue #607 and PR #611 with a registry-driven readiness contract so aicr validate --phase deployment becomes deeper, more symmetric, and deployer-neutral for all current registry components, not just baseline namespace / expectedResources checks plus the targeted GPU-specific checks.

This follow-up is the broader next step: make deployment-phase readiness declarative in the registry and apply it consistently across all components.

This issue has two explicit deliverables:

1. Registry-driven deep deployment checks: for every component, deployment validation should run the deep checks declared in recipes/registry.yaml:
   • generic workload readiness
   • CRD readiness where applicable
   • fixed custom checks where generic readiness is not sufficient
2. Registry enforcement for new components: after the current migration lands, adding a new registry component must require declaring at least one readiness coverage path:
   • readiness
   • customChecks
   • top-level crds

Important framing: this issue is not trying to reproduce the full expressiveness of the existing repo-side Chainsaw health checks. The v1 contract is intentionally narrower and more operationally stable.

Problem / Use Case

After #611, deployment validation is stronger but still asymmetric:

• baseline checks run for enabled components (Active namespaces, healthy declared expectedResources)
• targeted deep checks run for the three GPU-specific gaps (skyhook-customizations, nvidia-dra-driver-gpu, gpu-operator)
• many other components still have only shallow validation unless they declare expectedResources

That means:

• new registry components do not automatically get deep deployment validation
• the validator surface is still partly code-driven instead of registry-driven
• some shared-namespace components still cannot be mapped cleanly without an exact-workload escape hatch
• deployer-neutral deep readiness is not yet symmetric across the registry

The goal of this follow-up is to make the deployment validator:

1. deeper and more symmetric across all registry components
2. deployer-neutral (no Helm API, no release metadata, no release-scoped labels)
3. simple, manageable, and extensible when new components are added

Implementation Options

Option 1. Add explicit readiness fields and implement deployment validation in pure Go

Add a registry-driven readiness contract and have the deployment validator interpret it using Kubernetes API state.

• Shape:
  • readiness with required namespace
  • optional selector for discovery / scoping in shared namespaces
  • optional workloads for exact Deployment / DaemonSet / StatefulSet identities when selector-only matching is not precise enough
  • top-level crds for CRD-only components
  • sibling customChecks
• Runtime transport:
  • readiness / customChecks / crds must be hydrated from the registry onto each resolved ComponentRef in RecipeResult during recipe resolution
  • do not introduce a parallel top-level readiness map or a validator-only side channel under validation
  • validator Jobs must rely on mounted recipe.yaml, not read registry.yaml directly at runtime
  • this preserves layered / external data-dir behavior and avoids validator-side drift from the resolved recipe
• Complexity:
  • requires schema additions and validator changes
  • keeps the deployment validator self-contained and dependency-light
  • straightforward to unit test with fake clients
• Architecture alignment:
  • keeps deployment validation on the same Kubernetes-native, pure-Go model already used by aicr validate --phase conformance
  • avoids introducing a second validator runtime inside aicr validate
  • makes the validator easier to extend toward broader CNCF AI Conformance coverage without taking on Chainsaw as a runtime dependency
• Completeness / function:
  • covers the common readiness cases generically (Deployment, DaemonSet, StatefulSet, CRDs)
  • handles special cases through narrow customChecks
  • does not try to reproduce every Chainsaw assertion feature in deployment validation
• Overlap and consistency:
  • some semantic overlap with existing Chainsaw health checks can remain
  • higher consistency risk because deployment readiness and repo-side Chainsaw checks are maintained separately
  • acceptable if the deployment contract is intentionally narrower than full Chainsaw expressiveness

Option 2. Reuse existing Chainsaw health checks in deployment validation

AICR already has Chainsaw health check files under recipes/checks/*, and healthCheck.assertFile already exists in recipes/registry.yaml. On main, the deployment validator also has a dormant Chainsaw execution path that would run if ComponentRef.HealthCheckAsserts were populated.

However, that path is not currently triggered in normal validation:

• healthCheck.assertFile is not hydrated into ComponentRef.HealthCheckAsserts
• the normal recipe and registry loading flow therefore does not activate Chainsaw during aicr validate --phase deployment
• validator Jobs currently consume mounted recipe.yaml, not registry.yaml, so this option would still need an explicit transport step just like Option 1:
  • either hydrate assert content into the resolved recipe
  • or add a validator-side registry mount / lookup path
• the current deployment validator image is distroless and does not include the chainsaw binary, and pkg/recipe/metadata.go intentionally avoids loading healthCheck.assertFile for exactly that reason

To adopt this option, the implementation would need to wire healthCheck.assertFile into ComponentRef.HealthCheckAsserts and keep or restore the deployment-phase Chainsaw runner.

• Complexity:
  • less schema work because the repo already has healthCheck.assertFile for many current components
  • but not all current registry components are covered today
  • gke-nccl-tcpxo, dynamo-crds, and kgateway-crds still lack healthCheck.assertFile, so this option would still need new health checks or a side mechanism for those components
  • more runtime, build, and pipeline complexity because deployment validation would depend on Chainsaw execution again
  • requires image / packaging changes to ship the chainsaw binary in the deployment validator path
  • harder to unit test than typed Go readiness checks
  • introduces a second validator runtime inside aicr validate: pure Go / Kubernetes-client checks for conformance and Chainsaw-driven checks for deployment
  • adds ongoing Chainsaw binary packaging, version pinning, and image maintenance burden
• Completeness / function:
  • most expressive option
  • best reuse of existing recipes/checks/* logic
  • strongest parity with the current repo-side health checks for components that already have healthCheck.assertFile
• Overlap and consistency:
  • lowest risk of semantic drift between deployment validation and repo-side Chainsaw checks
  • strongest consistency story if exact parity with existing Chainsaw health checks is the main goal
• Tradeoffs:
  • reintroduces a runtime dependency we have explicitly tried to avoid in deployment validation
  • makes the deployment validator heavier and less self-contained
  • works against the goal of a simple, manageable, pure-Go deployment readiness path
  • couples runtime behavior more tightly to upstream Chainsaw semantics and upgrade cadence
  • expands the maintenance and review surface because full Chainsaw Test execution brings scripts, waits, catches, and binary execution into runtime validation

Recommendation

Recommend Option 1: explicit readiness fields plus pure Go deployment validation.

Why:

• best fit for the project criteria: deeper, more symmetric, deployer-neutral, and simple / manageable / extensible
• best fit for the actual deliverable: a typed, enforceable component contract in recipes/registry.yaml
• matches the current validator runtime model: validator Jobs consume the resolved recipe.yaml, so registry-derived readiness data can be hydrated once during recipe resolution and then executed consistently in-cluster
• aligns with the existing aicr validate architecture, where --phase conformance already uses Kubernetes-native pure-Go checks rather than Chainsaw
• easier to test and reason about than a deployment-phase Chainsaw runner
• easier to unit test and enforce with fake Kubernetes clients than a Chainsaw-execution path
• keeps aicr validate --phase deployment free of extra runtime dependencies
• keeps the deployment phase lightweight and self-contained instead of reintroducing the chainsaw binary as a runtime dependency
• keeps aicr validate on one validator model instead of splitting it across Go checks for conformance and Chainsaw for deployment
• makes the validator easier to extend toward broader CNCF AI Conformance validation without adding a second runtime model or external binary dependency
• keeps the validator Kubernetes-native end to end
• still leaves room to preserve Chainsaw as a separate repo-side health-check workflow

The cost of this choice is that some readiness semantics may overlap with existing Chainsaw checks. That duplication is acceptable because the deployment-phase contract is intentionally narrower and more operationally stable than full Chainsaw expressiveness.

Option 2 remains a viable alternative if the project later decides that exact parity with repo-side health checks is more important than keeping deployment validation self-contained and dependency-light. For this issue as scoped, however, it is not the preferred path because it would still require:

• transport of healthCheck.assertFile content into validator Jobs
• deployment validator image / packaging changes to ship the chainsaw binary
• additional coverage work for current registry components that still do not have healthCheck.assertFile

Proposed Solution

Introduce a registry-driven readiness contract and have the deployment validator interpret it generically.

Concretely, this proposal has two parts:

1. make deployment-phase deep checks come from the component contract in recipes/registry.yaml
2. later enforce that every newly registered component declares one of the supported readiness coverage mechanisms

Locked design

• Add a readiness field to recipes/registry.yaml.
  • namespace is required.
  • selector is optional.
    • selector keys must remain deployer-neutral stable labels already present on rendered objects, for example app.kubernetes.io/name.
    • release-identity labels such as app.kubernetes.io/instance remain out of bounds.
  • workloads is optional.
    • Each item is an exact workload identity: {kind, name}.
    • Supported kinds in v1 are the exact-case enum values Deployment, DaemonSet, and StatefulSet; schema validation must reject other spellings or case variants.
    • workloads items do not carry their own namespace; component-level readiness.namespace is authoritative in v1.
    • Every exact workload named under workloads is required. If any listed workload is absent, the component fails even when selector-discovered workloads exist.
    • workloads extends readiness discovery by exact identity, not by selector post-filtering.
    • When both selector and workloads are present, the effective workload set is the union of selector-discovered workloads and exact named workloads, deduplicated by {kind, namespace, name}.
    • Explicit workload names are appropriate only when those names are deployer-neutral stable render outputs. If a future chart or manifest change makes them variable or release-scoped, the mapping must be updated in the same PR or moved to a customChecks path.
    • Use workloads when selector-only matching would be too broad or too ambiguous.
• Keep crds as a top-level sibling of readiness and customChecks.
  • This issue does not require nesting CRDs under readiness.
  • crds is the exact list of CRD names such as foos.example.com.
• Keep customChecks as a sibling of readiness.
  • customChecks alone counts as valid readiness coverage.
  • skyhook-customizations is the model example: customChecks: [skyhookReady] with no generic readiness entry.
• Hydrate registry readiness / customChecks / crds into RecipeResult during recipe resolution.
  • These fields live on each resolved ComponentRef in RecipeResult.
  • The validator Job must consume only the mounted recipe.yaml.
  • Do not make the validator read embedded registry.yaml at runtime.
  • This is required so external layered data directories behave correctly in deployment validation.
• Overlay merge policy is pinned for the implementation PR.
  • readiness: overlay replaces the base readiness object if set; otherwise the base value remains.
  • customChecks: overlay replaces the base list if set; otherwise the base list remains.
  • crds: overlay replaces the base list if set; otherwise the base list remains.
• Keep the deployment validator pure Go and Kubernetes-client based.
• Generic v1 readiness checks cover:
  • Deployment
  • DaemonSet
  • StatefulSet
  • CustomResourceDefinition with Established=True
• Generic per-kind readiness semantics are intentionally narrower than the existing Chainsaw checks.
  • v1 does not attempt pod-phase parity.
  • v1 is about deployer-neutral workload completeness plus a small set of fixed special checks.
• Generic per-kind semantics are pinned for the implementation PR:
  • Deployment: availableReplicas >= desiredReplicas, where desiredReplicas = spec.replicas if set, otherwise Kubernetes default 1; explicit spec.replicas: 0 is allowed
  • StatefulSet: readyReplicas >= desiredReplicas, where desiredReplicas = spec.replicas if set, otherwise Kubernetes default 1; explicit spec.replicas: 0 is allowed
  • DaemonSet: desiredNumberScheduled > 0 and numberReady == desiredNumberScheduled
    • This intentionally avoids a vacuous generic 0/0 pass for node-gated DaemonSets such as aws-efa.
    • This tightened DaemonSet rule applies to both generic readiness and expectedResources, because both paths use the same typed workload health primitive.
    • As of issue filing, no current expectedResources entry targets a DaemonSet, so no migration of existing expected-resource data is required; the implementation PR should re-verify that claim before merge.
    • If a component truly needs a weaker or different DaemonSet rule, it should use customChecks or a temporary explicit migration exception instead of silently weakening the generic path.
  • CustomResourceDefinition: Established=True
• Zero-match semantics are locked for v1:
  • if readiness is present for a component and the effective workload set resolves to zero matching workload objects, the check fails closed rather than passing vacuously
  • if top-level crds is present and any named CRD is absent or not Established=True, the check fails
  • components that cannot be expressed cleanly with generic readiness must use customChecks or a temporary explicit migration exception
• Defer Job readiness and pod-phase checks (Pending / Failed / Unknown) from v1.
• Reserve customChecks for the current special cases that cannot be expressed generically:
  • gpu-operator
  • skyhook-customizations
  • nvidia-dra-driver-gpu
• Lock the v1 customChecks key set to:
  • clusterPolicyReady
  • skyhookReady
  • draKubeletPluginReady
• Implement customChecks via a fixed Go registration map and fail fast on unknown keys referenced by the registry.
  • Each registered custom check has the uniform callback shape func(ctx *validators.Context, ref recipe.ComponentRef) error.
  • A custom check runs only for components whose resolved ComponentRef explicitly lists that key.
  • Adding a new custom check is intentionally a coordinated Go + registry change in the same PR.
• Relationship with expectedResources:
  • expectedResources continues to coexist in v1 and still runs when present
  • registry-driven readiness is the new standardized component-coverage path
  • temporary overlap and duplicate reporting between expectedResources and registry-driven readiness is acceptable during the migration window
  • the first implementation PR does not need to remove or migrate every existing expectedResources use, but it may opportunistically remove redundant entries where the new generic path fully supersedes them
  • the long-term deprecation, retention, or removal plan for expectedResources is out of scope for this issue and can be decided in a later follow-up once registry coverage is complete and stable
• CRD lists must be explicit in the registry mapping PR.
  • Do not leave placeholders such as "chart-owned CRD names" in the final mapping.
  • For CRDs that are not vendored in-repo, the implementation PR description must cite the pinned chart version and the extraction source / method used to derive the exact list; registry.yaml itself should contain only the resolved CRD names.
  • Any future chart version bump for a component that declares top-level crds must re-verify and update that CRD list in the same PR.
• Define the migration-window exit criterion explicitly.
  • For this issue, the migration inventory is complete when all 21 current registry components listed in the mapping table below have declared one of readiness, customChecks, top-level crds, or a documented temporary migration exception in the implementation PR.
  • Any temporary migration exception is PR-scoped documentation, not a registry schema field: record it in the implementation PR description with the component name, the reason it cannot yet use the locked contract, and the follow-up issue tracking its removal.
  • The follow-up CI guard should be opened immediately after that inventory is complete.
• Add the CI guard that blocks new components without readiness coverage only after the migration is complete for the current registry.
  • the enforcement rule is explicit: a new component registration must declare at least one of readiness, customChecks, or top-level crds
  • temporary migration exceptions are only for the initial migration inventory above; they are not part of the steady-state rule for newly added components
  • the deferred guard must land by extending registry validation in pkg/recipe, surfaced through existing repo CI / lint, not as additional runtime deployment-validator behavior or a separate standalone validator workflow

Deployer-neutrality constraints

The design must stay strictly deployer-neutral:

• no Helm API calls
• no reads of release metadata
• no dependence on release-scoped labels such as app.kubernetes.io/instance
• no reintroduction of Chainsaw as a runtime dependency of the deployment validator

Every readiness decision should come from the Kubernetes API plus the registry-derived contract carried in the resolved recipe.

Component Mapping (current registry)

The first implementation PR should include the actual registry mapping for all current components. Current target mapping:

Component	Proposed coverage
gpu-operator	readiness.namespace: gpu-operator; customChecks: [clusterPolicyReady]
network-operator	readiness.namespace: nvidia-network-operator
gke-nccl-tcpxo	readiness.namespace: kube-system; workloads: [{kind: DaemonSet, name: nccl-tcpxo-installer}, {kind: DaemonSet, name: device-injector}]
aws-efa	readiness.namespace: kube-system; workloads: [{kind: DaemonSet, name: aws-efa-k8s-device-plugin}]
cert-manager	readiness.namespace: cert-manager
skyhook-operator	readiness.namespace: skyhook
skyhook-customizations	customChecks: [skyhookReady]; no generic readiness entry in v1
nvsentinel	readiness.namespace: nvsentinel
nvidia-dra-driver-gpu	readiness.namespace: nvidia-dra-driver; customChecks: [draKubeletPluginReady]
kube-prometheus-stack	readiness.namespace: monitoring; selector: app.kubernetes.io/name=kube-prometheus-stack; workloads: [{kind: Deployment, name: kube-prometheus-operator}]
prometheus-adapter	readiness.namespace: monitoring; selector: app.kubernetes.io/name=prometheus-adapter
aws-ebs-csi-driver	readiness.namespace: kube-system; selector app.kubernetes.io/name=aws-ebs-csi-driver
k8s-ephemeral-storage-metrics	readiness.namespace: monitoring; selector app.kubernetes.io/name=k8s-ephemeral-storage-metrics
kai-scheduler	readiness.namespace: kai-scheduler
dynamo-crds	top-level crds: [...]; the implementation PR must enumerate the exact CRD names from the pinned dynamo-crds chart version and cite the source / extraction method used
dynamo-platform	readiness.namespace: dynamo-system
kgateway-crds	top-level crds must explicitly include the vendored names gatewayclasses.gateway.networking.k8s.io, gateways.gateway.networking.k8s.io, grpcroutes.gateway.networking.k8s.io, httproutes.gateway.networking.k8s.io, referencegrants.gateway.networking.k8s.io, inferencemodelrewrites.inference.networking.x-k8s.io, inferenceobjectives.inference.networking.x-k8s.io, inferencepoolimports.inference.networking.x-k8s.io, inferencepools.inference.networking.k8s.io, and inferencepools.inference.networking.x-k8s.io; the implementation PR should verify both vendored inferencepools CRDs exactly match the in-repo manifests, and if the pinned kgateway-crds chart also owns additional CRDs, the PR must enumerate those too
kgateway	readiness.namespace: kgateway-system
k8s-nim-operator	readiness.namespace: nvidia-nim; temporary overlap with overlay expectedResources is acceptable in v1
kueue	readiness.namespace: kueue-system
kubeflow-trainer	readiness.namespace: kubeflow

The intent is that every current registry component ends up with one of:

• readiness
• customChecks
• an explicit top-level crds declaration
• a temporary explicit migration exception, if and only if the implementation PR documents why the component cannot yet be expressed by the locked design

Acceptance Criteria

• aicr validate --phase deployment remains the existing deployment phase; no new phase or new standalone validator is introduced.
• The registry schema supports:
  • readiness.namespace (required)
  • readiness.selector (optional)
  • readiness.workloads (optional)
  • sibling customChecks
  • top-level crds
• The resolved recipe carries the readiness contract into validator Jobs.
  • readiness / customChecks / crds are hydrated onto each resolved ComponentRef in RecipeResult
  • the implementation does not introduce a parallel top-level readiness map or a validator-only side channel under validation
  • validator Jobs read mounted recipe.yaml
  • validator Jobs do not perform their own registry lookup
• Overlay merge policy is pinned for the implementation PR:
  • readiness: overlay replaces the base readiness object if set; otherwise the base value remains
  • customChecks: overlay replaces the base list if set; otherwise the base list remains
  • crds: overlay replaces the base list if set; otherwise the base list remains
• Workload discovery semantics are pinned for the implementation PR:
  • readiness.workloads.kind accepts only the exact-case enum values Deployment, DaemonSet, and StatefulSet
  • readiness.workloads is resolved by exact Get / identity lookup within readiness.namespace
  • readiness.workloads does not support a per-item namespace override in v1
  • every exact workload named under readiness.workloads is required; a missing named workload is a failure even if selector-based discovery found other workloads
  • when both selector and workloads are present, the effective workload set is their union, deduplicated by {kind, namespace, name}
• The deployment validator interprets the registry contract using only Kubernetes API state.
• Generic v1 readiness checks cover Deployment, DaemonSet, StatefulSet, and CustomResourceDefinition (Established=True).
• Generic per-kind readiness semantics are pinned for the implementation PR:
  • Deployment: availableReplicas >= desiredReplicas, where nil spec.replicas is treated as Kubernetes default 1; explicit 0 is allowed
  • StatefulSet: readyReplicas >= desiredReplicas, where nil spec.replicas is treated as Kubernetes default 1; explicit 0 is allowed
  • DaemonSet: desiredNumberScheduled > 0 and numberReady == desiredNumberScheduled; this tightened rule applies to both generic readiness and expectedResources
  • CustomResourceDefinition: Established=True
• Generic zero-match behavior is pinned for the implementation PR:
  • when a component declares generic readiness and the effective workload set resolves to zero matching workload objects, validation fails rather than passing vacuously
• Job readiness and pod-phase checks remain out of scope for the first implementation.
• customChecks are implemented through a fixed Go registration map and unknown keys fail fast.
  • each registered custom check has the uniform callback shape func(ctx *validators.Context, ref recipe.ComponentRef) error
  • a custom check runs only for components whose resolved ComponentRef explicitly lists that key
• The v1 customChecks key set is exactly:
  • clusterPolicyReady
  • skyhookReady
  • draKubeletPluginReady
• The current GPU-specific deployment checks added by #611 are migrated under customChecks without changing their deployer-neutral semantics.
• expectedResources continues to run where present; temporary overlap and duplicate reporting with registry-driven readiness is acceptable during the migration.
• The first implementation PR includes a concrete mapping for all current registry components listed above.
• The first implementation PR lands explicit CRD name lists for CRD-only components; no placeholder wording remains in the registry mapping.
• The deployment validator stays free of Chainsaw as a runtime and build dependency.
• Documentation explains how to add readiness coverage for a new component, including when to use selector vs exact workloads vs customChecks vs top-level crds.
• The implementation PR includes a live-cluster validation run against h100-eks-ubuntu-inference-dynamo to verify the registry-driven readiness contract end to end.
• The implementation PR documents any mapped components not exercised by that primary live-cluster run, and why.
  • expected examples include environment-specific or hardware-specific components such as aws-efa, network-operator, gke-nccl-tcpxo, or overlays not present on the chosen recipe such as kubeflow-trainer.
• If the implementation PR changes or finalizes the gke-nccl-tcpxo mapping, it should also include a live-cluster validation run against h100-gke-cos-training, or explicitly document why that environment was unavailable.
• The initial migration inventory is complete only when all 21 current registry components listed in the mapping table above have declared one of readiness, customChecks, or top-level crds, or have a documented temporary migration exception in the implementation PR.
• A later guard blocks new registry components from landing without one of: readiness, customChecks, or top-level crds.
  • that deferred guard must land as registry validation in pkg/recipe, surfaced through existing repo CI / lint

Alternatives Considered

1. Keep adding bespoke Go checks per component

Rejected because it does not scale and keeps deep readiness asymmetric across the registry.

2. Reuse Chainsaw health checks directly in deployment

Not recommended for this issue implementation, for the reasons described above under Option 2.

3. Namespace-wide heuristics with no registry contract

Rejected because shared namespaces such as kube-system and monitoring would produce false failures and would not give a clean, maintainable story for new components.

Notes

• #611 is already the narrow GPU-scope step; this issue is the broader follow-up, not a rewrite of #607.
• The CI guard belongs after the migration PR, not in the same initial implementation, so the migration can land cleanly before enforcement begins.

Related

• Narrow GPU-scope issue: #607
• Narrow implementation PR: #611
• Messaging and docs split-out: #609