🌿 Preview your docs: https://nvidia-preview-docs-adr-006-deployment-validation.docs.buildwithfern.com/aicr

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

I made this comment on the last PR, including it here.
#630 (comment)

This design has a lot of overlap with #610 (#610 has 2 halves, only referring to deployment half). feels like we might want to share/coordinate, if does not add a lot of complexity to share the config. Would hate to have to maintain this in 2 places.

This design has a lot of overlap with #610 (#610 has 2 halves, only referring to deployment half). feels like we might want to share/coordinate, if does not add a lot of complexity to share the config. Would hate to have to maintain this in 2 places.

Yes, there is real overlap. My take is

• 610 deployment half is about deploy-time flow control:
  • post-install hook Jobs / sync waves
  • deployer-specific rendering
  • images, RBAC, timeouts, shell logic
• 622 (ADR-006) is about post-install validation in aicr validate --phase deployment:
  • deployer-neutral
  • intended to stay usable as a standalone validator

So the overlap is mainly in the readiness intent for each component, not in the full runtime/config model. I agree we should coordinate so we do not end up maintaining two unrelated readiness stories per component.

#630 (comment)

Good point, agree "Chainsaw can’t compile cleanly with our Go toolchain” objection is weaker now and library-backed reuse is more viable than before, but direct reuse of our current checks still implies either binary support or rewriting those assets into a narrower form, IMO.

@lockwobr Updated the ADR:

• The ADR now explicitly links to Bridge Jobs for deploy/undeploy lifecycle flow control #610 and acknowledges the real overlap around per-component readiness intent.
• It draws the boundary more clearly:
  • Bridge Jobs for deploy/undeploy lifecycle flow control #610 is deploy-time flow control / undeploy cleanup
  • ADR-006 is post-install validation in aicr validate --phase deployment
• The intended coordination model is:
  • align semantics and component mappings where that is simple
  • avoid two unrelated readiness stories
  • but do not force a single shared config model if that would pull deploy-time hook/job details into the validator contract

Circling back with a synthesis that ties this ADR to #610. I think the two designs aren't competing — they're solving adjacent problems that share one primitive: "is this component ready?"

The shared primitive: aicr validate with a blocking wait mode.

If aicr validate --component <name> --phase deployment --wait --timeout 30m blocks until readiness checks pass (or the timeout fires), and we ship an aicr container image with the CLI inside, then both designs consume the same implementation:

• ADR-006 / Add registry-driven deployment readiness for all components #622 — operators run aicr validate --phase deployment directly (local CLI, day-N re-check, CI gates)
• Bridge Jobs for deploy/undeploy lifecycle flow control #610 bridge Jobs — the deploy-time hook Job uses the aicr image and calls the same aicr validate ... --wait internally

# #610-style bridge Job under this synthesis
containers:
  - name: wait
    image: ghcr.io/nvidia/aicr:v1.2.3
    command: [aicr, validate, --component, gpu-operator, --phase, deployment, --wait, --timeout, 30m]

Key property: the two designs stay in sync by default, with an escape hatch when needed. Since deploy-time readiness (bridge Job) and validate-time readiness (aicr validate) both dispatch through the same CLI codepath, there's no way for them to diverge accidentally — a change to how "ready" is computed automatically applies to both sites. No documentation discipline required, no drift between parallel implementations, no "did we update both?" risk. And when a component genuinely needs logic the shared path can't express, the readiness.yaml override (from #610's convention) provides a clean exit without abandoning the shared model for everyone else.

--wait --timeout is a general validator capability, not just a deployment-phase primitive. Plenty of checks benefit from "wait until this converges" semantics: post-snapshot agents reporting in, taint/label operations applying across nodes, health checks reaching a stable state, cluster-wide constraints that take time to propagate. Others are genuinely one-shot (static version checks, static capacity thresholds) and would just resolve immediately. The validator can expose --wait uniformly — checks that are wait-capable honor it, one-shot checks return immediately without penalty. Designing this as a general capability (not a bridge-Job-specific hack) means future validation work inherits it for free.

Precedence model for the two config shapes:

• Structured readiness fields in ComponentRef (this ADR's contribution) cover the common cases declaratively — the default for most components. Both deploy-time and validate-time consume them identically.
• readiness.yaml shipped in recipes/components/<name>/ (Bridge Jobs for deploy/undeploy lifecycle flow control #610's convention) acts as an override when a component needs logic that can't be expressed structurally.

File presence wins. Components stay on the structured path unless they explicitly need the escape hatch.

Addresses @ayuskauskas's Chainsaw point indirectly: how aicr validate internally evaluates readiness (pure Go, shells out to Chainsaw, something else entirely) becomes an implementation detail behind a stable CLI interface. Whatever the validator chooses under the hood, consumers see the same aicr validate entrypoint. That lets us pick the simplest thing now and revisit the executor later without breaking #610's bridge Jobs or any external tooling.

Tradeoffs worth naming early:

1. aicr validate --wait --timeout is meaningful new surface (polling, backoff, SIGTERM handling for clean hook termination). Not hard, but not free.
2. The aicr container image becomes another cluster-side artifact — signing, SBOM, CVE scanning. Precedent already exists (snapshot-capture), so no new threshold crossed.
3. Air-gap: if the aicr image isn't pullable, bridge Jobs fail. The readiness.yaml override path lets users ship a raw kubectl wait Job as a fallback — supported escape hatch.
4. ADR-006's "Fixed special-case surface" framing should explicitly leave room for readiness.yaml to override the structured fields, so we don't accidentally close off the override path.

Coordination point: the boundary between #610 (deploy-time flow control) and this ADR (post-install validation) stays as you've drawn it. The synthesis is just that both delegate to the same underlying readiness implementation rather than re-implementing the check logic in two places.

Happy to fold any of this into #610 or open a third issue for aicr validate --wait specifically if that scope feels like it deserves its own tracking.

@lockwobr I updated the ADR based on your synthesis. It now makes the shared validator entrypoint explicit, adds the --wait --timeout convergence idea, and leaves room for a readiness.yaml escape hatch without overcommitting the exact precedence rules yet. PTAL.

@ayuskauskas thanks, this is helpful. A few reactions point by point:

• On the DSL concern: agreed. I do not want to introduce a general-purpose DSL plus arbitrary parser/executor here. The intent is a small typed schema over a few common Kubernetes check primitives, not a new assertion language.
• On distroless: agreed that shipping Chainsaw does not inherently mean the image cannot stay distroless. My concern is less about distroless specifically and more about the broader runtime, packaging, signing, SBOM/CVE, upgrade, and testing cost of making Chainsaw part of the validator path.
• On reducing deployment checks to selector + standard Ready condition: I think that is too narrow for cases we already know we need, such as exact named workloads, tightened DaemonSet semantics, declarative CRD establishment, and a small set of component-specific CR/status checks.
• On customChecks as inline Chainsaw assert or file path: I think that is a reasonable escape hatch when the structured schema is not enough.
• Where I still differ is on dropping the third type. I think CRD establishment should stay first-class rather than being folded into generic customChecks, because it is a common declarative signal, not just a bespoke edge case.

So my current preference is to keep the structured contract small (readiness, crds, customChecks), avoid turning it into a general DSL, and leave room for a Chainsaw- or asset-based override path when the typed schema is not sufficient.

how much do use chainsaw now, by not using it we are also then having to go patch all the places it used today. I think we are using it a "good bit" we should consider it before just removing it from the table. @xdu31 would removing chainsaw be a lot of work? Additionally, moving to using it as a lib, how much work is that?

closed it. It looks like Chainsaw is the right direction.

The path to enhance aicr validate --phase deployment using Chainsaw is straightforward. We already have per-component Chainsaw health checks. The gap is wiring them into the deployment validation:  triggering the existing Chainsaw assets from the validator and enhancing/extending individual Chainsaw component checks as needed.