Summary

GPU CI reliability is constrained by three categories of issues:

1. Build/deploy performance: H100 x2 workflows are severely bottlenecked by slow Docker/registry I/O on the linux-amd64-gpu-h100-latest-2 runner pool. Build aicr takes up to 40 min, and the kai-scheduler crd-upgrader hook frequently exceeds Helm timeout due to slow in-cluster image pulls. PR fix(ci): improve GPU test reliability and deploy timeout handling #539 mitigates with selective validator builds, per-component Helm timeouts, and retry logic. The structural fix is migrating from kind load to a local registry.

2. Workflow duplication: Training and Conformance workflows are mostly redundant, doubling H100 x2 runner consumption. Training and Inference trigger on every PR regardless of intent.

3. Behavioral test fragility: Dynamo inference tests are slow (multi-GB image pulls, model loading) and fragile (hardcoded sleep-poll loops, no backoff). Chainsaw health checks can fail on timing-dependent assertions (e.g., grafana availability window).

Problems

1. H100 x2 Build and Deploy Time Budget

H100 x2 workflows consistently exhaust their time budget due to slow runner I/O. Findings from April 11, 2026 debugging:

kai-scheduler crd-upgrader hook is the single biggest deploy bottleneck:

• The pre-install hook Job runs inside the kind cluster and must pull its container image from ghcr.io
• On H100 x2 runners, this frequently exceeds 10 min, likely due to slow cold image pull and pod startup on these runners
• At the original 10m Helm timeout, this consistently timed out on the first attempt
• Retries then succeed — likely because the image or its layers are partially or fully cached on the node from the first attempt
• The hook image reference is hardcoded in the chart template (filed upstream: Request nvcr.io mirror for kai-scheduler images to reduce Helm hook pull timeouts on cold CI runners kai-scheduler/KAI-Scheduler#1404)

Downstream failures observed (appear infrastructure/runner-dependent, not PR regressions):

• Training: grafana deployment missed chainsaw health-check window (assert-monitoring ERROR on apps/v1/Deployment @ monitoring/grafana)
• Conformance: gang-scheduling validator failed, then job cancelled at 90 min during ai-service-metrics

2. Conformance workflow is a duplicate of Training

Training and Conformance are mostly redundant — same runner (H100x2), same recipe (--intent training), same aicr validate --phase conformance, same chainsaw test dir (kind-training), same 5 evidence assertion files. The intentional difference is step ordering (training runs chainsaw before validate; conformance runs validate before chainsaw), which can expose different timing-dependent failures. However, running both on the same PR doubles H100x2 runner consumption for marginal additional signal.

3. Training and Inference are mutually exclusive but both always run

A Kind cluster is configured with either --intent training or --platform dynamo (inference), never both. Most PRs only affect one intent, yet both workflows trigger, doubling H100 runner consumption for no additional signal.

4. Runner contention from parallel H100x2 workflows

Training and Conformance both require linux-amd64-gpu-h100-latest-2 (2× H100 GPUs). A single PR triggers 2 H100x2 jobs; two concurrent PRs trigger 4, all competing for what appears to be a single-runner pool. Jobs fail at "Set up runner" or get cancelled waiting.

5. Dynamo inference test is slow and fragile

The inference test deploys a full Dynamo stack and runs live inference:

• Pulls multi-GB nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0 into Kind
• Loads Qwen/Qwen3-0.6B model into GPU memory
• Polls DGD readiness: 60 × 10s = 10min max, no backoff
• Polls model registration: 30 × 10s = 5min max, no backoff
• Port-forward with fixed sleep 3 before first curl

Any of these can fail due to slow image pull, OOM, operator race, or port-forward delay.

6. Hardcoded sleep-poll loops

All retry loops use fixed intervals with no exponential backoff. Device plugin DaemonSet wait: 30 × 10s = 5min max.

7. Existing path filters miss actual workflow inputs

All GPU workflows (training, inference, smoke) use composite actions (setup-build-tools, install-karpenter-kwok, load-versions) and .settings.yaml that are not consistently represented in their path filters. This means changes to tool versions or shared actions can silently skip the wrong jobs. Any workflow consolidation or trigger change must include a path filter audit for the affected workflows.

Known gaps:

• .github/actions/setup-build-tools/** — used by inference but not in its path filter
• .github/actions/install-karpenter-kwok/** — used by both but missing from some filters
• .github/actions/load-versions/** — used by both, inconsistently filtered
• .settings.yaml — drives tool versions, not in all filters

8. Debug diagnostics duplicated as inline scripts

All 3 H100 workflows have largely identical inline debug diagnostics (GPU operator pods, non-running pods, node status, custom metrics API). Not reusable.

9. kind load bottleneck for image distribution

The current GPU workflow design uses kind load docker-image to push images into kind nodes via tar pipe through Docker. This is inherently slow on runners with limited disk I/O and scales poorly with node count. The e2e action already uses a local registry pattern that avoids this bottleneck entirely.

Fixes Implemented (PR #539)

The following mitigations are implemented in PR #539:

Proposal

1. Image Build and Deployment Optimization

• Migrate from kind load to local registry (structural fix, highest impact)
  • Replace kind load docker-image with the local registry pattern already used in .github/actions/e2e/action.yml
  • Set up localhost:5001 registry in .github/actions/gpu-cluster-setup/action.yml
  • aicr-build pushes images to localhost:5001 instead of kind load
  • Snapshot agent uses --image=localhost:5001/aicr:local, validators use AICR_VALIDATOR_IMAGE_REGISTRY=localhost:5001
  • Nodes pull on demand — no tar-pipe bottleneck, no topology assumptions, no special cases for 1-node vs 2-node clusters
  • The validator path already supports localhost registries (pkg/validator/job/deployer.go:255)
  • Use a run-specific tag (e.g., :local) instead of :latest — in pkg/validator/job/deployer.go:255, :latest maps to PullAlways which defeats caching; :local gives IfNotPresent
• Reduce build-side waste
  • Stop building the aicr binary twice in aicr-build (once for smoke-test image, once standalone)
  • Reduce Docker build context for smoke-test and validator images (currently sends full repo)
• Upstream: kai-scheduler hook image configurability
  • Filed: Request nvcr.io mirror for kai-scheduler images to reduce Helm hook pull timeouts on cold CI runners kai-scheduler/KAI-Scheduler#1404 — requesting the crd-upgrader hook image registry/tag be configurable in values.yaml so consumers can pre-pull or mirror it
  • Until resolved, the 20m per-component timeout (PR fix(ci): improve GPU test reliability and deploy timeout handling #539) is the workaround

2. Combine Conformance into Training and Inference Workflows, Make Triggers Intent-Aware

• Deduplicate conformance (immediate runner savings)

  • Delete the standalone conformance workflow — Training and Inference already run aicr validate --phase conformance
  • Consolidating them loses no meaningful validation coverage. The only thing removed is incidental timing variation from running the same checks in a different order (the current conformance workflow runs validate→chainsaw while training runs chainsaw→validate — this ordering difference is accidental, not an intentional testing strategy).
  • Fold conformance's broader path filter into training to avoid coverage gaps (includes tests/chainsaw/ai-conformance/cluster/**, docs/conformance/cncf/**)
  • This change must include a path filter/dependency audit for the conformance workflow being folded — verify all composite actions and file dependencies are represented in the training workflow's paths: filter before deleting the standalone workflow

  Current (3 clusters)	Runner	What it does
  Training	H100x2	Bundle deploy → Chainsaw → Conformance → Evidence
  Conformance	H100x2	Bundle deploy → Conformance → Chainsaw → Evidence (mostly redundant)
  Inference	H100x1	Bundle deploy → Conformance → Chainsaw → Dynamo → Inference test → Evidence

  Proposed (2 clusters)	Runner	What it does
  Training + Conformance	H100x2	Bundle deploy → Chainsaw → Conformance → Evidence
  Inference + Conformance	H100x1	Bundle deploy → Chainsaw → Conformance → Dynamo → Inference test → Evidence

  Canonical step order: Chainsaw health checks → conformance validation → intent-specific checks (Dynamo, inference) → evidence collection. This order ensures: (1) failures are easier to interpret — deployment health is verified before deeper checks; (2) no time is wasted on conformance or intent-specific tests when the deployment is obviously unhealthy; (3) metrics- and operator-dependent components get warm-up time before conformance checks.

• Audit and fix path filters (prerequisite for intent-aware triggers)

  • For each workflow, verify all composite actions it uses: have corresponding paths: filter entries
  • Add .settings.yaml to all workflow filters (drives tool versions)
  • Add missing action paths (setup-build-tools, install-karpenter-kwok, load-versions)

• Make triggers intent-aware

  • PR touches training-specific paths → run Training only
  • PR touches inference-specific paths (Dynamo, kgateway, inference overlays) → run Inference only
  • PR touches shared infra (validators, CI actions, recipes/registry.yaml, collectors) → run both
  • Split run-gpu-tests label into run-gpu-training-test and run-gpu-inference-test

Additional Improvements

• Reduce retry timeouts — Tune based on measured startup times (p95 + margin), not fixed targets. Validate against real data before cutting.
• Replace sleep-poll with kubectl wait — Replace hardcoded sleep-poll loops in inference test with kubectl wait --for=condition=.... Add exponential backoff to curl retries.
• GPU-specific chainsaw config — Create tests/chainsaw/chainsaw-gpu-config.yaml with failFast: false for GPU CI health-check steps, instead of modifying the shared config.
• Extract debug diagnostics — Move duplicated inline debug scripts to .github/actions/gpu-debug-diagnostics/action.yml.

Sequencing

1. Deduplicate conformance (includes path filter audit for the folded workflow) — immediate runner savings
2. Image build and deploy optimizations — highest performance impact
3. Audit remaining path filters + intent-aware triggers — land together
4. Additional improvements — independent, can land in any order

Related

• PR fix(ci): improve GPU test reliability and deploy timeout handling #539: deploy.sh fixes, selective validators, timeout bumps, per-component Helm timeout
• Request nvcr.io mirror for kai-scheduler images to reduce Helm hook pull timeouts on cold CI runners kai-scheduler/KAI-Scheduler#1404: hook image configurability request

Future Work (out of scope)

These are valuable but deferred until the workflow shape stabilizes:

• Move inference to nightly + on-demand: For PRs that don't touch inference paths, run inference on nightly schedule + run-gpu-inference-test label only. Part of a broader tiering strategy.
• Tiered testing: Structural-only PR gate (~15-20min) with change-aware behavioral coverage
• --conformance-mode=fast|thorough: Requires plumbing through job deployer, runner, context, catalog
• Parallel + serial check execution: Requires isolation analysis of shared GPU state across checks
• Increase H100x2 runner pool: Request from infra team (nv-gpu-amd64-h100-2gpu runner group)
• Investigate H100 x2 runner pool I/O degradation: The 10-24x slowdown vs H100 x1 suggests infrastructure issues beyond node count