Summary
nvidia-dra-driver-gpu.nvidiaDriverRoot and gpu-operator.hostPaths.driverInstallDir MUST point to the same path. They're independently configurable across overlays and silently break if they drift. There's no schema link between the two, so an overlay editor can change one without the other and CI won't catch it.
Why
The DRA kubelet plugin loads the NVIDIA driver userspace (libnvidia-ml.so, nvidia-smi, nvidia-ctk) from nvidiaDriverRoot. GPU operator's hostPaths.driverInstallDir is where the operator-managed driver container mounts its rootfs onto the host. If these diverge:
- DRA fails CDI spec generation (
Driver/library version mismatch or missing libnvidia-ml.so)
- DRA-allocated pods stay in
ContainerCreating
aicr validate deployment phase fails
This is the symptom that surfaced on the krusty cluster during BCM service validation. The root cause was a different host-vs-container driver question, but the same class of bug (the two paths assuming different driver locations) is the underlying risk.
Acceptance
- Test enumerates every overlay × intent × platform combination (use the recipe builder to fully hydrate) and asserts the two resolved values match
- Test runs in
make qualify / CI
- Test emits a clear diff message identifying the overlay file when they don't match
Implementation notes
Likely lives next to existing overlay-resolution tests in pkg/recipe/.... Iterate AllCriteriaAcceleratorTypes, AllCriteriaServiceTypes, AllCriteriaIntentTypes, etc., resolve each combination, and assert. Skip combinations the registry rejects.
Related: #1086 (BCM overlay gaps), where this lockstep would have prevented an earlier debug iteration.
Summary
nvidia-dra-driver-gpu.nvidiaDriverRootandgpu-operator.hostPaths.driverInstallDirMUST point to the same path. They're independently configurable across overlays and silently break if they drift. There's no schema link between the two, so an overlay editor can change one without the other and CI won't catch it.Why
The DRA kubelet plugin loads the NVIDIA driver userspace (
libnvidia-ml.so,nvidia-smi,nvidia-ctk) fromnvidiaDriverRoot. GPU operator'shostPaths.driverInstallDiris where the operator-managed driver container mounts its rootfs onto the host. If these diverge:Driver/library version mismatchor missinglibnvidia-ml.so)ContainerCreatingaicr validatedeployment phase failsThis is the symptom that surfaced on the krusty cluster during BCM service validation. The root cause was a different host-vs-container driver question, but the same class of bug (the two paths assuming different driver locations) is the underlying risk.
Acceptance
make qualify/ CIImplementation notes
Likely lives next to existing overlay-resolution tests in
pkg/recipe/.... IterateAllCriteriaAcceleratorTypes,AllCriteriaServiceTypes,AllCriteriaIntentTypes, etc., resolve each combination, and assert. Skip combinations the registry rejects.Related: #1086 (BCM overlay gaps), where this lockstep would have prevented an earlier debug iteration.