Skip to content

Add integration test: nvidiaDriverRoot and gpu-operator.hostPaths.driverInstallDir must stay in lockstep across overlays #1087

Description

@yuanchen8911

Summary

nvidia-dra-driver-gpu.nvidiaDriverRoot and gpu-operator.hostPaths.driverInstallDir MUST point to the same path. They're independently configurable across overlays and silently break if they drift. There's no schema link between the two, so an overlay editor can change one without the other and CI won't catch it.

Why

The DRA kubelet plugin loads the NVIDIA driver userspace (libnvidia-ml.so, nvidia-smi, nvidia-ctk) from nvidiaDriverRoot. GPU operator's hostPaths.driverInstallDir is where the operator-managed driver container mounts its rootfs onto the host. If these diverge:

  • DRA fails CDI spec generation (Driver/library version mismatch or missing libnvidia-ml.so)
  • DRA-allocated pods stay in ContainerCreating
  • aicr validate deployment phase fails

This is the symptom that surfaced on the krusty cluster during BCM service validation. The root cause was a different host-vs-container driver question, but the same class of bug (the two paths assuming different driver locations) is the underlying risk.

Acceptance

  • Test enumerates every overlay × intent × platform combination (use the recipe builder to fully hydrate) and asserts the two resolved values match
  • Test runs in make qualify / CI
  • Test emits a clear diff message identifying the overlay file when they don't match

Implementation notes

Likely lives next to existing overlay-resolution tests in pkg/recipe/.... Iterate AllCriteriaAcceleratorTypes, AllCriteriaServiceTypes, AllCriteriaIntentTypes, etc., resolve each combination, and assert. Skip combinations the registry rejects.

Related: #1086 (BCM overlay gaps), where this lockstep would have prevented an earlier debug iteration.

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions