Skip to content

Commit f68b443

Browse files
fjvbn2003claude
andcommitted
fix(validator): say why host driver validation failed
When the host has nvidia-smi but it cannot talk to the driver (for example after the driver packages were upgraded without a reboot, so the user-space libraries and the loaded kernel module differ), the driver-validation init container logs: No pre-installed driver detected on the host: exit status 18 ... failed to validate the driver, retrying after 5 seconds\n and then retries the driver container path forever. Both lines point away from the real problem: a driver is installed on the host, and the retry warning never says what failed (it also prints a literal "\n"). - validateHostDriver: when nvidia-smi was found on the host but failed to run, return an error that says so and names the binary. - runValidation: log "No usable pre-installed driver detected" instead of "No pre-installed driver detected". - validateDriverContainer: include the error in the retry warning and drop the trailing "\n". No change in control flow. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: YoungjuKim <[email protected]>
1 parent dbb11e8 commit f68b443

1 file changed

Lines changed: 9 additions & 3 deletions

File tree

‎cmd/nvidia-validator/main.go‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -745,7 +745,13 @@ func validateHostDriver(silent bool) error {
745745
command := "chroot"
746746
args := append([]string{"/host", nvidiaSMIPath}, nvidiaSMIArgs()...)
747747

748-
return runCommand(command, args, silent)
748+
if err := runCommand(command, args, silent); err != nil {
749+
// nvidia-smi exists on the host, so a driver is most likely installed but
750+
// not usable, e.g. after a driver upgrade without a reboot. Say so, rather
751+
// than reporting only the exit status.
752+
return fmt.Errorf("found '%s' on the host, but running it failed: %w", nvidiaSMIPath, err)
753+
}
754+
return nil
749755
}
750756

751757
// With driver-validation-skip-gpu-init, validate with 'nvidia-smi --version',
@@ -794,7 +800,7 @@ func validateDriverContainer(silent bool, driverManagedByOperator bool) error {
794800
if !withWaitFlag {
795801
return fmt.Errorf("error validating driver: %w", err)
796802
}
797-
log.Warningf("failed to validate the driver, retrying after %d seconds\n", sleepIntervalSecondsFlag)
803+
log.Warningf("failed to validate the driver, retrying after %d seconds: %v", sleepIntervalSecondsFlag, err)
798804
time.Sleep(time.Duration(sleepIntervalSecondsFlag) * time.Second)
799805
continue
800806
}
@@ -809,7 +815,7 @@ func (d *Driver) runValidation(silent bool) (driverInfo, error) {
809815
return getDriverInfo(true, hostRootFlag, hostRootFlag, "/host"), nil
810816
}
811817

812-
log.Infof("No pre-installed driver detected on the host: %v", err)
818+
log.Infof("No usable pre-installed driver detected on the host: %v", err)
813819

814820
log.Info("Validating containerized driver installation")
815821

0 commit comments

Comments
 (0)