fix(validator): say why host driver validation failed - #3086

Description

When a node has a pre-installed driver (driver.enabled=false) and nvidia-smi on the host cannot talk to the driver, for example after the driver packages were upgraded without a reboot, the driver-validation init container logs the following. This output is from the reproduction described under Testing, not from a real node: the fake host nvidia-smi exits with 18, which I picked because it is NVML_ERROR_LIB_RM_VERSION_MISMATCH in nvml.h. I later measured a real mismatch: on a host with kernel module 580.95.05, the 610.43.03 nvidia-smi and libnvidia-ml (extracted from the installer, not installed) print Failed to initialize NVML: Driver/library version mismatch and exit 18 in 10 of 10 runs, so the reproduction matches. The validator logs whatever exit status nvidia-smi returns.

Attempting to validate a pre-installed driver on the host
Failed to initialize NVML: Driver/library version mismatch
No pre-installed driver detected on the host: exit status 18
Validating containerized driver installation
Attempting to validate a driver container installation
failed to validate the driver, retrying after 5 seconds\n
failed to validate the driver, retrying after 5 seconds\n
...

and stays in Init forever. Both log lines point away from the problem. A driver is installed on the host, and the retry warning never says what failed (it also prints a literal \n). #1276 and #1687 show users stuck on the same retry line without a reason.

This PR:

• validateHostDriver: when nvidia-smi was found on the host but failed, return found '<path>' on the host, but running it failed: <err>.
• runValidation: log No usable pre-installed driver detected on the host: ... instead of No pre-installed driver detected ....
• validateDriverContainer: include the error in the retry warning and drop the trailing \n.

Control flow does not change. Validation still falls through to the driver container path, which is still needed when the host has a stray nvidia-smi but the driver comes from the driver container.

This overlaps with #3034 on the retry warning line. If #3034 merges first, I will rebase and move the error into the warning in waitForDriver.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint) (golangci-lint not available locally; gofmt -s and go vet with GOOS=linux are clean)
• Generated assets in-sync (make validate-generated-assets) (no generated files changed)
• Go mod artifacts in-sync (make validate-modules) (no module changes)
• Test cases are added for new code paths (log/error text only; validateHostDriver chroots into /host, so it is not unit-testable without a refactor)

Testing

• go test ./cmd/nvidia-validator/ passes.
• On a cluster running GPU Operator v26.3.3, I ran the validator on a GPU node in a temporary pod with the nvidia-operator-validator service account. /host was a fake root whose /usr/bin/nvidia-smi prints Failed to initialize NVML: Driver/library version mismatch and exits with 18. The run used WITH_WAIT=true, --component driver, and a 12s timeout, and I ran it once with the main binary and once with the patched binary, both built from the same commit (CGO_ENABLED=0 GOOS=linux).

Before (main):

level=info msg="No pre-installed driver detected on the host: exit status 18"
level=warning msg="failed to validate the driver, retrying after 5 seconds\n"

After:

level=info msg="No usable pre-installed driver detected on the host: found '/usr/bin/nvidia-smi' on the host, but running it failed: exit status 18"
level=warning msg="failed to validate the driver, retrying after 5 seconds: failed to locate driver libraries: error locating \"libnvidia-ml.so.1\""

Prepared with help from Claude (AI), including the local checks described above.

Fixes #3085

🤖 Generated with Claude Code