[Bug]: driver-validation hides why a pre-installed driver failed and loops without a reason #3085

Describe the bug

With driver.enabled=false, if nvidia-smi exists on the host but fails (for example Failed to initialize NVML: Driver/library version mismatch after an unattended upgrade of the driver packages without a reboot), the driver-validation init container of nvidia-operator-validator does three things. It logs No pre-installed driver detected on the host: exit status 18. It then falls through to the driver-container path. After that it logs failed to validate the driver, retrying after 5 seconds\n every 5 seconds without the underlying error. The pod stays in Init, and nothing in the validator log says that the host driver is installed but broken.

To Reproduce

1. Use a node with a pre-installed driver and driver.enabled=false.
2. Make the host nvidia-smi fail. One way is to upgrade the driver user-space packages without reloading the kernel module. A fake /host/usr/bin/nvidia-smi that prints the mismatch message and exits non-zero gives the same logs.
3. Read the driver-validation container logs of a fresh nvidia-operator-validator pod on that node.

Expected behavior

The log says that nvidia-smi was found on the host and failed, and the retry warning includes the error it is retrying on.

Environment

• GPU Operator v26.3.3 (also current main), pre-installed driver on Ubuntu, Kubernetes v1.34, containerd.

Related: #1276, #1687 (users stuck on the same retry line without a reason). I have a small patch ready.

This issue was drafted with AI assistance and reviewed by me.