Skip to content

nvidia-validator failed in Talos #1687

New issue
New issue
Closed as not planned
Closed as not planned
nvidia-validator failed in Talos#1687
Labels
featureissue/PR that proposes a new feature or functionalitylifecycle/frozenneeds-triageissue or PR has not been assigned a priority-px label

Description

@hzhangxyz
hzhangxyz
opened on Sep 10, 2025

Describe the bug
A clear and concise description of what the bug is.

nvidia-operator-validator-xxxxx keep in Init status as the container driver-validation loop forever because of its command nvidia-validator:

INFO[0000] version: 616690d8-amd64, commit: 616690d
INFO[0000] Attempting to validate a pre-installed driver on the host
INFO[0000] Attempting to validate a driver container installation
WARN[0000] failed to validate the driver, retrying after 5 seconds
INFO[0005] Attempting to validate a driver container installation
WARN[0005] failed to validate the driver, retrying after 5 seconds

To Reproduce
Detailed steps to reproduce the issue.

  1. Install talos with factory.talos.dev/metal-installer/cecf9ff004dbd06f21360a148bb5b4e231ca87f18d55eafc1191038ad50b4255:v1.11.0
  2. Install gpu operator

Expected behavior

Expect nvidia-validator return successfully, but it keep looping

Environment (please provide the following information):

  • GPU Operator Version: v25.3.2
  • OS: factory.talos.dev/metal-installer/cecf9ff004dbd06f21360a148bb5b4e231ca87f18d55eafc1191038ad50b4255:v1.11.0
  • Kernel Version: 6.12.45-talos
  • Container Runtime Version: containerd://2.1.4
  • Kubernetes Distro and Version: v1.33.0

*Information to attach

# kubectl --namespace gpu-operator logs pods/nvidia-operator-validator-7hcvp --container driver-validation | head
time="2025-09-10T03:06:07Z" level=info msg="version: 616690d8-amd64, commit: 616690d"
time="2025-09-10T03:06:07Z" level=info msg="Attempting to validate a pre-installed driver on the host"
time="2025-09-10T03:06:07Z" level=info msg="Attempting to validate a driver container installation"
time="2025-09-10T03:06:07Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-09-10T03:06:12Z" level=info msg="Attempting to validate a driver container installation"
time="2025-09-10T03:06:12Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-09-10T03:06:17Z" level=info msg="Attempting to validate a driver container installation"
time="2025-09-10T03:06:17Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"
time="2025-09-10T03:06:22Z" level=info msg="Attempting to validate a driver container installation"
time="2025-09-10T03:06:22Z" level=warning msg="failed to validate the driver, retrying after 5 seconds\n"

I also tried stracing it in pod, the key result of strace -o log -ff nvidia-validator is:

write(2, "\33[36mINFO\33[0m[0000] Attempting t"..., 79) = 79
newfstatat(AT_FDCWD, "/host/usr/lib/wsl/lib/nvidia-smi", 0xc00001a858, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/host/usr/bin/nvidia-smi", 0xc00001a928, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

and

write(2, "\33[36mINFO\33[0m[0000] Attempting t"..., 76) = 76
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/libnvidia-ml.so.1", 0xc00001b218, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/usr", 0xc00001b488, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/usr", 0xc00001b6f8, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/usr", 0xc00001b968, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/lib64", 0xc00001bbd8, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/lib", {st_mode=S_IFDIR|0755, st_size=71, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/lib/x86_64-linux-gnu", 0xc000240378, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr", {st_mode=S_IFDIR|0755, st_size=41, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local", {st_mode=S_IFDIR|0755, st_size=28, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/lib", {st_mode=S_IFDIR|0755, st_size=71, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/local/lib/aarch64-linux-gnu", 0xc0006562a8, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
write(2, "\33[33mWARN\33[0m[0000] failed to va"..., 77) = 77

The environment of this pod(container driver-validation) has been set as:

env:
    - name: WITH_WAIT
      value: "true"
    - name: COMPONENT
      value: driver
    - name: OPERATOR_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: DRIVER_INSTALL_DIR
      value: /usr/local
    - name: DRIVER_INSTALL_DIR_CTR_PATH
      value: /usr/local

and this information maybe helpful:

# find /usr/local -name nvidia-smi
/usr/local/bin/nvidia-smi
# find /usr/local -name 'libnvidia-ml.so*'
/usr/local/glibc/usr/lib/libnvidia-ml.so
/usr/local/glibc/usr/lib/libnvidia-ml.so.1
/usr/local/glibc/usr/lib/libnvidia-ml.so.570.172.08

Also see: #1276 #1050

Activity

  1. sinanmohd commented on Sep 10, 2025

    @sinanmohd
    sinanmohd
    on Sep 10, 2025

    We are running NixOS on our nodes and encountering the same issue.

    on nixos the nvidia-smi bin is located at
    /run/current-system/sw/bin/nvidia-smi

  2. mfernd commented on Oct 20, 2025

    @mfernd
    mfernd
    on Oct 20, 2025

    Any updates on this?

    We have the same problem on Talos, the solution we found for now is to only use the nvidia-device-plugin (so one step below).

    But it would be better if we could use the gpu-operator fully (to use all its features).

  3. visheshtanksale
    added
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 14, 2025

  4. salexo commented on Jan 14, 2026

    @salexo
    salexo
    on Jan 14, 2026

    I also had that issue, and it wasn't fixed by using the device plugin. What fixed it for me was including this in my Talos config:

    machine:
  sysctls:
    net.core.bpf_jit_harden: 1

    I had the kernel modules, but not this hardening config. This is the full patch I used:

    machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset
  sysctls:
    net.core.bpf_jit_harden: 1

  5. tariq1890 commented on Jan 14, 2026

    @tariq1890
    tariq1890
    on Jan 14, 2026
    Contributor

    @salexo Thanks for sharing. As for the net.core.bpf_jit_harden: 1 sysctl setting, it is only required when toolkit is used in legacy mode(non-CDI). Starting from v25.10.0 of gpu-operator, CDI is enabled by default, so you should no longer be required to set that sysctl parameter (so long as you don't explicitly disable CDI)

  6. myeolenv
    added
    questionCategorizes issue or PR as a support question.
    featureissue/PR that proposes a new feature or functionality
    and removed
    questionCategorizes issue or PR as a support question.
    on Feb 21, 2026
  7. ssalagam
    added
    lifecycle/frozen
    on Feb 26, 2026
  8. Arsolitt
    added a commit that references this issue on Apr 17, 2026

    docs(gpu-operator): add native-pod Talos reference manifests

    5d6654c
  9. lexfrei
    added a commit that references this issue on Apr 28, 2026

    feat(monitoring): add GPU observability dashboards and recording rules (

    3977aef

  10. francisguillier commented on Sep 14, 2026

    @francisguillier
    francisguillier
    on Sep 14, 2026
    Contributor

    We’re closing this RFE for now, as Talos is not currently on our priority roadmap.
    If this capability is critical to your environment, please feel free to reopen the issue and include the relevant use case and business justification so we can reassess its priority.
    Thanks.

  11. francisguillier
    closed this as not plannedon Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureissue/PR that proposes a new feature or functionalitylifecycle/frozenneeds-triageissue or PR has not been assigned a priority-px label

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions