[Bug]: Latest relese does not work on Talos Linux because of incorrect version parsing

Describe the bug
Latest release fails to list gpus because it fails to process OS version:
{"level":"error","ts":1774112196.4380157,"msg":"Reconciler error","controller":"clusterpolicy-controller","object":{"name":"cluster-policy"},"namespace":"","name":"cluster-policy","reconcileID":"991b62e2-34f1-4844-9ed3-fe049d7a5c04","error":"failed to retrieve GPU node OS tag: error processing OS major version v1: strconv.Atoi: parsing "v1": invalid syntax"}

Annotations on node are the following:
"feature.node.kubernetes.io/system-os_release.ID": "talos",
"feature.node.kubernetes.io/system-os_release.VERSION_ID": "v1.12.6",

Operator 25.10.1 works fine. Likely, it was broken in 3fb6dd3

To Reproduce
Install on Talos linux

Expected behavior
OPerator works

Environment (please provide the following information):

GPU Operator Version: 26.3.0
OS: Talos 1.12.6
Kernel Version: 6.18.18-talos
Container Runtime Version: 2.1.6
Kubernetes Distro and Version: v1.34.0

@finomen

Thanks @finomen for filing this bug. Does the following image works fine for you? I've provided sample values.yaml change to pull this fix until we release a newer patched version.

operator:
  repository: ghcr.io/nvidia
  image: gpu-operator
  version: 670be908

@rahulait

Thanks @rahulait, I can confirm that image resolved the same issue I had (talos 1.10.6)

fix(gpu-operator): use nonfree drivers and patched operator for DGX S…

We checked have the same issue with the above provided Image and Talos 1.12.6 even when we define the new image.

failed to retrieve GPU node OS tag: unable to retrieve OS name from label feature.node.kubernetes.io/system-os_release.ID

    helm:
      values: |
        operator:
          repository: ghcr.io/nvidia
          image: gpu-operator
          version: 670be908
          defaultRuntime: containerd

@ch9hn

@ch9hn can you share what node labels you have on the node? You can also specify the version as "main-latest" as the fix has been merged to main. I would like to see what labels your node has and why its failing there.

Hello the default labels which are added to the nodes are these:

TL;DR;
Found the issue, it was coming from the fact that the node feature discovery was not running.

Furthermore we gave up, as the GPU Operator does not support Talos OS in a proper way (yet), see here:
siderolabs/talos#9014

Error creating: pods "nvidia-gpu-operator-node-feature-discovery-worker-lgdlg" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volumes "host-boot", "host-os-release", "host-sys", "host-usr-lib", "host-lib", "host-proc-swaps", "features-d")

beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
extensions.talos.dev/amd-ucode=20260309
extensions.talos.dev/bnx2-bnx2x=20260309
extensions.talos.dev/intel-ucode=20260227
extensions.talos.dev/modules.dep=6.18.18-talos
extensions.talos.dev/nvidia-container-toolkit-lts=580.126.20-v1.18.2
extensions.talos.dev/nvidia-gdrdrv-device=v2.5.1
extensions.talos.dev/nvidia-open-gpu-kernel-modules-lts=580.126.20-v1.12.6
extensions.talos.dev/nvme-cli=v2.14
extensions.talos.dev/qlogic-firmware=20260309
kubernetes.io/arch=amd64
kubernetes.io/hostname=xxx
kubernetes.io/os=linux
nvidia.com/gpu.present=true # set manually

Ive added manually then the following labels, as they where not added by default:

feature.node.kubernetes.io/kernel-version.full=6.18.18-talos
feature.node.kubernetes.io/pci-10de.present=true
feature.node.kubernetes.io/system-os_release.ID=talos
feature.node.kubernetes.io/system-os_release.VERSION_ID=v1.12.6

Image which I used:
ghcr.io/nvidia/gpu-operator:main-latest

Same error:

"level":"error","ts":1776152960.5453749,"logger":"controllers.ClusterPolicy","msg":"unable to initialize ClusterPolicy controller","error":"failed to retrieve GPU node OS info: unable to retrieve OS name from label feature.node.kubernetes.io/system-os_release.ID"}
{"level":"error","ts":1776152960.5566223,"msg":"Reconciler error","controller":"clusterpolicy-controller","object":{"name":"cluster-policy"},"namespace":"","name":"cluster-policy","reconcileID":"c8ffc1f7-079e-4af0-b173-8eb4a867fbf7","error":"failed to retrieve GPU node OS info: unable to retrieve OS name from label feature.node.kubernetes.io/system-os_release.ID"}
{"level":"info","ts":1776152963.557338,"logger":"controllers.ClusterPolicy","msg":"Sandbox workloads","Enabled":false,"DefaultWorkload":"container"}

Logs:

Node Feature Discovery stops and is not deploying more pods on each node:


I0414 07:48:31.706778       1 nfd-master.go:268] "Node Feature Discovery Master" version="v0.18.3" nodeName="xxxxx" namespace="gpu-operator"
I0414 07:48:31.706930       1 nfd-master.go:1167] "configuration file parsed" path="/etc/kubernetes/node-feature-discovery/nfd-master.conf"
I0414 07:48:31.707083       1 nfd-master.go:1215] "configuration successfully updated" configuration=<
	DenyLabelNs: {}
	EnableTaints: false
	ExtraLabelNs:
	  nvidia.com: {}
	InformerPageSize: 200
	Klog: {}
	LabelWhiteList: null
	LeaderElection:
	  LeaseDuration:
	    Duration: 15000000000
	  RenewDeadline:
	    Duration: 10000000000
	  RetryPeriod:
	    Duration: 2000000000
	NfdApiParallelism: 10
	NoPublish: false
	Restrictions:
	  AllowOverwrite: true
	  DenyNodeFeatureLabels: false
	  DisableAnnotations: false
	  DisableExtendedResources: false
	  DisableLabels: false
	  NodeFeatureNamespaceSelector: null
	ResyncPeriod:
	  Duration: 3600000000000
 >
I0414 07:48:31.707121       1 nfd-master.go:1299] "starting the nfd api controller"
I0414 07:48:31.807567       1 nfd-api-controller.go:202] "informer caches synced" duration="100.318814ms"
I0414 07:48:31.807620       1 updater-pool.go:143] "starting the NFD master updater pool" parallelism=10

These labels are added by node-feature-discovery and that is installed by default. I am not sure why NFD was not running in your environment (either it was intentionally disabled, or it must be having some errors running which needs to be identified first). We install upstream NFD as a helm dependency and then other gpu-operator related operands start to come up. I would suggest fixing issues with NFD first.

Talos doesn't allow compiling of driver modules on the node and many other things which gpu-operator needs for building and installing kernel specific nvidia drivers. One can have drivers pre-installed and then configure gpu-operator to install rest without driver management.

Is there a target date for the v26.3.1 release?

Is there a target date for the v26.3.1 release?

We are planning to release by this month end, so hopefully soon.

Its released (v26.3.1)

Navigation Menu

Repository navigation

[Bug]: Latest relese does not work on Talos Linux because of incorrect version parsing #2239

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Latest relese does not work on Talos Linux because of incorrect version parsing #2239

Description

Activity

rahulait commented on Mar 24, 2026

gimmw commented on Mar 31, 2026

ch9hn commented on Apr 13, 2026

rahulait commented on Apr 14, 2026

ch9hn commented on Apr 14, 2026

rahulait commented on Apr 14, 2026

mpepping commented on Apr 16, 2026

rahulait commented on Apr 16, 2026

rahulait commented on Apr 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions