Release v1.31.0

This release adds Akamai Cloud (LKE) support to janitor-provider and csp-health-monitor, adds IPv6 support, and lets janitor stop GPU clients outside the GPU Operator and patch the GPU reset Job. A new default fault-quarantine rule set cordons nodes for fatal health-events-analyzer events. It also fixes several PostgreSQL change-stream problems, stops a GPU reset that a retry recovers from rebooting the node, and keeps fault-quarantine running on a cluster with no GPU nodes.

Upgrade Notes

Fault-quarantine rule set for health-events-analyzer events (#2335, #2297)

health-events-analyzer publishes fatal derived events, such as MultipleRemediations and RepeatedXID74Reg0HardwareIssue. No default fault-quarantine rule set matched them, so these events set node conditions but did not cordon, drain or remediate the node. The new default rule set Health events analyzer fatal error ruleset cordons the node for a fatal health-events-analyzer event. This is a change to the default of fault-quarantine.ruleSets. helm upgrade --reuse-values keeps your previous rule sets. If you run health-events-analyzer and want these events to quarantine nodes, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults. If you set your own ruleSets, copy the rule set from the v1.31.0 chart.

XID 74 rules in health-events-analyzer (#2271, #2270)

The four XID 74 solo rules took their registers and their GPU from the node's stored events, not from the incoming XID 74 event. After one matching XID 74 on a GPU, every later XID 74 on the node fired the rule, and the CONTACT_SUPPORT event could name a GPU whose registers did not match, or a GPU that was not solo. The rules now read the registers and the GPU from the incoming event. The fix is in the default value of health-events-analyzer.config. helm upgrade --reuse-values keeps your previous value. If you run health-events-analyzer, upgrade with --reset-then-reuse-values. If you set your own config, copy the four XID 74 solo rules from the v1.31.0 chart.

Major New Features

Akamai Cloud (LKE) Support (#2114, #2115)

NVSentinel now supports Akamai Cloud (Linode Kubernetes Engine, LKE):

• janitor-provider (#2114): with janitor-provider.csp.provider: akamai, janitor reboots and replaces nodes through the Akamai Cloud API. A reboot is complete when the instance is running and the kubelet reports a new boot ID. A terminate recycles an LKE node, so LKE provisions a replacement and the node pool keeps its size. An instance outside LKE is deleted. The provider acts only on a node with a linode://<id> provider ID. Set the API token Secret in janitor-provider.csp.akamai.apiTokenSecretRef. The token needs the linodes:read_write and lke:read_write scopes.
• csp-health-monitor (#2115): with csp-health-monitor.cspName: akamai, the monitor polls the Akamai account maintenance API and quarantines and drains a node before its maintenance starts. Every maintenance type quarantines the node, live_migration included, because a GPU workload cannot live-migrate. The poll interval is configToml.akamai.pollingIntervalSeconds, default 60, minimum 30. Set the token Secret in csp-health-monitor.akamaiApiTokenSecret. The token needs the account:read_only scope.

These scopes cover the whole Akamai account, so use a dedicated token for NVSentinel.

IPv6 Support (#2252)

Set global.ipv6.enabled: true to run NVSentinel on an IPv6-only or dual-stack cluster. The default is false. With the flag on, gpu-health-monitor binds its metrics server to ::, Bitnami MongoDB sets MONGODB_ENABLE_IPV6, Percona MongoDB adds net.ipv6: true to the default replica set configuration, and the in-cluster file server also listens on [::]:8080. Every node must boot with IPv6 on. Also set the GPU Operator's DCGM host engine to bind :: with dcgm.args in the GPU Operator chart, which needs GPU Operator v26.7.1 or later. If DCGM binds only 0.0.0.0, the labeler sets no DCGM version label, and gpu-health-monitor schedules no pods.

With helm upgrade --reuse-values, the Percona replica set configuration keeps its previous value. To turn on IPv6 with the Percona backend, upgrade with --reset-then-reuse-values. If you set your own mongodb-store.psmdb-db.replsets.rs0.configuration, add net.ipv6: true to it.

GPU Reset Stops GPU Clients Outside the GPU Operator (#2276, #1691)

A GPU client that runs outside the GPU Operator namespace, for example a monitoring agent, keeps the GPUs open, and nvidia-smi --gpu-reset fails with In use by another client. GPU Operator 26.7 and later sets the nvidia.com/gpu.deploy.client=true node label, and GPU clients select it. Before a GPU reset, janitor now sets this label to false when at least one GPU client pod runs on the node. It waits until the DaemonSet controller removes the GPU client pods, and sets the label back to true after the reset. NVSentinel's metadata-collector selects this label too, so it also leaves the node during the reset. The janitor ClusterRole gains get on DaemonSets.

GPU Reset Job Patch (#2324, #1782)

The new janitor.config.controllers.gpuReset.resetJob.jobTemplatePatch value is a strategic merge patch that janitor applies on top of the built-in GPU reset Job, so you can change the Job without forking janitor. janitor always sets the Job name, namespace, owner reference and node name, and the NVIDIA_GPU_RESETS variable of each container, itself. janitor refuses at startup a patch with a misspelled field, a field in the wrong case or a value of the wrong type. Init containers now also get NVIDIA_GPU_RESETS. The patch is empty by default. The janitor configuration docs give the merge rules and an example.

Mount Only the Journal Directory (#2292, #2291)

The new syslog-health-monitor.journalDirectoryHostPath value mounts only the host journal directory in regular mode, in place of its parent in journalHostPath. Use it where a vcluster syncer rewrites a /var/log host path to a per-tenant directory, which hides the journal. The directory must exist on every node. The value is empty by default.

Metrics

• metadata_collector_collection_completed_timestamp_seconds (#2157): a new gauge. It is 0 until GPU metadata collection completes, and then holds the completion time. To find a collector that is stuck, alert on metadata_collector_collection_completed_timestamp_seconds == 0 and time() - process_start_time_seconds > 300.
• fault_quarantine_get_total_nodes_retry_attempts is removed (#2273, #2272): fault-quarantine no longer retries a GPU node count of 0. Remove dashboards and alerts that use this metric.
• External remediation metrics (#2161): nvsentinel_external_remediation_total{phase="closed"} gains the result value node_deleted, for a request whose Node is missing, terminating or replaced. nvsentinel_external_remediation_open is now set to 0 or 1, and janitor restores it after a restart.

Bug Fixes & Reliability

• A GPU reset that a retry recovers does not reboot the node (#2254, #2279): the reset Job runs gpu_reset.sh again after a failed run, and every run wrote its result to syslog. When the first run failed and the retry reset the GPU, syslog-health-monitor had already requested a reboot for the first failure. The reset Job now reports a failed reset only from its last run. When the last run fails, the node still reboots.
• Node patches retry after a connection error (#2267, #2266): a node patch whose connection failed before the response, for example when the API server restarted, was not retried. In fault-quarantine, this was the cordon of a fatal event, so the event was checkpointed and the node stayed schedulable until fault-quarantine restarted. The node patches of fault-quarantine, the labeler and lifecycle-manager now retry connection resets, unexpected EOFs, lost HTTP/2 connections and refused connecti...

Release v1.30.0

This release adds support for the XID 154 recovery actions of R610 and later drivers, lets the labeler's default GPU class follow a GPU Operator mode switch without manual configuration, adds an option to run metadata-collector without its pod device mapper, and labels platform_connector_health_events_received_total by processing strategy and health. It also fixes the bundled MongoDB network policy for a release that is not named after its namespace, and stops a truncated node condition from staying True after every GPU recovers.

Upgrade Notes

Bundled MongoDB network policy follows the release name (#2186, #2185)

mongodb-networkpolicy selected the bundled Bitnami MongoDB pods by the release namespace, but the pods carry the release name. With the default networkPolicy.enabled: true, the metrics-access policy still isolated the MongoDB pods, so it blocked port 27017, the replica set could not form, and platform-connectors could not reach MongoDB. The policy now selects the pods by release name.

If your release name is different from its namespace and you set networkPolicy.enabled: false, no policy selected the MongoDB pods before. Now mongodb-networkpolicy selects them and allows only port 27017 from the release namespace, so a Prometheus in another namespace can no longer scrape the MongoDB exporter on port 9216. To allow it, add a NetworkPolicy for port 9216, for example through mongodb-store.mongodb.extraDeploy.

Labeler GPU class with --reuse-values (#2108, #1892)

The new default GPU class (see Major New Features) is a change to labeler.expectedDeviceCounts.classes. helm upgrade --reuse-values keeps your previous class list, so GPUCluster mode still needs manual configuration. If you run GPUCluster mode, or switch between modes, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults. If you set your own classes, for example to enable the gpu-dra example class, replace your GPU classes with the new default GPU class, as the labeler configuration docs show. Without sources, a GPU DRA class can report 0 on a node that has only the ResourceSlices of another driver.

Major New Features

R610 and Later Driver Support for XID 154 (#2166, #2174, #2165)

XID 154 reports the recovery action that the GPU driver asks for. R610 and later drivers print PF FLR and OS Reboot where earlier drivers printed GPU Reset Required and Node Reboot Required, and they add Bus Reset and System Reboot. syslog-health-monitor now maps PF FLR to COMPONENT_RESET, and OS Reboot, Bus Reset and System Reboot to RESTART_BM. A bus reset covers more than the single-GPU reset that COMPONENT_RESET runs, so it gets a node reboot. With fault-remediation enabled, NVSentinel resets the GPU or reboots the node for these recovery actions on R610 and later drivers.

Labeler GPU Count Follows the GPU Operator Mode (#2108, #1892)

The labeler could already count GPUs in GPU Operator GPUCluster (DRA) mode, but only after you enabled the gpu-dra example class, and a switch between ClusterPolicy and GPUCluster mode needed a configuration change. The default GPU class now counts GPUs in both modes, so a mode switch needs no manual configuration. The class lists two sources, and the labeler uses the first source that applies to each node:

• dra counts the GPUs in the node's gpu.nvidia.com ResourceSlices, and groups nodes by instance type and by the GPU models in the slices. It applies when the node has slices from that driver. If a pool is incomplete or the expression fails, the labeler skips the class and does not use the GFD label.
• gfd reads the GFD nvidia.com/gpu.count label, but only while the NVIDIA device plugin advertises GPUs. Thus a label left from ClusterPolicy mode does not count on a node whose DRA plugin is down.

When no source applies, the labeler keeps the existing labels. A node that switches modes keeps its expected count. Any device-count class can now list ordered sources. Because the default class reads ResourceSlices, the labeler ClusterRole now gets get, list and watch on resourceslices. See the Upgrade Notes for --reuse-values.

Run metadata-collector Without the Pod Device Mapper (#2082, #2069)

On a vcluster, the kubelet /pods endpoint and the PodResources socket describe host pods. Their names do not match the pods that metadata-collector sees through the virtual cluster's API server, so the pod device mapper always failed and the container exited. The new metadata-collector.podMapper.enabled value, default true, turns the mapper off. With the mapper off, metadata-collector still writes the GPU metadata file, serves metrics, and keeps running. Partial drains need the dgxc.nvidia.com/devices annotation that the mapper writes, so keep node-drainer.partialDrainEnabled: false.

Metrics

• platform_connector_health_events_received_total gains processing_strategy and healthy labels (#2095, #2089): the counter had no labels, so actionable events could not be told apart from STORE_ONLY and recovery events. platform-connectors counts each event after it applies its defaults, so processing_strategy is the stored strategy, and UNSPECIFIED counts as EXECUTE_REMEDIATION. healthy is "true" or "false". The counter has at most eight series. It no longer counts rejected batches; platform_connector_request_duration_seconds{outcome="rejected"} shows them. A query that reads the counter without sum() now returns one series for each label pair.
• Labeler device-count metrics (#2108): the new labeler_device_count_source{node, class, source} gauge shows the source that the labeler used for each node. With the new default GPU class, the partition label of labeler_device_count_expected starts with the source, for example source=dra|node.kubernetes.io/instance-type=p5|hardware=NVIDIA H100 80GB HBM3. Update dashboards and alerts that match the earlier partition values of the gpu class.

Bug Fixes & Reliability

• Bundled MongoDB works when the release name is different from the namespace (#2186, #2185): see the Upgrade Notes.
• maxRemediationAttempts holds through post-remediation validation (#2149, #2148): fault-quarantine ends the quarantine session when it hands a node to post-remediation validation. fault-remediation cleared the attempt count at that point, so a fault found while validation still held the node started a new budget, and maxRemediationAttempts did not stop a node that kept failing validation. fault-remediation now keeps the count until validation releases the node, and the next fault after that starts from zero. With the cap set, the latestFaultRemediationState node annotation now stays on the node after the hand-off to validation.
• Fault-remediation remediates nodes with names longer than 63 characters (#2151, #2150): with the log collector enabled, fault-remediation put the node name in the dgxc.nvidia.com/node-name label of the log collector Job. Kubernetes limits a label value to 63 characters, so for a longer node name the API server rejected the Job lookup and creation, and fault-remediation never remediated the node. fault-remediation now cuts a longer name and adds the first 8 hex characters of the SHA-256 hash of the full name.
• Node conditions clear after every GPU recovers (#2168, #2167): when the entries of a node condition did not fit in platformConnector.k8sConnector.maxNodeConditionMessageLength after compaction, platform-connectors cut the last entry at a byte boundary. A fragment cut before its GPU and PCI tokens, such as Error;, matched no recovery event, so the condition stayed True after every GPU recovered. platform-connectors now first shortens each entry to its error code, entity tokens and recommended action, and cuts at a byte boundary only when the entries still do not fit. This fix does not repair a condition that an earlier version left in this state; clear it with ...

Release v1.29.0

This release lets node-drainer select a drain mode by the kind of a pod's controller and adds SELinux support for RKE2 and SUSE Linux Enterprise. It also stops the generic reboot from rebooting a node twice, stops fault-remediation from creating a second maintenance CR after a read error, and fixes node drains that never finished on a bundled MongoDB with fewer than three members. Several behaviours change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

Generic reboot image needs sh and cat (#2110, #2077)

With janitor-provider.csp.provider: generic, every reboot Job now runs its command through sh -c, because the command first checks the node boot ID. The default command was chroot /host reboot, which ran without a shell. The image in janitor-provider.csp.generic.rebootImage must now include sh and cat. The default busybox image has both. If your custom image has no sh, the reboot container does not start and the node does not reboot.

Bundled MongoDB disruption budget (#2112, #2111)

The bundled Bitnami MongoDB set mongodb-store.mongodb.pdb.minAvailable: 2. With fewer than three members, the PodDisruptionBudget allowed no evictions, so a drain of a node that hosted a member never finished. The chart now leaves pdb.minAvailable and pdb.maxUnavailable empty, and the subchart renders maxUnavailable: 1. With one or two members, a drain can now evict a member, and the replica set has no primary until the pod runs again. With more than three members, the budget allows one eviction at a time, where minAvailable: 2 allowed more.

helm upgrade --reuse-values keeps minAvailable: 2. If you run the bundled MongoDB with a member count other than three, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults.

RepeatedXIDErrorOnSameGPU skips XID 45 events (#2064, #2050)

The RepeatedXIDErrorOnSameGPU rule in health-events-analyzer never counted XID 45, but each XID 45 event still started the rule's 24-hour MongoDB query. A job teardown can produce a burst of XID 45 events, and these queries loaded the MongoDB primary and delayed change-stream processing. XID 45 events no longer start the rule.

The fix is in the default value of health-events-analyzer.config. helm upgrade --reuse-values keeps your previous value, so XID 45 events still start the query. If you run health-events-analyzer, upgrade with --reset-then-reuse-values. If you set your own config, copy the two XID 45 exclusions from the RepeatedXIDErrorOnSameGPU rule in the v1.29.0 chart.

Major New Features

Drain Modes by Pod Owner Kind (#2049, #1689)

A node-drainer.podDrainPolicies entry can now select pods by the kind of their controller, with the new ownerKinds list. A policy needs podSelector, ownerKinds, or both. With both, a pod must match both. node-drainer compares ownerKinds exactly with the kind in the pod's controller owner reference, and does not follow owners further, so Deployment pods match ReplicaSet and CronJob pods match Job. Pods without a controller do not match an ownerKinds policy. podDrainPolicies and userNamespaces are mutually exclusive, and userNamespaces is not empty by default, so set it to []:

node-drainer:
  userNamespaces: []
  podDrainPolicies:
    - name: stateful-workloads
      ownerKinds: ["StatefulSet"]
      mode: AllowCompletion

Bug Fixes & Reliability

• Generic reboot reboots a node once (#2110, #2077): with the generic CSP, the reboot stopped the reboot pod before the kubelet recorded a final status. After the node booted, the kubelet started the same pod again, and the pod rebooted the node a second time. janitor-provider now passes the node boot ID to the reboot container in PRE_REBOOT_BOOT_ID. The container reboots the node only when the node still has that boot ID. When the boot ID is different, the container exits without a reboot. When it cannot read either boot ID, it fails and does not reboot. See the Upgrade Notes for the image requirement.
• Fault-remediation does not create a second maintenance CR after a read error (#2093, #2092): fault-remediation treated any error from reading the maintenance CR as NotFound. It then removed the CR from the node annotation and created a new one, so a transient read error while the first CR ran gave the node a second CR. Only a NotFound error now counts as a missing CR. Fault-remediation counts other read errors in fault_remediation_processing_errors_total as cr_status_check_error and retries the event. In a partial recovery, an unreadable CR no longer sets the node to remediation-succeeded.
• Node-drainer dry run does not create custom drain CRs (#2076, #2075): in dry run, node-drainer sent evictions with DryRun: [All], but with custom drain enabled it created a real custom drain CR, and the external drain controller then drained the node. node-drainer now creates and deletes the custom drain CR with DryRun: [All]. The API server validates the CR but does not store it.
• Bundled MongoDB does not block drains below three members (#2112, #2111): see the Upgrade Notes.
• Kubernetes-object-monitor publishes healthy events after a restart (#2078, #1542): the monitor saves each unhealthy match in the nvsentinel.nvidia.com/k8s-object-monitor-policy-matches node annotation, so that it can publish the healthy event after a restart. It never loaded these matches, because it read them before its cache started. Also, a resource deleted while the monitor was stopped was not in its first list, so nothing reconciled it. In both cases, the node stayed cordoned until someone uncordoned it by hand. The monitor now loads the saved matches at startup and reconciles each one after its cache syncs. The first start after the upgrade also processes the matches that earlier versions saved. If the monitor cannot load the matches, it now exits instead of starting without them.
• Platform-connectors keeps the node condition of a fatal event (#2071, #2070): the connector wrote node conditions with UpdateStatus at the resourceVersion it read. For a fatal event, fault-quarantine, node-drainer and fault-remediation update the same Node within a few hundred milliseconds, and each update made the condition write fail with a conflict. When the reboot stopped the connector before its next retry, the node never showed the condition. The connector now writes the conditions with a JSON patch that tests only the current conditions, so label, annotation, taint and cordon changes do not conflict with it. The deployment platform connector's ClusterRole now also grants patch on nodes/status.
• Janitor waits for the reset Job before it restores GPU services (#2022, #1991): when a GPUReset was deleted after its services were torn down but before its reset Job finished, janitor re-enabled the managed GPU services while nvidia-smi --gpu-reset still ran. DCGM, and in GPUCluster mode the DRA kubelet plugin, could attach to GPUs during the reset. The deletion now waits for the Job to finish. If the Job runs past the GPUReset timeout, janitor restores the services and logs an error.
• A GPUReset without spec.selector starts (#2066, #2065): without spec.selector, a GPUReset resets every GPU on the node. The update webhook read the selector without a nil check, so it panicked on the first update, which adds the finalizer, and denied it. The reset never started. The webhook now treats a missing selector as an empty UUID list.
• platform-connectors and nic-health-monitor run on SELinux-enforcing nodes (#1959, #1953): on SELinux-enforcing nodes, for example RKE2 installed from RPM on SUSE Linux Enterprise Server 16 or RHEL, unprivileged pods run as container_t. platform-connectors could...

Release v1.28.0

NVSentinel is now graduated.

This release makes metadata-collector and embedded-mode gpu-health-monitor leave a node during GPU Operator driver upgrades and MIG changes, lets MongoDB change-stream consumers recover from transient failures, and adds Kubernetes 1.37 support to preflight gang discovery. It also detects XIDs on MIG GPUs and fixes remediation on Amazon DocumentDB. Several behaviours change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

metadata-collector and embedded gpu-health-monitor follow nvidia.com/gpu.deploy.client (#1988, #1868)

metadata-collector, and gpu-health-monitor in embedded-mode, now have a default required node affinity on the nvidia.com/gpu.deploy.client node label. A pod runs when the label is true or absent. The DaemonSet controller removes the pod for any other value, such as paused-for-driver-upgrade, paused-for-mig-change, or false. While the pod is off the node, embedded-mode gpu-health-monitor does not report GPU health for that node. If a driver upgrade or MIG change fails and leaves the label at a paused value, the pods stay off the node until the label is true again.

The default rule applies only when neither global.affinity nor the component's own affinity is set. global.affinity applies to every NVSentinel component, so if you set it for any reason, these DaemonSets do not get the rule. To keep the rule together with a custom affinity, add its two terms to your affinity, as shown in the metadata-collector and gpu-health-monitor configuration docs.

Remote DCGM probe hangs are store-only by default (#1984, #1978)

In the operator-service and external-hostengine modes, gpu-health-monitor reports a hung DCGM probe as a GpuDcgmConnectivityFailure event with error code DCGM_PROBE_HANG. This event used the global processing strategy (EXECUTE_REMEDIATION by default), so the default rules cordoned the node. It now follows gpu-health-monitor.dcgm.probeStoreOnly, which defaults to true, so a remote probe hang is stored and NVSentinel does not act on it. Other DCGM connectivity errors still use the global strategy. To restore remediation for remote probe hangs, set probeStoreOnly: false. That setting also enables remediation for the embedded-mode GpuDcgmUnresponsive event, which recommends RESTART_BM.

MongoDB metrics in a namespace other than nvsentinel (#2053, #2052)

The bundled Bitnami MongoDB metrics exporter connected to mongodb-headless.nvsentinel.svc in every namespace, so MongoDB metrics were missing when NVSentinel ran in any other namespace. The fix changes the default value of mongodb-store.mongodb.metrics.args. helm upgrade --reuse-values keeps your previous value, so the exporter still connects to the wrong namespace. If you run the bundled MongoDB in a namespace other than nvsentinel, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults.

Major New Features

GPU Operator Driver Upgrades and MIG Changes (#1988, #1868)

GPU Operator 26.7 and later uses the nvidia.com/gpu.deploy.client node label to move GPU clients off a node before it unloads the driver or changes the MIG layout. metadata-collector and embedded-mode gpu-health-monitor keep the GPUs open, but they ignored this label, so during a driver upgrade they could keep the old driver in use. Both DaemonSets now leave the node while the label holds any value other than true, and return when it is true again. Older GPU Operator releases and GPUCluster mode do not set the label, so the pods run there as before. The operator-service and external-hostengine modes of gpu-health-monitor do not open the GPUs, so they do not get the rule. See the Upgrade Notes for how the rule interacts with a custom affinity.

MongoDB Change-Stream Recovery (#2040, #2030)

A transient MongoDB failure ended change-stream processing for good: the watcher returned instead of reopening the stream. The watcher now reopens the stream from its stored resume token, and keeps trying for up to 5 minutes. The budget is wall-clock time, sized for a replica-set roll, which takes about two minutes on a 3-member Percona cluster. When the budget runs out, the stream stops and the consumer exits, so a persistent failure shows as a pod restart instead of a Ready pod that processes nothing. This applies to health-events-analyzer, fault-quarantine, node-drainer, and fault-remediation. The new change_stream_reopens_total{client,result} counter reports success, failure, and exhausted; alert on exhausted to see why a consumer restarted.

Preflight Gang Discovery on Kubernetes 1.35 and 1.37 (#2048, #2047)

Preflight's native gang discovery, the default when gangDiscovery is {}, now works on Kubernetes 1.35 and 1.37:

• Kubernetes 1.37: the discoverer used the scheduling.k8s.io/v1alpha2 PodGroup, but Kubernetes 1.37 serves PodGroup only as v1beta1. Preflight failed startup validation and crash-looped, and with failurePolicy: Fail, pod creation failed in every preflight-enabled namespace. Preflight now uses the PodGroup version that the cluster prefers.
• Kubernetes 1.35: preflight never detected a gang. The webhook dropped the pod's spec.workloadRef when it decoded the pod, and it could not read the field back from the API server, because the pod is not stored until admission completes. The webhook now reads the gang from the admitted pod. When spec.workloadRef.podGroupReplicaKey is set, the gang ID includes it, because Kubernetes applies the gang policy to each replica separately.

XID 154 Recovery Clears XID Conditions (#1997, #1992)

When XID 154 reports that the GPU recovery action has returned to None, syslog-health-monitor now publishes a healthy event for the same GPU entities. This clears the earlier XID conditions on that GPU. Conditions for other GPUs on the node stay in place. The documentation now lists the explicit signals that clear an XID condition, and states that a quiet period does not clear one.

Remote DCGM Probe Hangs Follow runtimeDebounce (#1987, #1978)

A hung remote DCGM probe was published at the probe deadline, whatever runtimeDebounce.failureThreshold said, because a hung call has no later cycle to count and liveness restarts the container first. gpu-health-monitor now publishes a remote probe hang only after DCGM has been unreachable for failureThreshold * pollIntervalSeconds. It saves the start of this window in its state file, so the window continues across liveness restarts. A successful cycle ends the window. This takes effect only when runtimeDebounce.failureThreshold is above its default of 1.

NIC Port-State Hold-Down (#1996, #1993)

A port that dropped for a few seconds was reported fatal at once, so a short fabric blip could cordon a node. The new portStateHoldDown value sets how long a port must stay unhealthy before InfiniBandStateCheck or EthernetStateCheck reports it. A port that recovers before the hold-down ends publishes nothing and increments nic_health_monitor_port_state_blips_total{node,check,device,port}. Recovery, newly seen ports, and the escalation of a port already reported unhealthy are not held. The hold-down starts again after a restart. It is off by default ("0s").

Configurable Driver Binary Path for GPU Reset (#2013, #1681)

The GPU reset Job runs every nvidia-smi call as chroot <driverRoot> nvidia-smi, so nvidia-smi must be on the default PATH inside the chroot. The GKE driver installer puts the driver in /home/kubernetes/bin/nvidia, so the reset failed with exit code 127 before it reset anything. The new janitor.config.controllers.gpuReset.resetJob.driverBinPath value names a directory inside driverRoot that holds nvidia-smi and nvidia-bug-report.sh. The reset script adds it to PATH for the chroot command only. janitor refuses at startup a value that is not an absolute, clean path without :. For the GKE driver installer:

janitor:
  config:
    controllers:
      gpuReset:
        resetJob:
          hostDriverRootPath: "/"
          driverBinPath: "/home/kubernetes/bin/nvidia/bin"

Bug Fixes & Reliability

• Fault-remediation sees drained nodes on Amazon DocumentDB (#2027): on DocumentDB, fault-remediation never saw that a node was drained, so it created no remediation CR and the node stayed cordoned, with nothing in the logs. The change-stream filter read the update with $objectToArray, which DocumentDB evaluates to null in a change-stream $match. DocumentDB also reports the drain status under a different key than MongoDB. The filter now reads the status with $getField under both keys.
• XIDs on MIG GPUs are detected ([#2060](https://github.com/NVIDIA/NVS...

Release v1.27.0

This release adds GPU Operator GPUCluster (DRA) mode support to janitor GPU reset, lifecycle-manager, node-drainer, and preflight. It also adds operator recovery for health-events-analyzer conditions, poll-stall detection for the NIC health monitor, and a percentage form of the standing cordon bound. Several defaults and behaviours change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

GpuThermalMarginWatch requires three consecutive polls (#1982, #1980)

GpuThermalMarginWatch now fails a GPU only when the thermal margin stays below the slowdown threshold for gpu-health-monitor.dcgmFieldsMonitoring.gpuTempLimitMinConsecutivePolls consecutive polls. The default is 3, which is 45 seconds at the default 15-second poll interval. A sample at or above the threshold resets the count. The check reads one sample per poll, and on GB200 under load the margin moves by tens of degrees within seconds. A single sample past the threshold is therefore usually a transient that the GPU's own hardware slowdown already handled. To restore the previous single-sample behaviour, set the value to 1.

NVLink BER error codes are suppressed by default (#1994, #1944)

DCGM 4.7.0 removed the NVLink BER health and diagnostic checks because they failed healthy GPUs. Older DCGM hostengines still report DCGM_FR_NVLINK_EFFECTIVE_BER_THRESHOLD and DCGM_FR_NVLINK_SYMBOL_BER_THRESHOLD. Both map to CONTACT_SUPPORT, so the default quarantine rules cordon the node. Both codes are now in the default gpu-health-monitor.dcgmHealthCheck.suppressedErrorCodes list. On DCGM 4.7.0 and later, this has no effect.

The new default does not reach every installation. helm upgrade --reuse-values keeps the list from your previous release, and a custom suppressedErrorCodes list replaces the default list. In both cases, add the two codes to your list.

Post-remediation validation (#1962, #1960)

These changes apply only when post-remediation validation is enabled (fault-quarantine.validation.enabled).

Quarantine taints stay on the node during validation. Fault-quarantine kept the cordon when it handed a node to validation, but it removed its quarantine taints, so a node with taint-only rule-sets accepted workloads while validation ran. It now keeps the taints too. After validation passes, lifecycle-manager lifts a taint only if lifecycle-manager.config.schedulingGate.taints lists it with remove: true. Any other taint stays on the node, and test pods do not tolerate it unless it is a NoExecute taint. Add every taint that your fault-quarantine rule-sets apply to that list, as described in Validation configuration. With the default rule-sets, this affects the certification-failure taint nvsentinel.dgxc.nvidia.com/nvcre-cert-failed. lifecycle-manager matches taints on key, value, and effect, so with remove: true it also lifts a matching taint that was on the node before the quarantine.

Partial drain must be configured for both components. Fault-quarantine now reads its own partial-drain setting, fault-quarantine.validation.partialDrainEnabled, which defaults to false. If you set node-drainer.partialDrainEnabled: true, fault-quarantine can count a partial drain as a full drain and start validation before the node is fully drained. Set global.partialDrainEnabled: true instead. It sets both components.

Janitor GPU reset with global.gpuDraEnabled (#1989, #1859)

GPUCluster (DRA) mode has no nvidia RuntimeClass, so a GPU reset Job that names one fails admission. When global.gpuDraEnabled is true, the janitor chart now refuses to render if janitor.config.controllers.gpuReset.resetJob.runtimeClassName is set. Its default is "nvidia". If you enable janitor and set global.gpuDraEnabled: true, set the following before you upgrade, or helm upgrade fails:

janitor:
  config:
    controllers:
      gpuReset:
        resetJob:
          runtimeClassName: ""

Deployment platform connector: idempotency key order and datastore pool (#1999, #1995)

These changes apply only when the deployment platform connector is enabled (platformConnector.deployment.enabled).

Publishers now mint the batch idempotency key as a time-ordered UUIDv7, and the connector stores it first in the key. Later keys sort later, so MongoDB appends to the unique index instead of writing across it. With the WiredTiger cache limited to 3.5 GB, reads on this index fell from 0.801 to 0.008 pages per insert. During a rolling upgrade, a batch that is retried across an old and a new connector replica gets its key in two different orders, so that one retry can be stored twice. Existing documents keep their old keys, and the unique index still applies to them.

The default platformConnector.deployment.datastore.maxPoolSize is now 30, up from 10. Each write holds a pooled connection until a majority of the replica set acknowledges it, so the pool size limits the connector's ingest rate. helm upgrade --reuse-values keeps your previous value; set the value to 30 to use the new default. On PostgreSQL, three replicas at 30 can open up to 90 connections, which is close to PostgreSQL's common default max_connections of 100.

Major New Features

GPU Operator GPUCluster (DRA) Mode Support (#1989, #1976, #1945, #1946)

This release adds GPU Operator GPUCluster (DRA) mode support to janitor GPU reset, lifecycle-manager, node-drainer, and preflight:

• Janitor GPU reset (#1989, #1859): before a reset, janitor stops the GPU Operator operands that hold device handles by setting their nvidia.com/gpu.deploy.* node labels to false. The operand list matched ClusterPolicy mode only. In GPUCluster mode, DCGM and its exporter run as nvidia-dcgm-dra and nvidia-dcgm-exporter-dra, with no device plugin or GFD, so janitor changed labels that nothing read and DCGM kept its handles. Janitor now stops only the operands that have a pod on the node, and records their labels in the new status.managedServices.nodeLabels field for restore and for the deletion finalizer. The deletion finalizer also no longer treats the pass that only initialises the restore status as a completed restore. See the Upgrade Notes for the reset Job's RuntimeClass.
• lifecycle-manager (#1976, #1899): the default gpu-allocatable readiness criterion read only the nvidia.com/gpu allocatable that the device plugin sets, which stays 0 in GPUCluster mode. Every ValidationRequest stayed Pending, and new-node validation never started. CEL criteria can now read the node's ResourceSlices as resourceSlices, and the default criterion also accepts a gpu.nvidia.com ResourceSlice with devices. lifecycle-manager also watches ResourceSlices, so a slice published after the last node update still wakes a pending request.
• node-drainer (#1945, #1943): node-drainer counted a pod as a GPU workload only when a container set nvidia.com/gpu or nvidia.com/pgpu limits. A DRA pod without the dgxc.nvidia.com/devices annotation, for example while metadata-collector was down, was skipped by a partial drain that then reported success while the workload stayed on the faulty GPU, and a drainGPUPods drain did not evict it. For such a pod, node-drainer now reads the referenced ResourceClaim or ResourceClaimTemplate and checks for a GPU device class. The lookup runs only for unannotated pods with resource claims, and the node-drainer ClusterRole gains get on resourceclaims and resourceclaimtemplates.
• Preflight (#1946, #1867): the preflight webhook found GPU pods only from nvidia.com/gpu, so pods that get GPUs only through DRA claims started with no preflight checks. The webhook now detects DRA GPU pods with no configuration, and reads claims only for pods with spec.resourceClaims. It copies all of the pod's claims into the preflight init containers, so the checks run on the same devices as the workload.

Operator Recovery for Health-Events-Analyzer Conditions (#1706, #1553)

An operator can now clear a health-events-analyzer condition after a verified repair, for configured rules. The operator annotates the node with the verification time and, optionally, an entity selector. The analyzer then publishes a matching healthy derived event through the normal ingestion path. A persisted verification boundary stops old history from recreating the condition after recovery or after an analyzer restart. The analyzer rechecks the annotations once a minute and reports results as node events. It gains read-only node access and permission to create events in the default name...

Release v1.26.0

This release makes the deployment platform connector (ADR-052) deployable as an opt-in alternative to the node-local socket, runs NVCRE tests as part of new-node and post-remediation validation, adds a standing bound on how many nodes NVSentinel keeps cordoned, and reduces the health-events-analyzer's per-event query cost. GPU Operator GPUCluster (DRA) mode support is added to metadata-collector. The MongoDB setup Jobs change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

MongoDB setup Jobs create the idempotency index; the external setup Job is no longer a Helm hook (#1927)

Both the bundled and the external MongoDB setup Jobs now create the unique idempotency index (healthevent_idempotency_key_unique), a partial index over events that carry an idempotency key. If existing documents already share a key, the Job prints an error with a query that lists them and completes the rest of the setup.

The external MongoDB setup Job (global.datastore.setupJob) is now a plain Job rather than a post-install,post-upgrade Helm hook, matching the bundled Job. A hook runs only once the release's pods are ready, while the deployment platform connector's replicas become ready only once this index exists, so the two would wait on each other. If you run external MongoDB with ArgoCD, the Job now syncs as a regular resource rather than at PostSync.

Major New Features

Deployment Platform Connector (#1927, #1934)

The platform connector can now run as a central Deployment that health monitors reach over the network with TLS and a projected ServiceAccount token, as an opt-in alternative to the node-local DaemonSet socket (ADR-052). The connector role, the publisher clients, and the datastore index landed in earlier releases; this release adds the Helm chart and documentation that make it deployable.

• platformConnector.deployment.enabled renders the Deployment on the system pool with a PodDisruptionBudget and anti-affinity, its Service, a cert-manager CA and leaf certificate, its ServiceAccount and RBAC, and a NetworkPolicy that admits only labelled health publishers.
• Every publisher — gpu-, syslog-, nic-, kubernetes-object-, slurm-drain-, csp- and nvcre-certification monitors, the health-events-analyzer, lifecycle-manager, and preflight — gets a publishTo switch: socket (the default) or deployment.
• The render refuses pairings that would lose events: a publisher on deployment while the Deployment is off, or on socket while the DaemonSet is off. Both enabled keys must be real YAML booleans; a quoted "false" is refused.

Nothing changes for an existing install: the Deployment is off, every publishTo is socket, and a --reuse-values upgrade keeps today's behaviour. Node condition writes on the deployment connector are retried in place until the best-effort deadline (#1934), so faults for one node that reach different replicas at once no longer lose the losing writes.

NVCRE Tests in New-Node and Post-Remediation Validation (#1922)

Adds an nvcre-provider test provider that runs NVIDIA Cluster Readiness Engine tests — nccl-loopback-nvswitch and nccl-all-reduce — as part of new-node and post-remediation validation, alongside the existing k8s-job-provider and its dcgm-diag-test. It is opt-in. Install NVCRE, enable lifecycle-manager and post-remediation validation in fault-quarantine (both off by default), and apply the overrides in values-nvcre.yaml.

Validation-State Node Label (#1947)

Nodes targeted by a ValidationRequest now carry a nvsentinel.nvidia.com/validation-state label, derived from the existing active-validation-request and validation-session annotations:

The validation-controller and node-validation-controller also log every state change they make.

Standing Bound on Cordoned Nodes (#1926, #1924)

Every existing circuit-breaker bound counts cordon events inside a window, so none of them limits how many nodes end up cordoned in total — a rate that stays under the threshold can cordon a whole fleet given enough windows. The new circuitBreaker.maxCordonedNodes is a level rather than a rate: the breaker trips when that many nodes are quarantined by NVSentinel at once. It counts only NVSentinel's own cordons, so GPU Operator upgrade flows do not consume the budget, and it latches like the existing bounds. It is off by default (0), and it satisfies the requirement that at least one bound is set.

Health-Events-Analyzer Rule Evaluation (#1913, #1912)

The analyzer evaluates 22 rules per health event, each one an aggregation query sent to the datastore one at a time. Two changes reduce that cost:

• Rules can be skipped when they cannot match (#1913): each rule takes an optional when CEL expression over the incoming event, and the analyzer runs the rule's query only when it is true. Most rules apply to one XID or one check, yet MongoDB previously read the node's events in the rule's window before the rule's own gate discarded them. 21 of the 22 default rules now carry a when that copies their existing gate; MultipleRemediations has no such gate and still runs for every event.
• Opt-in concurrent rule evaluation (#1912): --rule-concurrency sets how many rule queries run at once for a single event. It complements --workers, which cannot help when the backlog sits on one node because a node's events stay in order. The default is 1, which keeps the current behaviour.

Custom Drain Scoped by Node Label Selector (#1871, #1857)

customDrain.enabled=true previously routed every node through the drain plugin, and validation rejected userNamespaces or podDrainPolicies alongside it — so a cluster running a Slurm partition next to ordinary Kubernetes workloads could not express both. The new customDrain.nodeSelector takes a standard label selector: matching nodes use the drain plugin, and all other nodes keep the built-in eviction path. The mutual-exclusion checks apply only when the selector is unset, and an empty selector keeps today's behaviour.

[customDrain]
  enabled = true
  nodeSelector = "scheduler=slurm"

[[userNamespaces]]
  name = "*"
  mode = "AllowCompletion"

Configurable Kubelet Root Directory (#1951, #1936)

Kubelet creates its PodResources socket under --root-dir, but the chart mounted a fixed /var/lib/kubelet/pod-resources. On distributions that relocate kubelet, the mount was empty and metadata-collector crash-looped on every GPU node. The new global.kubeletRootDir (default /var/lib/kubelet) sets the host side of that mount. The value must be an absolute path, and a value ending in /pod-resources is refused at render time.

GPU Operator GPUCluster (DRA) Mode Support (#1923, #1930)

This release adds GPU Operator GPUCluster (DRA) mode support to metadata-collector:

• metadata-collector runs in GPUCluster mode (#1923): GPUCluster mode ships no Container Toolkit and no nvidia RuntimeClass, so the DaemonSet was rejected at admission. With global.gpuOperator.mode: gpucluster, the chart drops runtimeClassName and requests GPU access through a DRA ResourceClaimTemplate.
• DRA GPU pods are mapped for partial drain (#1930, #1865): the pod-to-GPU mapper read only device-plugin allocations, so no GPU pod on a GPUCluster node received its device annotation and every partial drain degraded to a full node drain. The mapper now also reads DRA allocations and resolves device names to GPU UUIDs locally through NVML.

Bug Fixes & Reliability

• GPU conditions written by older releases now clear on recovery (#1969, #1940): GPU node conditions written by older releases name the GPU only as GPU:<n>, while current recovery events carry GPU, PCI, and GPU_UUID. Matching still required the PCI token, so after an upgrade those conditions stayed True with no way to clear them except a manual status patch. A GPU-only compatibility pass now clears such an entry when the recovered GPU index and every other non-PCI entity match exactly.
• metadata-collector supports GPU Operator NRI plugin mode (#1949, #1948): with the NRI plugin enabled there is no nvidia RuntimeClass, so the default configuration failed admission and ...

Release v1.25.0

This release adds GPU Operator GPUCluster (DRA) mode support to specific modules (labeler and DCGM discovery), completes the MaintenanceRequest feature so it can be deployed, and continues the deployment platform connector work (ADR-052). Several publisher-side changes reach the existing socket path and its metrics today — please read the Upgrade Notes below.

Upgrade Notes

The deployment-platform-connector work refactored every health monitor onto a shared publishing client. The deployment mode itself is off and not yet wired into the Helm charts, so monitors keep publishing to the node-local socket — but three changes reach that existing path on upgrade.

Publisher metrics changed (#1900)

docs/METRICS.md gains a Health Event Publisher table (nvsentinel_health_events_publisher_*: sends, errors, skips, drops-by-reason, and retries, labelled by monitor). In the health-events-analyzer, the failure counter's per-attempt retryable_error and non_retryable_error values are removed — the shared client retries inside one call — and replaced by event_rejected and event_publishing_to_UDS_error. Update any dashboard or alert built on the old analyzer counter values.

A permanent rejection is no longer retried forever (#1900)

When the platform connector refuses a batch for good — an invalid event, or one naming a node the caller may not report on — the client now reports ErrPublishRejected and the monitor lets its state advance instead of requeuing the same event indefinitely. This applies on the socket path too, where such a rejection was previously retried or requeued. It removes a class of stuck-monitor loops, but it means an event the connector rejects is dropped rather than retried; the drop is counted.

PostgreSQL builds a new index on startup (#1896)

On PostgreSQL datastores, the table setup in fault-quarantine, fault-remediation, health-events-analyzer, and node-drainer now also creates the health-event idempotency index. It is built CONCURRENTLY so it does not block other components' inserts, and a setup lock serialises the four concurrent builds. No action is required; expect a one-time concurrent index build the first time each component starts on this version.

Major New Features

GPU Operator GPUCluster (DRA) Mode Support — In Progress (#1890, #1893, #1895)

GPU Operator GPUCluster (DRA) mode support is being added incrementally and is not yet complete. This release adds it to specific modules — the labeler's device-count classes and DCGM version detection, and DCGM service discovery — so those components work against both clusterpolicy (the default) and gpucluster (DRA) mode without per-mode source changes:

• Mode flag for labeler classes (#1890): documentation for using the gpu device-count class in clusterpolicy mode and the gpu-dra class in gpucluster mode.
• DCGM version detection in both modes (#1893, #1861): the labeler's DCGM app-label flag now accepts a comma-separated list (e.g. --dcgm-app-label=nvidia-dcgm,nvidia-dcgm-dra), so it finds the DCGM pod in either mode with no config change. The dcgmAppLabel, driverAppLabel, and gkeInstallerAppLabel args, previously hardcoded in Go, are now exposed as Helm values.
• DCGM service discovery in both modes (#1895, #1863): the gpu-health-monitor, the preflight dcgm-diag check, and the lifecycle-manager diag Job previously hardcoded nvidia-dcgm.gpu-operator.svc:5555. They now take a candidate list defaulting to the GPUCluster service name first and the ClusterPolicy name second, and use whichever resolves at runtime.

MaintenanceRequest Is Now Deployable (#1735)

Completes the MaintenanceRequest feature (controller and webhook landed in v1.22.0–v1.24.0) with the Helm chart that deploys it, gated behind controllers.maintenanceRequest.enabled (default false). The chart also configures the platform-connector authentication the feature requires: a MaintenanceRequest may name any node, but lifecycle-manager runs as a Deployment on one node, so platform-connector only accepts its cross-node health events when the publisher is on the cross-node allowlist and presents a ServiceAccount token minted for the configured audience. Without both halves the failure is silent — the MR is admitted, its HealthEventEmitted condition goes True, and the node never cordons.

Configurable PodMonitor Relabelings (#1916, #1915)

podMonitor.relabelings and podMonitor.metricRelabelings are now rendered on both the metrics and cleanup-metrics endpoints. Previously an operator needing a relabeling had to disable the chart's PodMonitor and maintain a private copy, which then silently stopped scraping any endpoint the chart added later. The motivating case: much of the DaemonSet-exported metric surface (including dcgm_health_active_events) carries no node identity, so a single target relabeling adds it for every metric at once rather than requiring a pod-to-node join at query time:

podMonitor:
  relabelings:
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: node

Deployment Platform Connector — Landing Incrementally (#1858, #1900, #1911, #1896)

ADR-052 introduces an optional central platform-connector Deployment that monitors reach over the network with TLS and a token, as an alternative to the node-local DaemonSet socket. This release lands the pieces: the connector's deployment role (PC_MODE=deployment) with token-auth, idempotent writes, and index-gated readiness (#1858); direct publishing from every Go monitor, the analyzer, and the Python gpu-health-monitor (#1900); direct publishing from preflight checks running in tenant pods (#1911); and creation of the idempotency index from the datastore setup (#1896). The deployment mode is not yet wired into the Helm charts and cannot be enabled through the supported path in this release — every monitor continues to publish to the node-local socket. The socket-path and metric changes that do apply now are covered under Upgrade Notes.

Metrics

• NVSwitch faults recorded in dcgm_health_active_events (#1910, #1902): the gauge tracked GPU faults but not NVSwitch faults. The NVSwitch branch now sets the gauge on detection and zeroes it on recovery, and — matching the trap fixed for GPUs in v1.24.0 — a full recovery clears every code the pre-reset cache entry named, rather than only an assumed one. No remediation semantics change; the events remain STORE_ONLY.
• Dropped Kubernetes writes labelled by health direction (#1917, #1881): k8s_platform_connector_dropped_writes_total gains an is_healthy label so a dropped clear and a dropped set are distinguishable. They are not symmetric — a dropped set is re-reported by the next change, but a dropped clear on a change-only monitor may never be re-reported — so a fleet dropping writes during planned restarts previously looked identical to one latching stale faults.

Bug Fixes & Reliability

• Cordon labels removed on manual uncordon (#1873, #1872): a manual kubectl uncordon removed only the nvsentinel-state label and left cordon-by, cordon-reason, and cordon-timestamp behind, so the node returned to service permanently labelled cordon-by=NVSentinel and anything attributing cordons from that label mis-attributed the node's next cordon. The manual path now removes the same three labels the automatic path does — but only when NVSentinel set cordon-by, since some users reuse that key.
• NPD policies treat an Unknown condition as unhealthy (#1891, #1876): the NPD KOM policies added in v1.24.0 matched status == "True", and because the Kubernetes Object Monitor treats any non-match as the healthy edge, an NPD probe that timed out or crashed (publishing status: "Unknown") on an already-cordoned node produced a false recovery event. The predicates now match status != "False", so an Unknown status is not read as recovery.

Performance

• Health-events-analyzer query optimization (#1901): the four XID 74 rules use a $lookup self-join whose sub-query reads only the node name and GPU UUID, but it re-ran once per matching document and re-read the node's full 24-hour window each time, so cost grew with the square of the node's event rate. The available index also included the entitiesimpacted arra...

Release v1.24.0

This release adds new-node validation, concurrent node-partitioned event processing in the health-events-analyzer, DCGM 4.7.0 support, and opt-in KOM policies for NPD GPU system-service conditions. It also removes preflight's Kubernetes API rate limits, which is a breaking change for some configurations — please read the Upgrade Notes below.

Upgrade Notes

Preflight Kubernetes API rate-limit settings removed (#1849, #1848)

Breaking change for some configurations.

Preflight's --kube-api-qps and --kube-api-burst flags are removed, along with their Helm values, schema entries, and documentation. Preflight now takes its REST config from ctrl.GetConfig() and uses controller-runtime's connection defaults.

The default of 5 QPS was actively harmful because preflight sits on the main path of workload creation. In a measured 1,028-pod gang, throttling left 51 of 1,028 peers registered with server-side admission P99 at 24,850 ms, and the gang never recovered — the coordination ConfigMap advertised a quorum that never formed, which is worse than registering nothing. Preflight drew 0.01 cores while stalled, waiting on its own limiter rather than the API server. Removing the limit rather than re-defaulting it avoids replacing one guessed number with another, since any fixed value must be chosen against the largest gang the product will admit.

What this means for you:

• Standard Helm deployments will not break. The chart no longer emits either flag, so the new behaviour reaches the deployment.
• If you set qps or burst in your preflight values, the schema removal surfaces it at install time — that is deliberate, so the setting is not silently ignored. Remove those keys from your values.
• If you run preflight outside Helm and pass either flag directly, the binary rejects the unknown flag and the pod enters CrashLoopBackOff with exit 2. Drop the flags from your invocation before upgrading.

Note that this removes rate-limit configurability for preflight only. The equivalent settings for fault-quarantine, node-drainer, and labeler are unaffected.

Percona version sets are now validated at render time (#1831)

mongodb-store gains a render-time guard that refuses version combinations that cannot work, following the same fail pattern as the existing oplog and auth validations. A render is refused when crVersion is more than one minor behind the operator, when major versions differ, when crVersion is ahead of the operator, or when initImage.tag differs from the operator tag (the init container runs the operator image, so a mismatch silently runs a different operator build). A new mongodb-store.psmdbVersion value lets you declare the intended operator version once and have all three checked against it.

One minor of skew is permitted deliberately, because that is Percona's documented upgrade path. If your deployment is currently in a refused state — most likely crVersion two minors behind the operator — helm upgrade will now fail rather than render. That is the intended behaviour: it surfaces a combination Percona does not support. Reconcile your Percona values before upgrading.

A companion opt-in check, mongodb-store.validateDeployedCrVersion (default false), extends this to the deployed resource (#1850). The render-time guard only compares chart values with each other, so it cannot see a live PerconaServerMongoDB still carrying the crVersion it was installed with — and lookup returns empty under helm template, leaving every ArgoCD user blind to it. When enabled, an init container on the bootstrap Job reads the deployed crVersion and applies the same rule.

Major New Features

New-Node Validation (#1821)

Adds the node-validation-controller, which batches nodes matching configured criteria and creates a single ValidationRequest covering the whole batch — so newly joined nodes are validated before they take work. Enabling it requires the validation-request-controller plus a spec.newNodeValidation block in the ValidationConfiguration:

newNodeValidation:
  condition: NewNodeValidated
  batchPeriodSeconds: 900
  criteria:
    - name: recently-joined
      expression: >-
        has(node.metadata.creationTimestamp) &&
        now() - timestamp(node.metadata.creationTimestamp) < duration("30m")
    - name: gpu-present
      expression: >-
        has(node.metadata.labels) && "nvidia.com/gpu.present" in node.metadata.labels
  newNodeTests:
    - dcgm-diag-test

Concurrent Node-Partitioned Event Processing (#1817, #1800)

The health-events-analyzer processed events serially at roughly 57 events/s. Events are now routed by node-name hash into dedicated FIFO worker channels, so events for the same node still process strictly in order while events for different nodes process concurrently. A low-water-mark tracker guarantees that out-of-order completion never checkpoints past an earlier unresolved event across restarts or crashes, and a MaxInFlight bound applies backpressure by pausing stream reads when worker queues or uncheckpointed events reach capacity. The design is recorded in ADR-056. The default remains workers: 1, so throughput is opt-in.

DCGM 4.7.0 Support (#1866, #1759)

Updates DCGM to 4.7.0 on Ubuntu 24.04 across the GPU health monitor, preflight diagnostics, and lifecycle diagnostics, with compatibility handling for older 4.x servers: thermal-margin and power-brake readings use a version-compatible API when client and server differ, and monitoring continues when an older server does not support the optional GPU serial-number request. Adds imexMonitoringEnabled (disabled by default), handling both the dedicated IMEX watch and older servers that report IMEX errors under NVLink. Recovery events are now sent for individual GPU and NVSwitch error codes rather than only when a whole watch clears. ConnectX monitoring is deliberately excluded until the GPU health monitor supports NIC discovery, identities, and health events.

Host-Native Authentication for metadata-collector (#1837, #1836)

metadata-collector can now authenticate as a host-native process while keeping hardware inventory and pod-to-GPU mapping intact, via --kubeconfig for the Kubernetes API and --kubelet-kubeconfig for kubelet HTTPS. Endpoints, credentials, and CA trust stay separate, and explicit configuration requires authenticated, verified HTTPS. In-cluster defaults are preserved and a home-directory kubeconfig is never loaded implicitly. Inventory and annotation schemas are unchanged. Recorded in ADR-058, which extends the deployment options in ADR-010 without changing publisher authentication.

Opt-In KOM Policies for NPD GPU System Services (#1385, #1382)

Ships the KOM-policy half of ADR-050 as an opt-in values overlay, values-npd-gpu-services.yaml, covering NPDFabricManagerDown and NPDFabricManagerFlapping (fatal, RESTART_BM), NPDFabricManagerNotInstalled (fatal, CONTACT_SUPPORT, because there is no unit to restart and a reboot will not install one), and NPDNvidiaPersistencedDown (non-fatal, CONTACT_SUPPORT). Each policy matches a condition published by the ADR-050 reference NPD configurations. The companion documentation PR adds the NPD plugin scripts, reference configs, and a tutorial.

Unified Platform-Connector Handler and On-Change Node Updates (#1845)

Part of ADR-052. The request handler and connector set become one code path, with Connector.ProcessBatch taking a single batch, Set fanning a batch to several connectors and failing on the first error, and BestEffort wrapping connectors whose failure or timeout must not fail the batch. Connectors and pipeline stages are sized and hardened for fleet scale. The wire contract is unchanged — same PlatformConnector gRPC service, same protobuf. Node updates are now written on change rather than unconditionally.

Configurable Node-Drainer Requeue Backoff (#1875, #1847)

The workqueue requeue backoff base was compiled in at 10 seconds, so for fast-terminating or zero-grace pods that floor dominated drain latency — 10.09 s P50 where the eviction work itself took milliseconds. The base is now a CLI flag and Helm value, keeping the 2-minute ceiling and the 10-second default. Operators with fast workloads can lower it while workloads with long grace periods keep the existing behaviour.

Configurable GPU Reset driverRoot (#1813)

gpu_reset.sh reaches nvidia-smi only through chroot "$DRIVER_ROOT", so that path must be a complete filesystem — binary, shared libraries, and dynamic loader. The container-side path was a Go constant, with only the host side configurable, which works against a GPU Operator driver container that bind-mounts its full root but fails against a driver root staging only driver surfaces, where the chroot fails before nvidia-smi starts. The new janitor.config.controllers.gpuReset.resetJob.driverRoot value makes the container-side path configurable too.

Metrics Add...

Release v1.23.0

This release adds post-remediation validation, a new NVCRE certification monitor, NPD check detection via KOM policies, a label-based janitor provider, and change-stream consumer lag metrics. It also fixes a GPU health monitor bug where suppressing one DCGM error code silently discarded every other incident on that GPU. Three changes alter observable behaviour — please read the Upgrade Notes below.

Upgrade Notes

No bundled datastore versions changed in this release: the Percona operator image, crVersion, mongod image, and init image are all unchanged from v1.22.0, so this release does not roll your replica set.

Three changes do alter behaviour operators may be calibrated on:

dcgm_health_check_suppressed_incidents now counts per incident (#1793)

The metric previously counted once per (watch, GPU) pair; it now counts once per suppressed incident. Anyone with dashboards or alerts calibrated on the old cadence will see a step change in this series after upgrading. This is a direct consequence of the suppression fix described under Bug Fixes — the old counting behaviour was tied to the bug.

metadata-collector now serves metrics on port 2112, bound on the host (#1780)

metadata-collector previously had no metrics surface at all. It now serves Prometheus metrics on --metrics-port, default 2112, matching every other NVSentinel component. metadata-collector is the only component that runs with hostNetwork: true, so unlike the others it binds this port on the node rather than inside a pod network namespace.

Check that host port 2112 is free on your nodes before upgrading. If it conflicts with something else in your environment, set --metrics-port to a free port or disable the endpoint. (On the fleet where this was developed, 2112 was unclaimed — node-exporter is 9100, cilium 9963/9964, vast-csi 9090/9091, etcd 2381 — but that will not hold everywhere.) The existing PodMonitor already selected these pods and matched a container port named metrics; it had simply never scraped them because no such port existed, so no PodMonitor change is needed.

OCI janitor provider now defaults to hard reset (#1819, #1818)

Node reboots issued through the OCI janitor provider now use a hard reset rather than a soft reset. Soft reset was observed in production not to clear faults that a hard reset does, and this matches OCI's own recommendation. The trade-off is an abrupt shutdown, which is acceptable here because NVSentinel has already cordoned and drained the node before the reboot is issued.

Major New Features

Post-Remediation Validation (#1749, #1427)

Fault-quarantine can now automatically create a ValidationRequest when a node is unquarantined, so a node is verified before it returns to service rather than being trusted implicitly. The request carries tests drawn from every event in the node's quarantine session that required validation, with Validation.RuleSets mapping a HealthEvent to the tests it should trigger:

ruleSets:
  - enabled: true
    version: "1"
    name: "Post-reboot validation"
    match:
      all:
        - kind: "HealthEvent"
          expression: >-
            event.componentClass == 'GPU' && event.isFatal == true &&
            event.recommendedAction in [15, 24]
    tests:
      - dcgm-diag-test

Events requiring validation are tracked for the session in a quarantineValidationHealthEvent annotation. This builds on the validation-controller shipped in v1.22.0.

NVCRE Certification Monitor (#1745)

A new health monitor for the NVIDIA Cluster Readiness Engine. It watches Certification CRs, reads the per-category failed and succeeded node lists, and publishes one health event per (node, variant, reason) so fault-quarantine can act on nodes that failed burn-in. A default 15-minute sweep diffs the desired set against an observed set held in a node annotation, so the monitor is stateless and restart-safe; a node that later passes the same variant recovers automatically, and ownership between overlapping certifications is first-come-first-served by the cert's terminal-condition time.

NPD Check Detection via KOM Policies (#1755, #1633)

Adds Kubernetes Object Monitor policies that surface node-problem-detector signals alongside NVSentinel's own checks. Because NPD reports through node conditions, KOM can watch them directly and emit fatal unhealthy events. Three NPD checks are covered: XfsShutdown, CperHardwareErrorFatal, and ReadonlyFilesystem.

Label-Based Janitor Provider (#1763, #1730)

A generic label CSP provider that requests node reboot and termination by labelling the Node, leaving an external controller (for example NKE) to watch those labels and perform the action. The node is treated as ready only once the reboot label is removed. Configured via janitor-provider.csp.provider: label with settable rebootKey, terminateKey, and value; selecting this provider grants the chart node patch/update permissions.

janitor-provider:
  csp:
    provider: "label"
    label:
      rebootKey: nke.nvidia.com/reboot
      terminateKey: nke.nvidia.com/terminate
      value: requested-by-nvsentinel

Change-Stream Consumer Lag Metrics (#1762, #1709)

Implements ADR-054. Four of the five change-stream consumers — event-exporter, health-events-analyzer, node-drainer, and fault-remediation — exposed nothing that distinguished "idle" from "stuck", which is why the stale-timestamp problem fixed in v1.22.0 was only diagnosed after the fact. Lag is computed as now - max(lastEmptyBatchAt, lastEventReadAt), using an empty batch as the stream telling that specific consumer it is caught up against its own filtered pipeline — which is what position age and head-minus-position both get wrong for filtered consumers, and those are precisely the ones most likely to sit idle for weeks. Two series are exposed, change_stream_lag_seconds{client} and change_stream_lag_known{client}; the lag series is deliberately absent rather than zero while unknown, so a consumer that never started does not read as healthy.

Pod-Label Drain Policies (#1751, #1689)

Workloads sharing a namespace can have different shutdown needs, so podDrainPolicies lets the drain mode be selected by pod labels rather than namespace alone — a worker can be evicted immediately while a training job beside it is allowed to finish. Policies are matched in order and the first match wins. Operators choose either userNamespaces or podDrainPolicies; a configuration setting both is rejected, and Helm users enabling policies clear the default namespace rule with userNamespaces: []. Unmatched pods stay outside the drain scope, and the rules apply equally on drain retries and node-drainer restarts.

DCGM Startup Gate and Connectivity Debouncing (#1756, #1765, #1642)

Two changes that stop the GPU health monitor reporting connectivity faults that are really startup or transient conditions:

• An optional init container waits for a functional remote DCGM endpoint before the monitor starts, requiring both a successful connection and supported GPU discovery. It is disabled by default, supports operator-service and external-hostengine modes, runs each attempt in a separate process under a hard timeout, retries indefinitely without publishing health events, and rejects embedded-mode where the main container starts DCGM itself. This prevents GpuDcgmConnectivityFailure events before DCGM has ever been available.
• GpuDcgmConnectivityFailure and its recovery are now debounced, requiring consecutive failures before publishing and consecutive successes before recovering (defaults: failureThreshold: 3, successThreshold: 1). The current failure and success streaks are exposed as Prometheus metrics.

Pod Mapper Failure Metrics for metadata-collector (#1780, #1767)

Adds metadata_collector_pod_mapper_failures_total (counter, for alerting on a rising failure rate) and metadata_collector_pod_mapper_consecutive_failures (gauge, the streak the exit threshold acts on, so an alert can fire before the container dies). The gauge is the more directly alertable of the two, being the exact quantity compared against --pod-mapper-max-consecutive-failures. See the Upgrade Notes above regarding the new host-bound metrics port.

Bug Fixes & Reliability

• Suppressing one DCGM error code no longer discards every other incident on that GPU (#1793, #1791): suppressedErrorCodes removed far more than intended. The accumulator kept only the first error code for each (watch, GPU) pair — a later incident contributed...

Release v1.22.0

This release upgrades the bundled Percona MongoDB database to realign it with the operator shipped in v1.21.0 (please read the Upgrade Notes below before upgrading if you are using the percona version of MongoDB). It also lands the validation-controller implementation, startup recovery of unresolved events in fault-quarantine, gating of externally-owned nodes, a bound on remediation attempts, and two large-scale performance wins (fault-remediation memory down from 8.5 GB to 17.6 MB on a 53k-node cluster, and the Kubernetes Object Monitor no longer OOM-killing at scale).

Upgrade Notes

MongoDB / Percona Database Upgrade (#1741, #1737)

Action may be required before upgrading. Review your current Percona version and follow upstream Percona guidance for the upgrade path.

Before upgrading NVSentinel, check the Percona version your cluster is currently running and step through intermediate minor versions as required rather than jumping straight to the target — see Percona's Upgrade the Operator documentation for the supported path and for database-upgrade considerations.

Major New Features

Performance at Scale (#1758, #1720)

Two components were holding far more cluster state than they needed:

• Fault-remediation no longer builds a cluster-wide Node informer (#1758, #1757): fault-remediation registers no Kubernetes watches, but it read Nodes through the manager's cached client — and controller-runtime creates informers lazily on the first cached read, so the first remediation silently built a cluster-wide Node informer. Nodes are only ever fetched by name here, so the cache bought nothing. Caching is now disabled for Nodes. Measured on a 53,513-node cluster: container working set 8.50 GB → 17.63 MB, live heap 4.46 GB → 4.77 MB, and first-remediation CR generation 43.02 s → 0.091 s.
• Kubernetes Object Monitor derives cache transforms from its CEL policies (#1720, #1718): every watched GVK was registered as unstructured.Unstructured with no transform, so a 53.6 KB node was cached whole even though the NodeNotReady predicate reads only status.conditions — and unstructured costs roughly 12 bytes of heap per byte of object. The reported result was 3.15 GB at 5k nodes, 21.85 GB at 30k, 58.9 GB at 90k, and 65 OOM-kills against a 32Gi limit. Because policies are configuration loaded at startup, the monitor now derives what to retain from the CEL expressions themselves, and resolves lookup() target GVKs ahead of time instead of calling the API server on every evaluation.

Validation-Controller Implementation (#1694)

Adds the core reconciliation logic to the lifecycle-manager's validation-controller (ADR-049). Reconciliation is driven by edge-based signals: ValidationRequest create/update/delete events, plus node events for nodes carrying the validation-session annotation, which fan out to each ValidationRequest named in that annotation.

Startup Recovery of Unresolved Events in Fault-Quarantine (#1693, #1674)

Fault-quarantine relies on its datastore resume position to continue processing health events; if that position is lost or expires, events already stored in MongoDB or PostgreSQL could remain unresolved indefinitely. Startup recovery now opens the live watcher before recovery so new events are not missed, replays unresolved events in creation order through the existing rule and circuit-breaker paths, suppresses obsolete events superseded by newer state, and marks skipped, invalid, and superseded events complete so later startups converge. The CREATE reset boundary is preserved, so deliberately abandoned backlogs are not revived. During recovery, node-dependent rules read live API-server state rather than the informer cache; transient failures stop recovery and retry on restart, while permanent per-event failures are recorded and skipped.

Gate Health Events for Externally-Owned Nodes (#1670)

Continues the ExternalRemediationRequest work (ADR-040) by preventing NVSentinel from acting on nodes handed off to an external owner. After enriching an event with node metadata, the MetadataAugmentor checks for nvsentinel.dgxc.nvidia.com/managed=false and sets the event's ProcessingStrategy to STORE_ONLY — blocking downstream remediation while preserving the audit trail in the datastore. The metadata cache TTL drops from one hour to 30 seconds so label changes take effect promptly, the deduplicator returns early for STORE_ONLY events so it cannot overwrite the strategy, and configurable skipNodeLabelKey/skipNodeLabelValue fields are validated symmetrically. MetadataAugmentor is now enabled by default in all Helm values files, and a startup warning fires if a skip label is configured while the augmentor is disabled.

Maximum Remediation Attempts per Equivalence Group (#1640, #1543)

Adds a configurable maxRemediationAttempts bound to stop infinite remediation loops. The attempt counter is tracked per equivalence group in the node annotation, so it persists across fault-remediation restarts and stays independent between groups, and it resets when the quarantine session ends. Default 0 means unlimited, preserving existing behavior.

Absolute Node Bound for the Circuit Breaker (#1713, #1686)

The fault-quarantine circuit breaker threshold could only be expressed as a percentage of GPU nodes. On a 288-node cluster an integer percentage means 1% is the floor (~3 nodes) and each step adds another 3, so "never quarantine more than 5 nodes in a window" was inexpressible — and a percentage silently tracks fleet growth, which is the wrong behavior for a control whose job is bounding blast radius. An optional absolute maxNodes bound now sits alongside the percentage; when both are set the lower one binds. maxNodes: 0 disables it, leaving existing configs unchanged. The initializer now rejects a config with neither bound set, which previously computed a threshold of 0 and tripped on the first cordon — a silent halt rather than a safety control.

Debounce GPU Health Check Incidents (#1715, #1701)

DCGM_FR_NVLINK_DOWN was published from a single poll observation, but on SXM systems NVLink links are briefly down after every boot while they train — so a routine maintenance reboot produced a FATAL event carrying RESTART_VM, and with remediation enabled that maps to a RebootNode. Measured on a 288-node GB200 NVL72 fleet in detection-only mode, 214 nodes emitted the code over three and a half weeks with no actual NVLink fault: the median span from first to last event within a node-day was 0.0 minutes, affected nodes arrived in whole racks of 18 matching the planned reboot schedule, and the eight days after the last reboot saw zero occurrences while the same watch correctly reported 30 genuine BER events. Health check incidents are now debounced across consecutive polls so single-poll transients no longer escalate.

Health Events as a Prometheus Counter (#1723, #1687)

recommendedAction reached the datastore and the CloudEvents payload but was never exported to metrics, so answering "what is actionable right now" required a datastore query. A new Prometheus connector exposes health events as a counter. Context from a 288-node GB200 fleet over 7 days: of ~190k events/day, 99.1% carry NONE — 797,965 unhealthy NONE and 529,081 healthy NONE, against 12,031 CONTACT_SUPPORT, 91 COMPONENT_RESET, 9 RUN_DCGMEUD, and 7 RESTART_VM.

CEL Filter for Event Exporter (#1724, #1702)

event-exporter published every HealthEvent from the change stream, so delivering the ~200 actionable events in a three-day window meant the sink had to accept roughly 570,000. An optional CEL filter now selects which events reach the sink. Exporting everything remains the default — filtering at the destination is still the right model for a data lake — but a source-side filter earns its place when the sink is a notification path.

Drain Scope and Partial-Drain Entity Metrics (#1714, #1685)

A partial drain was invisible in metrics: node_drainer_events_processed_total carried only drain_status and node, so an operator seeing reduced capacity could not tell whether one GPU or the whole node had been drained, nor which GPU, without querying the datastore or grepping logs. That metric now carries a drain_scope label of full or partial (two extra label values, no unbounded cardinality). A separate opt-in node_drainer_partial_drains_total{node, entity_type, entity_value} metric exposes the targeted entity; it is registered only when partialDrainEntityMetricEnabled is s...