Problem
When AICR generates a static OCI bundle, GPU node tolerations are baked in at build time via --accelerated-node-toleration. Once the image is built there is no mechanism to inject additional tolerations at install time without rebuilding the bundle.
This becomes a blocker when customers add arbitrary taints to their GPU nodes (e.g., reserved-by:NoSchedule during application upgrades). The GPU Operator daemonsets — driver, toolkit, device-plugin, DCGM, GFD, etc. — cannot schedule on those nodes, causing driver upgrades to stall.
Request
Extend the dynamic value mechanism (--dynamic component:path) to support node scheduling toleration paths, so that daemonsets.tolerations (and equivalent paths registered under nodeScheduling.accelerated.tolerationPaths in recipes/registry.yaml) can be left as HelmRelease dynamic placeholders at bundle time and filled at install/deploy time.
Proposed change
pkg/bundler/bundler.go:applyNodeSchedulingOverrides() — when a toleration path is marked as dynamic, emit a placeholder in the HelmRelease rather than baking in the toleration values- New flag or extension to existing
--dynamic to mark accelerated/system node toleration paths for install-time injection
- At install time: the operator passes the additional taint(s) as a HelmRelease value override
Context
This was surfaced from a customer issue on an mk8s-managed cluster (vclpksojdd-dgxc-k8s-aws-use1-prod) where reserved-by:NoSchedule was added to GPU nodes. The platform needs a consistent way to handle arbitrary customer taints across both AOR-managed (mk8s) and AICR-managed clusters.
Wildcard tolerations (operator: Exists with no key) are not an option because they bypass the Skyhook node hardening gate (skyhook.nvidia.com=runtime-required:NoSchedule, sync-wave 019).
Problem
When AICR generates a static OCI bundle, GPU node tolerations are baked in at build time via
--accelerated-node-toleration. Once the image is built there is no mechanism to inject additional tolerations at install time without rebuilding the bundle.This becomes a blocker when customers add arbitrary taints to their GPU nodes (e.g.,
reserved-by:NoScheduleduring application upgrades). The GPU Operator daemonsets — driver, toolkit, device-plugin, DCGM, GFD, etc. — cannot schedule on those nodes, causing driver upgrades to stall.Request
Extend the dynamic value mechanism (
--dynamic component:path) to support node scheduling toleration paths, so thatdaemonsets.tolerations(and equivalent paths registered undernodeScheduling.accelerated.tolerationPathsinrecipes/registry.yaml) can be left as HelmRelease dynamic placeholders at bundle time and filled at install/deploy time.Proposed change
pkg/bundler/bundler.go:applyNodeSchedulingOverrides()— when a toleration path is marked as dynamic, emit a placeholder in the HelmRelease rather than baking in the toleration values--dynamicto mark accelerated/system node toleration paths for install-time injectionContext
This was surfaced from a customer issue on an mk8s-managed cluster (
vclpksojdd-dgxc-k8s-aws-use1-prod) wherereserved-by:NoSchedulewas added to GPU nodes. The platform needs a consistent way to handle arbitrary customer taints across both AOR-managed (mk8s) and AICR-managed clusters.Wildcard tolerations (
operator: Existswith no key) are not an option because they bypass the Skyhook node hardening gate (skyhook.nvidia.com=runtime-required:NoSchedule, sync-wave 019).