Skip to content

feat(gpu): name GPU classes in vm-instance and kubernetes-nodes - #5030

Open
Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
feat/gpudevice-controller-prfrom
feat/gpudevice-webhooks-pr
Open

Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
feat/gpudevice-controller-prfrom
feat/gpudevice-webhooks-pr

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Builds on the GPU design approved in cozystack/community#99 at 30da976.

What this PR does

VMs and worker pools can name a GPU class instead of a device-plugin resource name. This adds the charts and admission webhooks for step 1, on top of the GPUClass API, tenant budgets and class status controller.

  • Both charts accept gpus: [{className, count}] beside the existing {name} form. A list holds one form only. Class entries fail at render where the cluster does not serve GPUs to VMs, and a Talos worker pool cannot name a vGPU class.
  • The VMI webhook resolves each class entry to one KubeVirt GPU entry per unit. Capacity is left to scheduling, so a VM waits for a free unit instead of being refused at admission.
  • A licensed vGPU class gets a Secret-backed disk labelled GPULICENSE. The launcher pod webhook copies the classes' tolerations onto its pods. Both webhooks fail closed.
  • The gpuclass option source returns class-level availability and the remaining budget from the same pool calculation admission uses. It exposes no node identities. The hoster's gpu.resourceNames list supplies GPU names that neither a class nor the KubeVirt table names.

Admission accepts at most 1024 GPU units per VMI across its class entries. The charts and Application admission enforce the same limit before a VMI is created. This bounds webhook memory use; it is separate from a class's maxCount, available capacity and tenant budgets.

Existing raw name entries keep working. GPU claims, holds, scheduling gates and policies that forbid raw names belong to step 2 and are not part of this PR.

Screenshots

Not applicable: this changes chart values and option data, not the UI.

Downstream repositories

Both follow-ups are tracked in the issues linked above; implementation pending. The Terraform provider needs className and count in its VMInstance and KubernetesNodes GPU schemas and expand/flatten code. The website needs the class-entry and guest-license-disk instructions, plus gpu.resourceNames in its platform configuration reference.

Release note

feat(gpu): name GPU classes in vm-instance and kubernetes-nodes, resolve them through admission webhooks, deliver licensed vGPU classes' license disks, and expose class availability and remaining budgets through the gpuclass option source.

@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files labels Oct 21, 2026
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

Malformed class annotations must fail rather than silently boot a VM
without its requested GPUs. Admission also needs to distinguish GPUs
from unrelated host devices when refusing additional requests beside
class entries.

No single resource-name source identifies every GPU: a pod-only resource
may appear only in the hoster's list, while KubeVirt's permitted-device
table can also contain NICs. Sharing these interpretations prevents the
consumers from accepting different annotations or treating a NIC as a
GPU.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
KubeVirt derives launcher requests from the admitted VMI and does not
allow another controller to change that spec afterwards. Class
resolution and offer validation must therefore finish at VMI creation. A
resource outside permittedHostDevices otherwise leaves a VMI Pending
without a pod. Resolving from uncached reads avoids admitting against an
obsolete class or entitlement; asynchronous class status cannot
establish resource-name uniqueness.

Capacity shortages belong to the scheduler: refusing admission would put
a VM into KubeVirt's retry backoff instead of starting it when a unit
frees up. Tenant-controlled per-unit expansion still needs a fixed
safety bound to prevent a small annotation from exhausting controller
memory.

A guest needs its class's license without embedding the token in an
image or changing which disk boots. Class tolerations must survive
LiveUpdate, which resets VMI tolerations from the VM template; the
launcher pod is the stable place to apply them. Migration and volume-
binding pods need the same scheduling access.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
A replica must not receive fail-closed admission traffic until its
webhook server serves. Otherwise startup and rollout can block GPU VM
creation.

GPU resource names requested only by pods cannot always be inferred from
a class or the KubeVirt permitted-device table. The hoster needs a way
to identify them for admission.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Fail-closed GPU admission must remain available during controller
rollouts and node disruptions. Replicas sharing one node would fail
together, and an unready replica must not receive admission traffic.

The availability dependency must be limited to VMs using class entries.
Other VMs and pods should not wait for a GPU webhook they do not need.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
A pod-only GPU resource may have no GPUClass or KubeVirt table entry.
The hoster needs to identify those resources explicitly so admission can
detect requests that bypass class entries. Commas in a name would
corrupt the controller's comma-separated flag; native resource names
would misclassify CPU or memory as GPUs.

A class reference is unusable where no GPU webhook will resolve it.
Charts need that cluster capability to refuse such inputs rather than
boot a VM without GPUs. Keeping the capability key absent while disabled
avoids unnecessary upgrades through the shared tenant values channel.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
A tenant choosing an offer needs its availability and remaining budget
before submitting a VM, rather than discovering them through a refused
start. Reusing admission's pool calculation keeps the budget estimate
consistent with writes.

The catalog is visible to every authenticated user. Class-wide figures
provide the information needed to size an entry without disclosing node
identity or per-node inventory. An unentitled tenant cannot use a vGPU
offer even when its budget has room.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Every KubernetesNodes pool runs Talos workers, which carry no vGPU
guest driver, so a pool that names a vGPU class would start workers
that never use their GPUs. The write is refused while it is made,
before any machine exists. A class must exist before
the pool is stored, so a later class cannot change the mode the pool
was checked against.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Tenants need to select an offer independently of the hoster's device-
plugin naming. Class references preserve that distinction when the
backend resource name changes.

Mixing raw GPU entries with class references would create two sources of
GPU requests on one VMI. A homogeneous list avoids that ambiguity. A
cluster without the resolving webhook must refuse references rather than
silently boot a VM without its requested GPUs.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Worker pools need the same GPU offers as individual VMs, with the
requested count applying to each worker. Existing pools on raw names
must retain their rendered configuration: a changed template hash would
unnecessarily replace their workers.

Talos needs the vendor's modules before a worker can use its devices.
The class's immutable vendor is suitable for this choice; a mutable
device-plugin resource name is not a reliable vendor identifier. An
unresolved class cannot provide a complete worker configuration.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant