Repository navigation
feat(gpu): name GPU classes in vm-instance and kubernetes-nodes - #5030
Open
Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
Open
Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
Conversation
Aleksei Sviridkin (lexfrei)
requested review from
Kingdon Barrett (kingdonb) and
Kirill Ilin (sircthulhu)
as code owners
October 21, 2026 10:47
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/gpudevice-controller-pr
branch
from
October 26, 2026 08:46
ccaef2f to
ebca80f
Compare
Aleksei Sviridkin (lexfrei)
requested review from
Andrey Kolkov (androndo) and
Matthieu ROBIN (matthieu-robin)
as code owners
October 26, 2026 08:46
1 of 12 tasks
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/gpudevice-webhooks-pr
branch
from
October 26, 2026 10:47
0c18c86 to
a283579
Compare
12 tasks
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/gpudevice-webhooks-pr
branch
from
October 27, 2026 07:35
a283579 to
57f709d
Compare
This was referenced Nov 3, 2026
Malformed class annotations must fail rather than silently boot a VM without its requested GPUs. Admission also needs to distinguish GPUs from unrelated host devices when refusing additional requests beside class entries. No single resource-name source identifies every GPU: a pod-only resource may appear only in the hoster's list, while KubeVirt's permitted-device table can also contain NICs. Sharing these interpretations prevents the consumers from accepting different annotations or treating a NIC as a GPU. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
KubeVirt derives launcher requests from the admitted VMI and does not allow another controller to change that spec afterwards. Class resolution and offer validation must therefore finish at VMI creation. A resource outside permittedHostDevices otherwise leaves a VMI Pending without a pod. Resolving from uncached reads avoids admitting against an obsolete class or entitlement; asynchronous class status cannot establish resource-name uniqueness. Capacity shortages belong to the scheduler: refusing admission would put a VM into KubeVirt's retry backoff instead of starting it when a unit frees up. Tenant-controlled per-unit expansion still needs a fixed safety bound to prevent a small annotation from exhausting controller memory. A guest needs its class's license without embedding the token in an image or changing which disk boots. Class tolerations must survive LiveUpdate, which resets VMI tolerations from the VM template; the launcher pod is the stable place to apply them. Migration and volume- binding pods need the same scheduling access. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
A replica must not receive fail-closed admission traffic until its webhook server serves. Otherwise startup and rollout can block GPU VM creation. GPU resource names requested only by pods cannot always be inferred from a class or the KubeVirt permitted-device table. The hoster needs a way to identify them for admission. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Fail-closed GPU admission must remain available during controller rollouts and node disruptions. Replicas sharing one node would fail together, and an unready replica must not receive admission traffic. The availability dependency must be limited to VMs using class entries. Other VMs and pods should not wait for a GPU webhook they do not need. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
A pod-only GPU resource may have no GPUClass or KubeVirt table entry. The hoster needs to identify those resources explicitly so admission can detect requests that bypass class entries. Commas in a name would corrupt the controller's comma-separated flag; native resource names would misclassify CPU or memory as GPUs. A class reference is unusable where no GPU webhook will resolve it. Charts need that cluster capability to refuse such inputs rather than boot a VM without GPUs. Keeping the capability key absent while disabled avoids unnecessary upgrades through the shared tenant values channel. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
A tenant choosing an offer needs its availability and remaining budget before submitting a VM, rather than discovering them through a refused start. Reusing admission's pool calculation keeps the budget estimate consistent with writes. The catalog is visible to every authenticated user. Class-wide figures provide the information needed to size an entry without disclosing node identity or per-node inventory. An unentitled tenant cannot use a vGPU offer even when its budget has room. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Every KubernetesNodes pool runs Talos workers, which carry no vGPU guest driver, so a pool that names a vGPU class would start workers that never use their GPUs. The write is refused while it is made, before any machine exists. A class must exist before the pool is stored, so a later class cannot change the mode the pool was checked against. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Tenants need to select an offer independently of the hoster's device- plugin naming. Class references preserve that distinction when the backend resource name changes. Mixing raw GPU entries with class references would create two sources of GPU requests on one VMI. A homogeneous list avoids that ambiguity. A cluster without the resolving webhook must refuse references rather than silently boot a VM without its requested GPUs. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Worker pools need the same GPU offers as individual VMs, with the requested count applying to each worker. Existing pools on raw names must retain their rendered configuration: a changed template hash would unnecessarily replace their workers. Talos needs the vendor's modules before a worker can use its devices. The class's immutable vendor is suitable for this choice; a mutable device-plugin resource name is not a reliable vendor identifier. An unresolved class cannot provide a complete worker configuration. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/gpudevice-controller-pr
branch
from
November 3, 2026 12:08
ebca80f to
2b61498
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/gpudevice-webhooks-pr
branch
from
November 3, 2026 12:08
57f709d to
74847c8
Compare
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the GPU design approved in cozystack/community#99 at
30da976.What this PR does
VMs and worker pools can name a GPU class instead of a device-plugin resource name. This adds the charts and admission webhooks for step 1, on top of the GPUClass API, tenant budgets and class status controller.
gpus: [{className, count}]beside the existing{name}form. A list holds one form only. Class entries fail at render where the cluster does not serve GPUs to VMs, and a Talos worker pool cannot name a vGPU class.GPULICENSE. The launcher pod webhook copies the classes' tolerations onto its pods. Both webhooks fail closed.gpuclassoption source returns class-level availability and the remaining budget from the same pool calculation admission uses. It exposes no node identities. The hoster'sgpu.resourceNameslist supplies GPU names that neither a class nor the KubeVirt table names.Admission accepts at most 1024 GPU units per VMI across its class entries. The charts and Application admission enforce the same limit before a VMI is created. This bounds webhook memory use; it is separate from a class's
maxCount, available capacity and tenant budgets.Existing raw
nameentries keep working. GPU claims, holds, scheduling gates and policies that forbid raw names belong to step 2 and are not part of this PR.Screenshots
Not applicable: this changes chart values and option data, not the UI.
Downstream repositories
Both follow-ups are tracked in the issues linked above; implementation pending. The Terraform provider needs
classNameandcountin its VMInstance and KubernetesNodes GPU schemas and expand/flatten code. The website needs the class-entry and guest-license-disk instructions, plusgpu.resourceNamesin its platform configuration reference.Release note