Skip to content

feat(gpu): compute GPUClass status and copy vGPU licenses - #5013

Open
Aleksei Sviridkin (lexfrei) wants to merge 5 commits into
feat/gpudevice-quotas-prfrom
feat/gpudevice-controller-pr
Open

feat(gpu): compute GPUClass status and copy vGPU licenses#5013
Aleksei Sviridkin (lexfrei) wants to merge 5 commits into
feat/gpudevice-quotas-prfrom
feat/gpudevice-controller-pr

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Builds on the GPU design approved in cozystack/community#99 at 30da976.

What this PR does

GPU classes now show how many units are free, and a vGPU class's license reaches the tenants that may use it. This is the controller part of step 1 of the design, in cozystack-controller:

  • Class status. free and maxFreePerNode come from node allocatable minus pod requests. Only nodes that are Ready, not cordoned and without a taint the class does not tolerate count (decision 0015). A class gets ResourceNameConflict when an older class has its resource name, and ResourceNameNotPermitted when KubeVirt's permittedHostDevices leaves the name out. Allocatable and requested units per class go to the metrics cozystack_gpu_class_capacity_units and cozystack_gpu_class_used_units. The chart does not serve the metrics endpoint yet. That comes in a follow-up.
  • License copies (decision 0018). The hoster's Secret named by the class's license is copied as gpuclass-<class>-license into every namespace labelled namespace.cozystack.io/vgpu: "true". A copy holds the token and a gridd.conf with the class's FeatureType. It is deleted when the namespace loses the label, or the class or its license goes away. While the hoster's Secret is missing or has no token, existing copies stay and the class shows LicenseSecretMissing.
  • Controller wiring. Both controllers run behind --enable-gpu. With the flag off, the copier deletes every copy and clears the computed status. No RBAC rule is added.
  • Platform wiring. The platform sets the flag when gpu-operator serves GPUs to VMs, so with its default and vgpu variants.

Disabling GPU-for-VMs deletes the managed license copies even if Helm retains the GPU operator Package through helm.sh/resource-policy: keep. Existing VMs must stop using the platform license before disabling it, as required by the design's rollback procedure. A retained Package does not keep the license copies available for those VMs.

The earlier version of this PR had the binder, the claim lifecycle and the GPU node agent. They are step 2 of the design now, or dropped in the case of GPUNode (decision 0015).

This needs the GPUClass API from #4998 and the vGPU entitlement label from #5008. The VMI webhook that mounts the license copy into VMs is in #5030.

Screenshots

Not applicable: no UI change.

Downstream repositories

The diff adds a controller, a cozystack-controller chart value and a platform helper. It changes no app chart, no key in the platform values.yaml, no CRD and no telemetry metric. The GPU user guide will come with the PR that makes class entries usable.

Release note

feat(gpu): cozystack-controller computes GPUClass free capacity and copies vGPU license Secrets into entitled namespaces when GPU-for-VMs is enabled. Disabling it deletes managed license copies and clears computed status, even if the GPU operator Package is retained. Existing VMs must stop using the platform license before disabling GPU-for-VMs.

@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature labels Oct 21, 2026
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@github-actions github-actions Bot added the size/XXL This PR changes 1000+ lines, ignoring generated files label Oct 21, 2026
@lexfrei Aleksei Sviridkin (lexfrei) mentioned this pull request Oct 21, 2026
Open
12 tasks
@lexfrei Aleksei Sviridkin (lexfrei) mentioned this pull request Oct 26, 2026
Closed
12 tasks
@lexfrei Aleksei Sviridkin (lexfrei) mentioned this pull request Nov 3, 2026
Open
1 of 12 tasks
@lexfrei
feat(api): check GPU budgets when a VM or a worker pool is written
02d1264 
A tenant's GPU budget is declared per class as
<class>.gpuclass.cozystack.io/devices and over every class as
gpuclass.cozystack.io/devices. These keys are not pod resources, so no
ResourceQuota enforces them. The proposal hands the check to the
pool-wide admission gate that counts declared sizes; until that gate
is on, the Cozystack API server does it for GPU keys only, with the
inputs and the pool arithmetic that gate reads, so the switch changes
no verdict.

On a VMInstance or KubernetesNodes create or update, every class entry
reserves its count, times maxReplicas for a worker pool, read from the
chart-defaulted values. The tenant pool arithmetic is applied to the
declared resourceQuotas and these entries. Only growth is checked on
an update, so shrinking and unchanged entries pass even in a pool that
is already over budget. Like the gate it stands in for, this is a
read-check-write without a transaction.

The tenant declaration gate already treats GPU keys like any other;
it now reads their pool usage from the class entries, since the
ResourceQuota it reads for other keys never carries them. It also
refuses a key in the GPU budget domain that is not a budget key: the
tenant chart keeps such a key out of the ResourceQuota, so nothing
would enforce it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei
feat(gpu): compute GPUClass free capacity from node allocatable
02df4af 
A tenant picking a GPU class needs to know whether a VM of it is
likely to start, and a hoster needs to see a class that can never
serve a VM before a tenant's start is refused. With no GPUNode and no
binder in step 1, both figures come from the Node and Pod objects
alone: free units are a node's allocatable count of the class's
resource name minus what its pods request, over nodes that are Ready,
not cordoned and carry no taint the class does not tolerate.

One pass covers every class, because a class's ResourceNameConflict
depends on the classes created before it. The KubeVirt CR is read on
each pass and on a one-minute recheck instead of watched, so the
controller needs no KubeVirt kinds served to start.

The class status is readable by every authenticated user, so the
inventory and load behind it, the allocatable and requested totals
per class, go to metrics instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei
feat(gpu): copy vGPU license Secrets into entitled namespaces
5fa3133 
A VM of a licensed vGPU class mounts its license from a Secret in its
own namespace, so the token reaches entitled tenants only and is
rotated by a VM restart rather than by rebuilding a shared image. The
copier keeps one copy per class in every namespace whose vGPU label
is "true", with the token and a gridd.conf carrying the class's
FeatureType, and deletes copies the label, the class or its license
no longer call for.

A missing hoster Secret, or one without the token, keeps the copies
already written: deleting them would stop every restarting VM over
what is usually a rotation in progress. New copies wait, and the
class shows LicenseSecretMissing so the hoster sees why.

Only Secrets carrying both the copy's name and the class label are
ever written or deleted, so a tenant's own Secret of the same name is
left alone. The token is read through the uncached reader and stays
out of every cache.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei
feat(cozystack-controller): run the GPU class controllers
620a4bd 
The class status and the license copier run in cozystack-controller
behind --enable-gpu, which the chart passes from gpu.enabled. The
status controller is off without the flag, since it caches every
Node. The copier always runs: without the flag it deletes every
copy, so a token does not stay in tenant namespaces after GPU
features are switched off, and clears the free counts and conditions
the stopped status controller would otherwise leave stale.

License Secrets are watched on the metadata-only Secret cache the CA
projection already owns. The manager's own Secret cache is scoped by
label to wildcard replicas, and a watch on it would never see a
hoster's Secret. The envtest suite drives both controllers through
their watches, since a mis-scoped watch fails without an error.

No RBAC rule is added: the controller already reads every kind, holds
every verb on cozystack.io and the cluster-wide Secret write.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei
feat(platform): enable the GPU class controllers with VM GPUs
2b61498 
GPUClass status and license copies matter only where gpu-operator
hands GPUs to VirtualMachines, which is its default and vgpu
variants in the iaas bundle. The platform now passes gpu.enabled to
cozystack-controller there, so a hoster enabling VM GPUs gets the
class controllers without editing the generated Package, and a
cluster on the container variant or without gpu-operator keeps them
off.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei Aleksei Sviridkin (lexfrei) changed the title feat(gpu): add the GPU controller and the GPU node agent feat(gpu): compute GPUClass status and copy vGPU licenses Nov 3, 2026
@lexfrei Aleksei Sviridkin (lexfrei) mentioned this pull request Nov 3, 2026
Draft
12 tasks
@lexfrei Aleksei Sviridkin (lexfrei) mentioned this pull request Nov 3, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kvaps Andrei Kvapil (kvaps) Awaiting requested review from kvaps kvaps is a code owner
@lllamnyp Timofei Larkin (lllamnyp) Awaiting requested review from lllamnyp lllamnyp is a code owner
@myasnikovdaniil myasnikovdaniil Awaiting requested review from myasnikovdaniil myasnikovdaniil is a code owner
@IvanHunters IvanHunters Awaiting requested review from IvanHunters IvanHunters is a code owner
@sircthulhu Kirill Ilin (sircthulhu) Awaiting requested review from sircthulhu sircthulhu is a code owner
@mattia-eleuteri mattia-eleuteri Awaiting requested review from mattia-eleuteri mattia-eleuteri is a code owner
@kingdonb Kingdon Barrett (kingdonb) Awaiting requested review from kingdonb kingdonb is a code owner
@tym83 Timur Tukaev (tym83) Awaiting requested review from tym83
@androndo Andrey Kolkov (androndo) Awaiting requested review from androndo
@matthieu-robin Matthieu ROBIN (matthieu-robin) Awaiting requested review from matthieu-robin matthieu-robin is a code owner
@scooby87 scooby87 Awaiting requested review from scooby87 scooby87 is a code owner

Assignees

@lexfrei Aleksei Sviridkin (lexfrei)

Labels

area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lexfrei