Skip to content

feat(tenant): add per-class GPU budgets and the vGPU entitlement - #5008

Open
Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
feat/gpudevice-api-prfrom
feat/gpudevice-quotas-pr
Open

Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
feat/gpudevice-api-prfrom
feat/gpudevice-quotas-pr

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Builds on the GPU design approved in cozystack/community#99 at 30da976.

What this PR does

This adds per-class GPU budgets and the vGPU entitlement for tenants. It builds on #4998.

A tenant declares budgets under <class>.gpuclass.cozystack.io/devices or gpuclass.cozystack.io/devices. These keys stay out of ResourceQuota. A tenant with only GPU budgets gets no ResourceQuota or LimitRange.

The API server checks the pool's declared class entries when a VM or worker pool is written. A VM reserves count; a worker pool reserves count times maxReplicas. Only growth is checked on update. This is the interim check until the pool-wide gate from community#48 is on. Tenant writes count the same entries when allocating GPU budgets to children.

GPU-only declarations leave ordinary-resource usage in the parent's runtime pool. The namespace helper no longer depends on the calling application's release prefix.

The entitlement is effective only when the parent is entitled too. Tenant.spec.gpu.vgpu grants it, and the effective value goes on the namespace as namespace.cozystack.io/vgpu. The root takes its value from the platform's gpu.vgpu.

Every authenticated user can read GPUClass through cozy:gpuclasses:view.

Class entries in the app charts follow in #5030. The separate usage-key fix in #5184 conflicts on one line in quota.go; whichever merges second needs a rebase.

Screenshots

Not applicable: no UI change.

Downstream repositories

Follow-ups for the website's platform values table and the provider's Tenant schema to add gpu.vgpu are tracked in the issues linked above; implementation pending.

Release note

feat(tenant): add per-class GPU budgets checked on VM and worker-pool writes, and the vGPU entitlement through gpu.vgpu on Tenant and the platform. GPU-only declarations do not change ordinary-resource pool membership.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@github-actions github-actions Bot added area/tenant Issues or PRs related to the tenant chart and multi-tenancy kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files labels Oct 21, 2026
A hoster publishes each GPU offer as a cluster-scoped GPUClass: one
resource name per offer, the tolerations its VMs need on the class's
nodes, and for vGPU classes the license Secret and FeatureType the
guests get. VMs and worker pools name a class instead of a raw
device-plugin name, so the offer can be budgeted per tenant and shown
in a catalog with its free units.

The kind joins cozystack.io instead of a group of its own: a CRD
cannot move between groups in place, and nothing in the design needs
separate versioning or serving. The class name becomes part of the
budget key <class>.gpuclass.cozystack.io/devices, so it is a DNS
label, and a resource name in that domain is refused.

Vendor, model, mode, profile, memory and the source kind never
change, so a class name stands for one offer for its whole life and
billing records keyed by it stay meaningful. A class carries no node
selector: its nodes are the nodes that advertise its resource name.

The CEL tests run the shipped CRD through the apiserver's own prune,
default, schema and CEL steps, so a marker that failed to generate
fails a test instead of admitting a bad class.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
The class status controller, the admission webhooks and the API
server's budget check all need the same answers: which units of a
class are free on a node, which classes clash on a resource name,
how a budget key is spelled, and whether a namespace is entitled to
vGPU. Kept in one package of pure functions, each rule has a single
implementation that every component calls, tested without a cluster.

Free units come from Node allocatable and pod requests alone, so no
inventory kind or node agent is needed. A node counts only while it
is Ready, not cordoned and carries no NoSchedule or NoExecute taint
the class does not tolerate, except the three taints Kubernetes sets
for a cordon or a node that is not Ready, which the first two rules
already decide.

Budget keys follow DRA's class-scoped form, so a key never collides
with a resource name a device plugin advertises, the key without a
class prefix needs no reserved class name, and any key in the domain
is recognised as a budget even when it is malformed.

When two classes share a resource name, the older one keeps it and
the newer one is in conflict, so a mistyped new class cannot take an
offer tenants already run on out of service.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
The VMI webhook, the launcher pod webhook, the license copier and the
budget check land separately, yet each depends on what the others
write: the template annotation and label, the gpus entry names, the
license copy's name, label and keys, and the volume label guest images
look for. The package comment states that contract next to the names
it uses, so each later piece is written and reviewed against one
description rather than against the design document.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
A vGPU class may be used only by an entitled tenant, and no tenant may
grant itself the entitlement. The Tenant field gpu.vgpu is set by the
parent's administrator, so the effective value is the field and the
parent's value together, and it reaches the namespace as the
namespace.cozystack.io/vgpu label. The label is always written, so
only "true" entitles and a missing label cannot be read as consent.
_namespace.vgpu is written only when entitled, since every app in the
tenant reads that Secret and an always-present key would upgrade them
all.

GPU budget keys in the gpuclass.cozystack.io domain are checked when
a VM or a worker pool is written, not against pods, so they are
dropped before the ResourceQuota render, and a tenant with only GPU
budgets gets no ResourceQuota or LimitRange. The domain test matches
the one the API server uses, so a malformed budget key never becomes
a pod quota either.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
A tenant is entitled to vGPU only when its parent is, and the root
tenant's namespace and _namespace values are rendered here rather than
by the tenant chart, so the chain has to start here. The value comes
from _cluster.gpu-vgpu, which the platform fills from its gpu.vgpu.
Like a tenant namespace, the root namespace always carries the label,
and _namespace.vgpu appears only when entitled.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Every tenant's vGPU entitlement needs its parent's, so the chain needs
a start the hoster controls. gpu.vgpu is it: the platform passes it to
cozystack-basics as _cluster.gpu-vgpu, which renders the root tenant.
The key is written only when on, since _cluster reaches every
tenant's values and an always-present key would upgrade every release.
The flag feeds only the entitlement, never the KubeVirt or GPU
Operator packages.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
The GPU catalog is what a tenant picks a class from, from a portal or
from kubectl and GitOps without one, so every authenticated user reads
it. GPUClass is cluster-scoped and tenant roles are bound per
namespace, so the read goes through its own ClusterRole bound to
system:authenticated. Nothing grants a write: classes are the
hoster's.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
The tenant chart renders its ResourceQuota and the default LimitRange
only for keys other than the gpuclass.cozystack.io GPU budgets, so a
tenant with only GPU budgets has no LimitRange. The comments and the
system memory limits page that explain sidecar sizing and system
defaults by that LimitRange said any declared resourceQuotas brings
it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei Aleksei Sviridkin (lexfrei) changed the title feat(tenant): add GPU budgets, the vGPU entitlement and the GPU admission policies feat(tenant): add per-class GPU budgets and the vGPU entitlement Nov 3, 2026

@kvaps Andrei Kvapil (kvaps) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Budgets, the interim check and the vGPU flag follow sections 5 and 6. I checked that the budget check does nothing for current VMInstance and KubernetesNodes values until class entries land. One regression:

parentPoolUsage in quota.go treats a tenant as its own pool root when it declares any key, and now this includes GPU-only tenants. Such tenant no longer gets a ResourceQuota, so the tenantquota controller keeps it in the parent's cpu pool, but the declaration gate removes its usage from the parent pool. Example: parent has cpu: 10, child c declares only gpuclass.cozystack.io/devices: 2 and uses 8 cpu. A new sibling asking for cpu: 10 is admitted, while without the GPU key on c the same request is refused. Section 5 says a GPU-only tenant is a pool root for the interim check and not for that controller, so for ordinary keys pool roots should be decided from non-GPU keys only. Could you add a test for this case please?

Also:

  • #5184 conflicts with this in quota.go (git apply --check fails on the used := poolUsed[...] line). Not a dependency, whoever merges second rebases.
  • tenant README should say that a sub-tenant which declares its own quotas leaves the parent's pool for every key it doesn't declare, GPU keys included. Hosters will expect GPU budgets to nest.
  • nit: gpu_budget.go:146 uses computeTenantNamespace, which builds the release name from the caller's prefix (vm-instance- here). It works only because of the root branch, a helper based on the tenant- constant would be safer.

GPU budgets are not pod resources, so ResourceQuota cannot enforce them.
Until the pool-wide admission gate is available, the API server needs
to check declared class reservations to avoid accepting applications
that exceed a tenant's budget. Reusing its inputs and pool arithmetic
keeps the eventual switch from changing admission decisions.

Tenant quota allocation must account for those reservations because
ResourceQuota usage cannot report them. Ordinary-resource accounting
must retain GPU-only descendants in the parent's pool, matching the
tenant chart and runtime controller. Unsupported keys in the GPU budget
domain must not create an unenforced promise when the chart omits them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei

Copy link
Copy Markdown
Contributor Author

Andrei Kvapil (@kvaps) I reproduced the CPU case and added regression tests for CPU, memory and ephemeral-storage. GPU-only declarations now keep that usage in the parent pool; the usage-summing tests cover extended resources and hugepages too. Namespace construction uses one prefix-independent helper. The usage-key fix is now merged in #5184, and this stack still needs to be rebased on it. I clarified the pool-scoped budget check in the tenant README.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tenant Issues or PRs related to the tenant chart and multi-tenancy kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants