Repository navigation
feat(tenant): add per-class GPU budgets and the vGPU entitlement - #5008
Aleksei Sviridkin (lexfrei) wants to merge 9 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
f668efa to
46ff481
Compare
f2f1d59 to
a883541
Compare
A hoster publishes each GPU offer as a cluster-scoped GPUClass: one resource name per offer, the tolerations its VMs need on the class's nodes, and for vGPU classes the license Secret and FeatureType the guests get. VMs and worker pools name a class instead of a raw device-plugin name, so the offer can be budgeted per tenant and shown in a catalog with its free units. The kind joins cozystack.io instead of a group of its own: a CRD cannot move between groups in place, and nothing in the design needs separate versioning or serving. The class name becomes part of the budget key <class>.gpuclass.cozystack.io/devices, so it is a DNS label, and a resource name in that domain is refused. Vendor, model, mode, profile, memory and the source kind never change, so a class name stands for one offer for its whole life and billing records keyed by it stay meaningful. A class carries no node selector: its nodes are the nodes that advertise its resource name. The CEL tests run the shipped CRD through the apiserver's own prune, default, schema and CEL steps, so a marker that failed to generate fails a test instead of admitting a bad class. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The class status controller, the admission webhooks and the API server's budget check all need the same answers: which units of a class are free on a node, which classes clash on a resource name, how a budget key is spelled, and whether a namespace is entitled to vGPU. Kept in one package of pure functions, each rule has a single implementation that every component calls, tested without a cluster. Free units come from Node allocatable and pod requests alone, so no inventory kind or node agent is needed. A node counts only while it is Ready, not cordoned and carries no NoSchedule or NoExecute taint the class does not tolerate, except the three taints Kubernetes sets for a cordon or a node that is not Ready, which the first two rules already decide. Budget keys follow DRA's class-scoped form, so a key never collides with a resource name a device plugin advertises, the key without a class prefix needs no reserved class name, and any key in the domain is recognised as a budget even when it is malformed. When two classes share a resource name, the older one keeps it and the newer one is in conflict, so a mistyped new class cannot take an offer tenants already run on out of service. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The VMI webhook, the launcher pod webhook, the license copier and the budget check land separately, yet each depends on what the others write: the template annotation and label, the gpus entry names, the license copy's name, label and keys, and the volume label guest images look for. The package comment states that contract next to the names it uses, so each later piece is written and reviewed against one description rather than against the design document. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
46ff481 to
9f607f0
Compare
A vGPU class may be used only by an entitled tenant, and no tenant may grant itself the entitlement. The Tenant field gpu.vgpu is set by the parent's administrator, so the effective value is the field and the parent's value together, and it reaches the namespace as the namespace.cozystack.io/vgpu label. The label is always written, so only "true" entitles and a missing label cannot be read as consent. _namespace.vgpu is written only when entitled, since every app in the tenant reads that Secret and an always-present key would upgrade them all. GPU budget keys in the gpuclass.cozystack.io domain are checked when a VM or a worker pool is written, not against pods, so they are dropped before the ResourceQuota render, and a tenant with only GPU budgets gets no ResourceQuota or LimitRange. The domain test matches the one the API server uses, so a malformed budget key never becomes a pod quota either. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
A tenant is entitled to vGPU only when its parent is, and the root tenant's namespace and _namespace values are rendered here rather than by the tenant chart, so the chain has to start here. The value comes from _cluster.gpu-vgpu, which the platform fills from its gpu.vgpu. Like a tenant namespace, the root namespace always carries the label, and _namespace.vgpu appears only when entitled. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Every tenant's vGPU entitlement needs its parent's, so the chain needs a start the hoster controls. gpu.vgpu is it: the platform passes it to cozystack-basics as _cluster.gpu-vgpu, which renders the root tenant. The key is written only when on, since _cluster reaches every tenant's values and an always-present key would upgrade every release. The flag feeds only the entitlement, never the KubeVirt or GPU Operator packages. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The GPU catalog is what a tenant picks a class from, from a portal or from kubectl and GitOps without one, so every authenticated user reads it. GPUClass is cluster-scoped and tenant roles are bound per namespace, so the read goes through its own ClusterRole bound to system:authenticated. Nothing grants a write: classes are the hoster's. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The tenant chart renders its ResourceQuota and the default LimitRange only for keys other than the gpuclass.cozystack.io GPU budgets, so a tenant with only GPU budgets has no LimitRange. The comments and the system memory limits page that explain sidecar sizing and system defaults by that LimitRange said any declared resourceQuotas brings it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
a883541 to
02d1264
Compare
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Budgets, the interim check and the vGPU flag follow sections 5 and 6. I checked that the budget check does nothing for current VMInstance and KubernetesNodes values until class entries land. One regression:
parentPoolUsage in quota.go treats a tenant as its own pool root when it declares any key, and now this includes GPU-only tenants. Such tenant no longer gets a ResourceQuota, so the tenantquota controller keeps it in the parent's cpu pool, but the declaration gate removes its usage from the parent pool. Example: parent has cpu: 10, child c declares only gpuclass.cozystack.io/devices: 2 and uses 8 cpu. A new sibling asking for cpu: 10 is admitted, while without the GPU key on c the same request is refused. Section 5 says a GPU-only tenant is a pool root for the interim check and not for that controller, so for ordinary keys pool roots should be decided from non-GPU keys only. Could you add a test for this case please?
Also:
- #5184 conflicts with this in
quota.go(git apply --checkfails on theused := poolUsed[...]line). Not a dependency, whoever merges second rebases. - tenant README should say that a sub-tenant which declares its own quotas leaves the parent's pool for every key it doesn't declare, GPU keys included. Hosters will expect GPU budgets to nest.
- nit:
gpu_budget.go:146usescomputeTenantNamespace, which builds the release name from the caller's prefix (vm-instance-here). It works only because of the root branch, a helper based on thetenant-constant would be safer.
GPU budgets are not pod resources, so ResourceQuota cannot enforce them. Until the pool-wide admission gate is available, the API server needs to check declared class reservations to avoid accepting applications that exceed a tenant's budget. Reusing its inputs and pool arithmetic keeps the eventual switch from changing admission decisions. Tenant quota allocation must account for those reservations because ResourceQuota usage cannot report them. Ordinary-resource accounting must retain GPU-only descendants in the parent's pool, matching the tenant chart and runtime controller. Unsupported keys in the GPU budget domain must not create an unenforced promise when the chart omits them. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
02d1264 to
f921aca
Compare
|
Andrei Kvapil (@kvaps) I reproduced the CPU case and added regression tests for CPU, memory and ephemeral-storage. GPU-only declarations now keep that usage in the parent pool; the usage-summing tests cover extended resources and hugepages too. Namespace construction uses one prefix-independent helper. The usage-key fix is now merged in #5184, and this stack still needs to be rebased on it. I clarified the pool-scoped budget check in the tenant README. |
9f607f0 to
e721884
Compare
Builds on the GPU design approved in cozystack/community#99 at
30da976.What this PR does
This adds per-class GPU budgets and the vGPU entitlement for tenants. It builds on #4998.
A tenant declares budgets under
<class>.gpuclass.cozystack.io/devicesorgpuclass.cozystack.io/devices. These keys stay out of ResourceQuota. A tenant with only GPU budgets gets no ResourceQuota or LimitRange.The API server checks the pool's declared class entries when a VM or worker pool is written. A VM reserves
count; a worker pool reservescounttimesmaxReplicas. Only growth is checked on update. This is the interim check until the pool-wide gate from community#48 is on. Tenant writes count the same entries when allocating GPU budgets to children.GPU-only declarations leave ordinary-resource usage in the parent's runtime pool. The namespace helper no longer depends on the calling application's release prefix.
The entitlement is effective only when the parent is entitled too.
Tenant.spec.gpu.vgpugrants it, and the effective value goes on the namespace asnamespace.cozystack.io/vgpu. The root takes its value from the platform'sgpu.vgpu.Every authenticated user can read GPUClass through
cozy:gpuclasses:view.Class entries in the app charts follow in #5030. The separate usage-key fix in #5184 conflicts on one line in
quota.go; whichever merges second needs a rebase.Screenshots
Not applicable: no UI change.
Downstream repositories
Follow-ups for the website's platform values table and the provider's Tenant schema to add
gpu.vgpuare tracked in the issues linked above; implementation pending.Release note