Skip to content

Commit 6735277

Browse files
authored
feat(kubernetes): let a node group set the CPU request of its worker VMs (#3862)
## What this PR does Adds `nodeGroups.<name>.podCpuRequest` to the kubernetes chart, rendering into `domain.resources.requests.cpu` on the worker VM, and the same field to `kubernetes-nodes`. It has no default, so a cluster that does not set it renders exactly what it rendered before. It exists because `podCpuLimit` does not do what #3859 said it does. That PR shipped the limit on the claim that it moves the ceiling alone and leaves the CPU request at the value KubeVirt derives from the guest vCPU count and the cluster allocation ratio. KubeVirt does the opposite: `setDefaultResourceRequests` in `pkg/defaults` copies a declared CPU limit into the CPU request whenever the VM declares no request of its own, and it runs in the VMI mutating webhook, before the template service gets there. The request the template service computes from the vCPU count is then overwritten by the copy. So a node group that set only the limit asked the scheduler for its whole ceiling, which is what took every e2e worker out in #3860: Pending on `Insufficient cpu` before a guest booted. The request is the half that has to hold still, and the chart had no way to declare it. Worth correcting one detail of #3860 while I am here: it attributes the defaulting to Kubernetes copying limits into requests. Kubernetes does that too, but here the copy happens a layer earlier, on the VMI, which is why KubeVirt's own vCPU-derived request never appeared on the container. The first commit fixes that documentation, in both charts and in the generated schema, README and CRD descriptions that carry it. It also promised a floor that does not exist, a limit below the derived request was said to leave the container requesting more than its limit, when the copy makes the two equal whatever the limit is. The field itself is the scheduler reservation, not guest sizing, and it is usable on its own. An explicit request does not stop KubeVirt deriving the ceiling: `doesVMIRequireAutoResourceLimits` returns early only on a declared *limit*, so a lone request changes the weighting a worker gets on a contended host and leaves the ceiling where KubeVirt puts it. That asymmetry is why the docs tell you to keep a lone request at or below the guest vCPU count. Two render-time guards, both in the style of the one #3859 added. The instancetype guard is the same conflict one field over, and it is the same upstream check: `validateCPU` in `pkg/instancetype/apply` reports `domain.resources.requests.cpu` and `domain.resources.limits.cpu` in consecutive branches, so a VM carrying an instancetype and either key is refused by admission, leaving a node group with no Machine and a Kubernetes CR that explains nothing. The second guard refuses a request above its own limit, because that rejection otherwise lands on API server validation of the worker's Pod, two objects away from the CR that named the values. The two quantities are compared as numbers through the chart library's `toFloat`, not as the strings that spell them: `500m` under a ceiling of `1` is the ordinary case for this field, and a lexical comparison refuses it. One thing I deliberately did not guard: a lone request above the ceiling KubeVirt derives is also invalid, but the chart cannot see whether it will be. The derivation only happens in a namespace carrying a ResourceQuota that limits CPU, and the derived value counts IO threads on top of the vCPU count. Failing the render on a value that is perfectly fine in a quota-free namespace would take away a working configuration, so that case is documented rather than enforced. Byte identity for a group that sets neither field is measured rather than asserted: rendering both charts at this branch and at `main` with the knobs unset gives byte-identical `KubevirtMachineTemplate` and `MachineDeployment`, content-hash suffix included. That is load-bearing rather than tidy, a stray key or a moved space renames the template for every node group in every cluster and rolls every live worker VM on upgrade. The existing hash pins in `cluster_test.yaml` cover it too, and stay green. `kubernetes-nodes` gets the field, both guards and its own suite, because that chart is the other half of a byte-identity contract rather than a separate feature surface. Its `tests/render-parity.sh` renders a pool through both charts and compares the four pool objects byte for byte; a field only one side knows would pass that gate and then quietly drop the request when a pool is adopted into a split-out release. The gate gains a case that sets the pair. Two defects in the above turned up while I was checking it, and each is its own commit rather than a quiet amendment. The first is in the chart library: `cozy-lib.resources.toFloat` knew every suffix of the quantity grammar except `u` and `n`, and a suffix it does not recognise is not an error, the unit letter stays on the string, `float64` yields zero silently, and the comparison read zero against zero. A request of `100u` above a limit of `50u` rendered. The library gains the two missing decades and a suite that pins the whole suffix grammar, one value per form, since that failure has no error to notice it by. The second is the value domain, and the third commit settles all of it at once rather than patching whichever edge turned up. Both fields were rendered under `with`, which skips a zero as readily as an absent field, and their guards were keyed the same way; the generated schema behind them admits a leading minus just as readily. So: presence, not truth, is the predicate everywhere either field is read. A negative value is refused, it never reaches a running Pod. A zero limit is allowed and means no ceiling at all rather than the tightest one, because the kubelet turns a zero CPU limit into no CFS quota (`MilliCPUToQuota` returns zero for zero) while the declared key still stops KubeVirt deriving one. A zero request on its own is allowed and reserves nothing. A zero request beside a non-zero limit is refused, and that one is worth stating: KubeVirt's defaulter tests the request with `IsZero`, which an explicit zero satisfies exactly as an absent key does, so it overwrites the zero with the limit and the worker asks the scheduler for its whole ceiling, the outcome `podCpuRequest` exists to prevent, reached silently and reported nowhere. Refusing beats documenting when nothing downstream will ever mention it. A zero limit is the one case that needed a second look, because two reviewers independently read it as "no ceiling, so a positive request beside it is fine". It is not: API server validation refuses a request above a limit that *exists*, and zero is a limit that exists, while KubeVirt keeps deriving a request from the vCPU count whenever the VM declares none, the defaulter that would have copied the limit across does not fire, precisely because the limit is zero. So a lone `podCpuLimit: 0` renders a Pod that is rejected on creation, and it is refused here instead. Paired with `podCpuRequest: 0` it is coherent and stays allowed: no ceiling, no reservation. This settles `podCpuLimit` along with its new pair, deliberately: the two are documented as a pair and cannot sanely disagree about what zero means. Worth stating plainly that `podCpuLimit: 0` changes meaning as a result, at `main` it renders nothing, because `with` drops a zero as readily as an absent field, and here it either renders or fails the render depending on what sits beside it. `podCpuLimit` landed on `main` earlier the same day and has never been in a release, so no shipped configuration is affected, but anyone who set it on `main` between then and now should read this paragraph rather than discover it. Tests are helm-unittest in both packages, mirrored case for case, with one per class of the domain above: the request lands, a fractional quantity survives, the pair renders both keys with the ephemeral-storage pair untouched on both sides, a request alone renders no limit beside it, nothing renders when the field is absent, every guard fails on the message that names its reason, an equal request is accepted, a milli request under an integer limit is accepted, a lone zero request renders, a zero request beside a limit is refused, a zero pair is accepted, and a negative value of either field is refused. I ran a mutation ladder over the result: dropping the request render in either chart, flipping the comparison operator, comparing the quantities as strings, dropping either instancetype guard, dropping the field from the child's group dict, reverting the two suffixes, reverting either predicate to truthiness, dropping either negative guard, and letting the zero-request guard stop excluding a zero limit. Every one is caught, each by the case that exists for it, the string comparison by `500m` under a ceiling of `1`, the suffixes by the micro-range pair, the predicate and the domain rules by their own zero and negative cases. The e2e suite gets the headroom experiment back, `podCpuLimit: 3` with `podCpuRequest: 200m`. The request is what KubeVirt derived for these workers before either field existed, so scheduling is unchanged against every historical run and the ceiling is the only variable that moves. It is still an experiment against a measured mechanism, not a claimed fix for #3513: a per-thread capture on a red run showed both vCPU threads of a stuck worker burning flat out at a ceiling equal to their vCPU count while the guest kernel made no progress, which is what a vCPU spinning on a lock looks like when the vCPU holding it is the one preempted out of the quota they share. If the join failure rate does not move, the lever is elsewhere. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff. One repository is reached, and it is the one #3859 already reached: `terraform-provider-cozystack` models every app field by hand, so `podCpuRequest` in `packages/apps/kubernetes/values.schema.json` and `packages/apps/kubernetes-nodes/values.schema.json` needs the schema, the model and the expand/flatten pair over there. That work is one edit, not two, both fields land in the same model and the same docs, so the pair is added to the open one, cozystack/terraform-provider-cozystack#32, with a comment that also corrects the description of what `podCpuLimit` does, which that issue took from the claim this PR retracts. The website is not reached: its trigger is adding, renaming or removing a package, and the reference page for an existing package is regenerated from its `README.md` by the release bot. The change to `hack/e2e-chainsaw/_lib/run-kubernetes.sh` is a value inside the suite, not a move, a rename or a make target, so `ccp` is untouched. The remaining repositories restate nothing this diff touches. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#32 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(kubernetes): a node group can now set `podCpuRequest`, the CPU request of its worker VMs, next to `podCpuLimit`. Without it KubeVirt copies a declared CPU limit into the request, so raising a worker's CPU ceiling also raised what it asked the scheduler for and could leave the group unschedulable. Unset by default, so existing clusters render and behave exactly as before. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `podCpuRequest` configuration for independently controlling Kubernetes scheduler CPU reservations. * CPU requests and limits now support explicit zero and fractional quantities. * Added validation for negative values, incompatible instance-type settings, invalid zero combinations, and requests exceeding limits. * Improved resource quantity conversion, including micro- and nano-unit support. * **Documentation** * Updated configuration references and schemas with CPU request, limit, validation, and rollout behavior. * **Tests** * Added comprehensive coverage for CPU resource rendering, validation, quantity conversion, and parity. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 7103fbe + 3cb1317 commit 6735277

24 files changed

Lines changed: 1350 additions & 99 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 3 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 3 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3163,12 +3163,21 @@ ${ouroboros_addon}
31633163
logSerialConsole: true
31643164
maxReplicas: 10
31653165
minReplicas: 2
3166-
# No podCpuLimit here for now: a limit with no request makes Kubernetes
3167-
# default the request to the limit, so the compute pod asks the scheduler
3168-
# for the whole ceiling and the workers sit Pending on Insufficient cpu
3169-
# before the guest exists. The headroom experiment for
3170-
# cozystack/cozystack#3513 returns once the chart can pair the limit with
3171-
# an explicit request.
3166+
# Headroom above the vCPU count, where KubeVirt would otherwise put the
3167+
# ceiling. Workers failing to join (cozystack/cozystack#3513) have been
3168+
# measured burning their vCPU threads flat out at a ceiling equal to their
3169+
# vCPU count while the guest kernel made no progress, which is what a vCPU
3170+
# spinning on a lock looks like when the vCPU holding that lock is the one
3171+
# preempted out of the quota they share. Whether the spare CPU lets the
3172+
# preempted one back in is what this is here to answer. The request is
3173+
# what keeps scheduling where it was: setDefaultResourceRequests, in
3174+
# KubeVirt's VMI mutating webhook, copies a declared CPU limit into an
3175+
# undeclared CPU request, so the limit alone would have every worker ask
3176+
# the scheduler for its whole ceiling and sit Pending on Insufficient cpu.
3177+
# Its value is the one KubeVirt derived for these workers before, from the
3178+
# vCPU count and the cluster CPU allocation ratio.
3179+
podCpuLimit: 3
3180+
podCpuRequest: 200m
31723181
# Two vCPUs at the memory the workers had before. Sized by resources
31733182
# rather than by instanceType: u1.large doubles memory along with the
31743183
# vCPUs, and four 8Gi workers do not fit beside the rest of the suite,

0 commit comments

Comments
 (0)