Repository navigation
Commit ef86974
committed
fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets (#4974)
On the kubevirt substrate `resources.cpu` is written into
`domain.cpu.cores` as spelled and decoded as YAML, so `"010"` gives the
VM eight cores, while the cluster-autoscaler capacity annotation parsed
the same string as decimal ten
(`capacity.cluster-autoscaler.kubernetes.io/cpu: "9000m"` on a node that
offers 7000m). This is the inconsistency reported in #4554 and in the
comment on #4546.
- The YAML-decoded value the range check already computes is kept, and
on kubevirt the capacity annotation and the reserved-CPU check start
from it.
- The kubelet reservations are not touched. The worker machineconfig
(`_talosconfigtemplate.tpl`) sizes them from the raw string read as a
decimal, and the TalosConfigTemplate name is a hash of that config, so
reading the decoded value there would roll every leading-zero pool once.
Instead the annotation subtracts the reservations the machineconfig
really carries from the cores the VM really has: for `"010"` that is
8000m − 2 × 500m = `"7000m"`, the node's real allocatable, while `"8"`
keeps `"7200m"` (2 × 400m). Tests pin both sides: the annotation and the
bootstrap `configRef` name in `resources_cpu_test.yaml`, and
`systemReserved.cpu` / `kubeReserved.cpu` in the machineconfig in
`talos_worker_config_test.yaml`.
- No pool rolls. Checked by rendering `nodegroup.yaml` on main and on
this branch for nine spellings (`"2"`, `2`, `2.0`, `"2.0"`, `"+2"`,
`"1e1"`, `"010"`, `999999`, `4294967295`): the only difference is the
capacity annotation of the `"010"` case, so the
`KubevirtMachineTemplate` name and the TalosConfigTemplate `configRef`
stay as they are. The machineconfig template itself is not changed.
- Proxmox parses the quantity as decimal for the machine template and
for the reservations alike, so it never had the inconsistency and is
unchanged (pinned by a test).
- The `resources` description says the value is decoded as YAML, what
that means for a leading zero, and that the reservations still read it
as a decimal. README, schema, the RD and the Go type are regenerated
from it.
This closes #4554 without the migration and the render-time refusal it
proposes. A refusal at render breaks a stored pool on its next upgrade,
a pattern in the OpenAPI schema breaks every update of an existing CR,
and a platform-wide migration over every `KubernetesNodes` and parent
`nodeGroups` entry is a lot of blast radius for a spelling nobody has
reported in a live cluster. With this change the autoscaler plans for
the node the VM really is. What is left of a leading zero is the octal
reading itself and reservations sized from the decimal value, and both
are documented. The unquoted zero of the issue is already harmless: it
is falsy, counts as unset, and renders the same bytes as an absent
field. If a migration is still wanted, please reopen.
Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38
suites, 233 tests, 12 snapshots; pointing the annotation back at the
decimal string turns the `"010"` case red with `9000m`), `make generate`
(no diff), and a trial merge with #4973 (clean, 243 tests green).
Closes #4554
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* KubeVirt VM sizing and autoscaler capacity now reflect YAML-decoded
CPU values. For example, `"010"` represents eight VM cores, while
kubelet reservations interpret it as decimal ten. `"8"` and `"10"` yield
`7200m` and `9000m` of autoscaler capacity, respectively. Proxmox
continues to interpret `"010"` as ten cores, yielding `9000m` of
capacity.
* **Documentation**
* Clarified explicit worker CPU and memory sizing behavior, including
how leading-zero CPU values affect VM sizing, reservations, and capacity
calculations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Backported-from: 60d1f538 files changed
Lines changed: 158 additions & 11 deletions
File tree
- api/apps/v1alpha1/kubernetesnodes
- packages
- apps/kubernetes-nodes
- templates
- tests
- system/kubernetes-nodes-rd/cozyrds
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | | - | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
108 | 120 | | |
109 | 121 | | |
110 | 122 | | |
| |||
113 | 125 | | |
114 | 126 | | |
115 | 127 | | |
| 128 | + | |
116 | 129 | | |
117 | 130 | | |
118 | 131 | | |
| |||
749 | 762 | | |
750 | 763 | | |
751 | 764 | | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
752 | 771 | | |
753 | 772 | | |
754 | 773 | | |
| |||
865 | 884 | | |
866 | 885 | | |
867 | 886 | | |
868 | | - | |
| 887 | + | |
869 | 888 | | |
870 | 889 | | |
871 | 890 | | |
872 | | - | |
| 891 | + | |
873 | 892 | | |
874 | 893 | | |
875 | | - | |
| 894 | + | |
876 | 895 | | |
877 | 896 | | |
878 | 897 | | |
| |||
944 | 963 | | |
945 | 964 | | |
946 | 965 | | |
947 | | - | |
| 966 | + | |
948 | 967 | | |
949 | 968 | | |
950 | | - | |
| 969 | + | |
951 | 970 | | |
952 | 971 | | |
953 | 972 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
188 | 188 | | |
189 | 189 | | |
190 | 190 | | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
191 | 255 | | |
192 | 256 | | |
193 | 257 | | |
| |||
294 | 358 | | |
295 | 359 | | |
296 | 360 | | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
77 | 112 | | |
78 | 113 | | |
79 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
71 | | - | |
| 71 | + | |
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
70 | | - | |
| 70 | + | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments