Repository navigation
Commit 5fe9450
authored
fix(kubernetes): widen and expose the worker node join timer budgets (#3898)
## What this PR does
Worker node join on nested tenants regularly outruns the timer budgets
the charts shipped, and the timers then destroy capacity that was about
to arrive: the cluster-autoscaler treats a machine as long-unregistered
after its built-in 15m default and orders a replacement, and the
MachineHealthCheck deletes a Machine that is still legitimately booting.
The timer audit in #3513
(#3513 (comment))
walked every timer on the join path against the cozyreport artifact of a
red run; this PR sizes the budgets from those measurements and makes the
hardcoded ones configurable. Context: #3513.
Three changes, one commit each:
- Every worker MachineDeployment in both the kubernetes and
kubernetes-nodes charts now carries the
`cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation,
rendered from a new `maxNodeProvisionTime` value defaulting to "30m".
The autoscaler starts this clock when the machine enters its
node-instances cache with a real providerID, not at Machine creation,
and on a value it cannot parse it silently falls back to its 15m default
rather than failing, so the template validates the value at render time
and rejects anything that does not parse to a positive duration.
- The MachineHealthCheck `unhealthyConditions` timeouts become values
(`readyUnknownTimeout` default "5m", `readyFalseTimeout` default "10m";
nested under `nodeHealthCheck` in the kubernetes chart, flat in
kubernetes-nodes) instead of the hardcoded 30s and 300s. 30 seconds of
`Ready=Unknown` deletes a worker whose guest was merely starved of CPU,
and 300 seconds of `Ready=False` is less than a CNI rollout takes on a
loaded host.
- The `nodeStartupTimeout` default rises from "10m" to "20m". The
MachineHealthCheck measures it from the later of the Machine's creation
and its InfrastructureReady transition, so the clock restarts when the
infrastructure reports ready: a worker gets up to 20m for the disk image
import and the VM to come up, and then up to 20m again for the guest
boot and kubelet registration. The `maxUnhealthy` docs now also spell
out the round-down trap in small groups: at "50%" a group of one never
remediates.
The budgets are coherent with the measured phases: in the audited runs a
healthy disk import took 2m22s and import plus VM start 2m53s, while the
stuck runs spent over 17 minutes holding a running VM with no Node, so
20m per MachineHealthCheck leg and 30m on the autoscaler clock absorb
that shape of slowness without letting a genuinely dead machine linger
much longer.
New helm-unittest suites cover the annotation rendering, the render-time
duration guard (including the values the guard exists for: unitless
numbers, unknown units, zero, negatives, parser overflow), and the
MachineHealthCheck timeout rendering in both charts. The render-parity
test no longer pins the MachineHealthCheck values on either side and
instead compares the two charts' defaults, so a future drift between
parent and child defaults fails the suite.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: TODO-LEAD-FILE-ISSUE (suggested title:
"Kubernetes/KubernetesNodes: model the new readyUnknownTimeout,
readyFalseTimeout and maxNodeProvisionTime fields (upstream
cozystack#NNNN)"; the provider models node_health_check for both
resources and gains no typed access to the three new fields until it
learns them)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(kubernetes): widen the worker node join timer budgets and make them configurable: the cluster-autoscaler maxNodeProvisionTime is now set per MachineDeployment (default 30m), the MachineHealthCheck Ready=Unknown and Ready=False timeouts are values (defaults 5m and 10m), and nodeStartupTimeout defaults to 20m
```17 files changed
Lines changed: 592 additions & 36 deletions
File tree
- api/apps/v1alpha1
- kubernetesnodes
- kubernetes
- packages
- apps
- kubernetes-nodes
- templates
- tests
- kubernetes
- templates
- tests
- system
- kubernetes-nodes-rd/cozyrds
- kubernetes-rd/cozyrds
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments