feat(kubernetes): run tenant worker pools on Kairos - #4911

The tenant cluster-autoscaler reads the newest Cluster API version the
management cluster serves unless CAPI_VERSION names one. With Cluster
API v1.13 that is v1beta2, and the 1.32 build shipped here predates
the autoscaler fixes for reading v1beta2: scale from zero
(kubernetes/autoscaler#8410, in 1.34) and infrastructure reference
lookup (kubernetes/autoscaler#9458, in 1.36). The pools here carry
capacity annotations for scale from zero, so that path is in use.

Pinning v1beta1 keeps the autoscaler on the API it was built against.
Cluster API serves v1beta1 until v1.16, which leaves time to move the
image to a release with both fixes and drop the pin.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The Kubernetes suite waited on status.v1beta2.readyReplicas through a
bare "machinedeployment" resource name. That name resolves to the
version the cluster prefers: v1beta1 under Cluster API v1.10, v1beta2
from v1.11. In v1beta2 the counter is status.readyReplicas and
status.v1beta2 does not exist, so after a Cluster API upgrade the wait
would run out its ten minutes and fail.

The wait now names the v1beta1 resource, which carries
status.v1beta2.readyReplicas on both sides of the upgrade and is
served until Cluster API v1.16.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Bootstrap and control-plane providers built for the v1beta2 contract
need Cluster API v1.11 or later. v1.13.7 is the newest release this
cluster can move to from v1.10 in one step: upstream supports
upgrades that skip at most three minor versions, so v1.14 has to
follow in a later release. v1.13 still serves the v1beta1 API and
accepts v1beta1-contract providers, so the providers that stay on
that contract keep working; upstream stops serving v1beta1 in v1.16.

The core package gains a components target in the style of the
capmox and in-cluster IPAM packages: it downloads the release assets,
checks them against pinned sums, applies patches/ and writes the YAML
and the blob the chart mounts. The startupProbe that kept the manager
alive through certificate provisioning becomes a patch instead of a
hand edit of the vendored file, so a refresh carries it forward.

The capmox and in-cluster IPAM pin comments named the v1.10.1 core as
the reason to stay where they are. That no longer holds; the comments
now give the reasons that do: the v1alpha2 API migration capmox v0.8
brings, and IPAM v1.1 moving to the v1beta2 contract while capmox v0.7
stays on v1beta1.

The check that the core blob matches its YAML moves from its own bats
file into the provider suite, so a failure now points at
make components instead of a hand-run gzip that skips the sums and
the patch.

The vendored YAML carries patches/, so its sum cannot be pinned
directly. The bats suite reverts the patches and checks the result
against the pinned release sum instead, which catches a hand edit of
the vendored file as well as a patch or a sum changed on its own.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The kubeadm bootstrap provider is cut from the same Cluster API
release as the core provider and moves with it.

Like the core package, it gains a components target that downloads
the release assets and checks them against pinned sums, and the bats
suite now holds its version, its blob and its sums to the Makefile the
way it already does for capmox and in-cluster IPAM. A test also keeps
the core and the kubeadm bootstrap provider on one release. The
package ships the Talos bootstrap provider too, so the version check
looks for the provider by its suffix rather than assuming a single
ConfigMap.

The package's .helmignore pattern matched only dot files, so the
readable YAML copies shipped in the chart next to the blobs the
templates read. It now excludes them and patches/, as core does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

v0.6.13 is the latest stable release; the only change from v0.6.12 is
support for Talos v1.14 machine configuration. The v0.6 series stays
on the v1beta1 contract, which Cluster API v1.13 still accepts. The
v0.7 series that moves to v1beta2 has only pre-releases so far.

The update target now checks the downloaded release assets against
pinned sums before applying patches/, and refreshes
metadata-talos.yaml along with the components. The bats suite holds
the Talos provider to the same checks as core and kubeadm: its
ConfigMap and BootstrapProvider carry the vendored version, its blob
matches its YAML, and the YAML with patches/ reverted matches the
pinned release sum.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

capmox v0.8 still serves v1alpha1 for its machine and cluster CRDs
through a conversion webhook, so the v1alpha1 templates the
kubernetes-nodes chart renders would still be accepted. What holds
capmox back is the move to the v1beta2 contract and v1alpha2 storage,
which has not been tried against those templates and should land
together with in-cluster IPAM v1.1.

Suggested-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Install cluster-api-provider-kairos next to the kubeadm and Talos
bootstrap providers, so a tenant worker pool can join its Kamaji
control plane as a Kairos node through `kubeadm join`.

The manager is built here from upstream release v0.2.1, the first
release that carries everything the worker pools rely on: the kubeadm
worker distribution, the patches directory named in the
JoinConfiguration (kubeadm refuses --patches next to --config), and
post-bootstrap units that no longer hold back the userdata stage. The
manager carries no downstream patch. The patches/ directory stays in
the image build so a fix can ship before an upstream release; its tests
run during the build, since go build never compiles them.

The rendered manifest is adapted the way the Talos provider's is.
capi-operator installs every provider into one namespace, where the
upstream selector, control-plane: controller-manager, also matches the
other providers' pods: the webhook Service could send a call to a
foreign pod, which fails TLS, and the Deployment selector overlapped the
others'. The Service and the Deployment now select the provider label as
well, which the pod template gains, as the Talos and KubeVirt providers
already do.

The component version is the upstream release with a cozystack.N
counter. The components and the clusterctl metadata are rendered from
the same release; the package tests tie the release, the version, the
metadata release series, the manager image and the gzip the chart
mounts to one another. The image is built by the root build, and its
reference inside the gzip is restitched like the Kamaji provider's.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Kairos worker pools need a disk with an OS the platform controls: a
Hadron system with containerd, and kubelet and kubeadm at the exact
patch release of the pool's Kubernetes version, because the Kairos
bootstrap provider refuses a join with any other kubeadm.

`make image` builds one disk per Kubernetes minor in
files/versions.yaml. The upstream kubelet binary links against glibc
and Hadron is musl-based, so kubelet is built without cgo from that
release's source tree, with the Go toolchain the tree names. kubeadm
is the static upstream binary of the same release, checked against
its published SHA-512. The OS image runs its own checks first,
including that both binaries are static and start, then AuroraBoot
turns it into a raw EFI disk per architecture whose first boot
installs the system, and the disks are wrapped as qcow2 in the
container disk CDI imports. Each disk's reference is written to
images/kairos-worker-<minor>.tag. The disk carries no user and no key;
the bootstrap provider's cloud-config supplies them.

The OS image is published too, as kairos-worker-base-<minor> with its
own .tag, so a platform operator who needs a kernel module or driver
the platform does not ship adds it in one FROM layer on that image,
pinned by digest. When the build pushes, each disk is made from the
published image pulled back by digest, so the disk and the image an
operator builds on are the same bytes; a build that does not push
publishes no OS image and makes the disk from a local build.

Renovate moves the hadron and hadron-toolchain pins together and
tracks the AuroraBoot image the Makefile runs.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The matrix build hands the refs it stamps to finalize as a patch made
with `git diff --binary HEAD`, which leaves out untracked files. A
package whose image target writes a new ref-bearing file, such as the
.tag of a newly added image, built it for nothing: finalize never saw
the ref. The step now marks the new files under the package being
built with intent-to-add, so the diff carries them, and files outside
that package stay out of its fragment.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Each Kairos worker disk compiles kubelet for its Kubernetes minor and
runs AuroraBoot in a privileged container that drives the runner's
docker daemon. One disk per supported minor outgrows the time limit of
a parallel matrix job, so kubernetes-nodes leaves the matrix and gets
its own pull-request job, gated on the disk inputs like the Talos
build. Finalize keeps the refs that job stamps, and the arm64 release
and nightly legs build the disks after the matrix.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Point every Kubernetes minor in files/versions.yaml at its Kairos
worker disk in the project registry, so a pool on a Kairos class
without its own image boots the platform disk for its minor, and at
the OS image the disk is made from, which operators build their own
worker images on. A test holds that each minor has a disk pinned by
digest and an OS image next to it, pinned from the same build once
that build has run, and that nothing is pinned for a minor the chart
no longer offers.

The disks in git are amd64-only; the release build adds arm64 through
its native arm64 leg and the multi-arch stitch.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The Talos worker template computed the kubelet system and kube
reservations and the eviction thresholds inline. The Kairos worker
configuration needs the same numbers, so the computation moves into a
helper both can call. The Talos render is unchanged: the snapshot
tests still match byte for byte.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

A worker pool needs a way to run an OS image the platform operator
supplies without each tenant pool carrying the image reference: a
shared reference on every pool would roll every tenant at once when it
changed, and a registry credential would have to be copied into every
tenant namespace. NodeGroupClass is a cluster-scoped, operator-owned
object a pool names, the way a PVC names a StorageClass. Tenant roles
come from RoleBindings, so a tenant cannot create one.

The spec is immutable as a whole, so a new image or version is a new
class and no edit can roll every pool on a class at once. An image is
allowed only on a Kairos class and carries the Kubernetes release its
kubelet and kubeadm are built at.

The platform ships the talos and kairos classes. The pool chart
resolves those two names without reading them, so a Talos pool does
not depend on an object existing; the CRD refuses any other content
under either name, so an operator cannot create one whose content the
chart would silently ignore. The objects ride in kubernetes-nodes-rd,
which depends on the package that installs the CRD.

The kind joins the cozystack.io group next to the platform's other
cluster-wide kinds instead of opening a group of its own: a group is a
contract, and a CRD cannot move to another group in place.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

A pool can now run Kairos workers instead of Talos ones. It names a
NodeGroupClass in nodeGroupClassName, talos by default. A pool on a
Kairos class boots the worker disk the platform builds for its
Kubernetes minor, or the image the class names, and joins the tenant
cluster with kubeadm through the Kairos Cluster API bootstrap
provider, referenced from the MachineDeployment as a
KairosConfigTemplate. A pool without a class renders exactly as
before.

The built-in talos and kairos classes are resolved without reading any
object, so those pools render offline and do not depend on the objects
existing. Any other class is read with lookup, and a missing one fails
the render with its name.

A class that names an image also fixes the Kubernetes release: the
Machine version and the version kubeadm is given come from the class,
not from files/versions.yaml, because that version is part of the hash
that names the template, and a platform patch bump would otherwise roll
every pool on an operator's image. The chart only checks that the
class's minor is the pool's version, and that the pool is not ahead of
its control plane.

The worker VM boots with EFI firmware, which the Kairos disk images
need, and imports its system disk from the registry image.

The KairosConfigTemplate needs the apiserver Service ClusterIP and the
management CoreDNS address, which exist only once the control plane
runs, so a kairos-reconcile Job applies it the way talos-reconcile
applies the Talos template. The spec travels in a ConfigMap the Job
mounts and fills in with envsubst, limited to those runtime addresses,
so no tenant value is ever part of the Job's script; a tenant value that
literally contains one of those variable names gets the address
substituted, which touches only that value. The worker joins the
apiserver by its Service name. kubeadm takes the server from the
cluster-info ConfigMap, which carries that name, so it has to resolve
during the first boot; the bootstrap files are written after
systemd-resolved has started, so the name goes into /etc/hosts, the same
mapping the Talos workers get through extraHostEntries. Search domains
and the nameserver follow the Talos workers through a resolved drop-in,
and the kubelet reservations and eviction thresholds through a kubeadm
kubelet patch in a patches directory the template names itself.

The bootstrap provider requires a user credential. The worker user gets
a password hash no input matches, so there is no password login, and an
optional SSH key is the only way in.

Kairos pools are refused on the proxmox substrate, and a non-empty
osImage, kernelModules or gpus is refused on them by name: the first
two only configure Talos, and the Kairos worker image carries no GPU
driver. talos.* cannot be refused the same way: the API fills its
defaults into every pool it returns, so a spec written back stores
them. A Kairos pool ignores talos.* instead, and nothing on the Kairos
path reads it, the Image Factory CA ConfigMap included. A running pool
cannot move between a Talos and a Kairos class in either direction:
the render checks the bootstrap template kind of the live
MachineDeployment, the way the substrate guard does; a pool still on
the kubeadm bootstrap moves onto whichever of the two it asks for,
Talos by default, as before.

The pull-request unit test job installs envsubst when the runner lacks
it, for the test that runs the reconcile script with the real tool, and
the helm-unittest plugin on docs-only runs as well, because a Bats test
of the class now renders the chart through it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Kairos pools join through a different bootstrap provider, a different
worker disk and a kubeadm join, none of which the Talos pools of the
suite exercise. After the tenant cluster passes its own checks, the
suite adds a pool on the built-in kairos NodeGroupClass and requires
its Machine to link a Ready Node that carries a KubeVirt providerID and
the cluster-namespace annotation, and a pod to run on it. Each wait on
the management side collects the pool's disk import, machines, health
check, events and guest console before it fails, so a registry that
refuses the disk is told apart from a worker that booted and never
joined.

The step runs inside the E2E job, which had about 16 minutes left
under its 215-minute cap on a full run of this branch. A red Kairos
step would end the job as cancelled and lose its log and report, so
the cap moves to 255 minutes in every lane that runs the suite, and
the step's ceiling is sized so that the old cap plus the step stays
under the new one; the sum is written beside the step.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>