Skip to content

feat(bootstrap): add kubeadm as a third distribution for worker joins - #141

Merged
wrkode merged 29 commits into
kairos-io:mainfrom
wrkode:feat/kubeadm-worker-distribution
Oct 13, 2026
Merged

wrkode merged 29 commits into
kairos-io:mainfrom
wrkode:feat/kubeadm-worker-distribution

Conversation

@wrkode

@wrkode wrkode commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Draft, opened for visibility. This is the work described in kairos-io/kairos#4920 on 25 September: kubeadm as a third distribution, starting with the hosted worker case. It should have been visible from the moment it started, and it was not. See the note on #135 below.

Setting spec.distribution: kubeadm on a KairosConfig or KairosConfigTemplate bootstraps a Kairos worker that joins a control plane this provider does not manage: a Kamaji KamajiControlPlane, or a Cluster API KubeadmControlPlane. Kamaji needs no changes on its side.

How the join material is obtained

It follows the same contract CABPK does. The controller mints a short-lived bootstrap token in the workload cluster and pins the cluster CA, both reached through <cluster>-kubeconfig.

That Secret is only trusted after a check, because its contents decide which CA a node pins and which API server it joins:

  • its controller owner must be the cluster's resolved control-plane object, matched on group and kind;
  • the kinds trusted without an operator flag are KubeadmControlPlane, KamajiControlPlane and KairosControlPlane; anything else needs --kubeadm-extra-controlplane-kinds;
  • the CA is cross-checked against <cluster>-ca where that exists;
  • the workload client is clustercache-backed and gated by a ClusterFilter, so the manager neither dials nor caches arbitrary clusters.

Tokens use a 15-minute TTL and rotate at half-TTL, the same rule as CABPK's shouldRotate, with a 24-hour cap. A machine that is slow to install does not get stranded by an expired token.

What runs on the node

The controller marshals a kubeadm JoinConfiguration with the CAPI version-aware marshaller and the node runs kubeadm join --config, with the provider ID applied through --patches. There is no CABPK, no node-side hook to add to a template, and no Jinja.

The join is gated so it happens once and only where it should:

  • only on an installed boot, active or passive, with persistent storage mounted; never on live, recovery, auto-reset or in-RAM boots;
  • a node that has joined never joins again on reboot, keyed off persistent markers under /var/lib/kairos plus kubeadm's own kubelet.conf preflight, because the kubelet.conf check alone fails open on affected Kairos versions;
  • a failed join is not retried on the node. Cluster API remediation replaces the Machine.

The Kubernetes version is resolved through the CAPI contract rather than pinned, and checked against the kubeadm the image actually ships before joining.

API

spec.distribution gains kubeadm, and spec.kubeadm.joinConfiguration takes the upstream CAPI kubeadm v1beta2 type rather than a hand-rolled subset, so the served CRD does not carry the whole kubeadm bootstrap surface. KairosControlPlane refuses distribution: kubeadm; running a kubeadm control plane on Kairos nodes is a separate design and not in this change.

Scope

Supported: Kairos workers from a MachineDeployment. Not supported, and refused: a kubeadm KairosControlPlane, MachinePool workers, and joining a k0s or k3s KairosControlPlane. CNI installation and image building are out of scope; docs/KUBEADM_WORKERS.md covers the prerequisites.

The diff is larger than the feature because the k0s and k3s generators were merged behind one distribution table first, so a third distribution adds a row instead of a third copy. The refactor commits come before the feature commits.

Verification status, stated plainly

Unit tests, golden fixtures and envtest only. This has not been run against a live Kamaji or KubeadmControlPlane cluster, or against a specific Kairos image. That is why this is a draft. The provider-kubernetes images are the reference for a node image that ships kubeadm, kubelet and containerd, and their Kubernetes range overlaps our validated workload band only at 1.35 and 1.36.

Relationship to #135

#135 implements the same distribution: kubeadm enum value for the same use case, independently, and arrived while this branch was sitting unpublished. The overlap is this repository's fault, not the contributor's. The two differ in where the join executes: #135 renders the Kairos cluster: block and lets provider-kubernetes perform the join, which is a route the internal decision records evaluated and set aside over lifecycle ownership (in-place kubeadm upgrade, its own reset, node-side minting), whereas this change keeps the lifecycle and the PKI on the management side. The specific review points are on #135.

wrkode added 22 commits October 1, 2026 10:05
…tribution, role and infrastructure

Freeze the byte-for-byte output of generateCloudConfig across the full
controller assembly (token resolution, providerID, management-endpoint
resolver, LB lookup, applyControlPlaneRenderData) so the upcoming P0
distribution-seam refactor can prove it changes no rendered bytes.

Matrix: {k0s,k3s} x {worker on CAPV/CAPK/Metal3/fleet; single/init/join
on CAPV and CAPK}, plus the S5 control-plane edge cases: a nil resolver,
a resolver returning (nil,nil), a DockerMachine control plane (push
disabled), and a k3s control plane with an explicit spec.serverAddress.

Every golden is asserted to be valid YAML. spec.Manifests is held out of
these fixtures: the manifest render path emits content and the heredoc
terminator at column 0 inside a content:| block scalar, which is not
valid YAML. That is a pre-existing template characteristic, out of P0
scope; the controller's copy of spec.Manifests is characterised
separately.

Signed-off-by: William Rizzo <[email protected]>
…d the regeneration contracts

Characterise behaviour the goldens cannot observe, before the P0
distribution-seam refactor:

- Field-set isolation (I1): a k3s config with a control character in
  spec.podCIDR and a k0s control plane with a control character in
  serverAddress both render successfully, proving neither generator
  routes the other distribution's field into the validated TemplateData.
- Exact error strings (M3b): one case per worker-token path on both the
  k0s and k3s chains (cross-namespace with the worker/k3s label, missing
  key, legacy keyless, hard Get failure), plus the controller's
  empty-token, missing-serverAddress and unsupported-distribution
  messages, asserted verbatim.
- errTokenNotReady for a referenced-but-absent worker Secret, and the
  token-before-password read order (the token error wins).
- The LB-endpoint error asymmetry as it stands today (OQ-A): k0s bare,
  k3s wrapped. The dedicated fix commit updates the k0s expectation.
- Render contract (S2): every distribution/infra variant rendered with a
  providerID contains its post-bootstrap marker and an SSH-enable stage.
- Regeneration decision (R4): an existing Secret with providerID, marker
  and SSH-enable stage is left alone; one missing either is regenerated.
- Manifests are copied into the render by both generators (held out of
  the YAML goldens because the manifest heredoc is not valid YAML).

Signed-off-by: William Rizzo <[email protected]>
Pin the user-visible distribution admission strings verbatim before the
P0 seam moves KairosConfig's per-distribution validation into
distribution.go, and before kubeadm P1 touches the enums:

- KairosConfig: the distribution enum message, the k0s and k3s worker
  empty-token messages, and the two-error aggregate for an unknown
  distribution on a worker (enum error + default k0s worker-token error,
  OQ-C). Plus the defaulter writing k0s for an unset distribution.
- KairosControlPlane and KairosControlPlaneTemplate: the distribution
  enum messages verbatim (KCPT including the spec.template.spec.* path).
  These webhook files must show zero diff across P0 (M1); the KCP/KCPT
  defaulters are already pinned by existing tests.

Signed-off-by: William Rizzo <[email protected]>
Add a k3s-worker companion to TestBootstrapIntegration (which covers a
k0s control plane): assert the bootstrap data Secret is created, named
deterministically after the KairosConfig, owned by it (controller ref,
resolvable GVK, matching UID), labelled with the cluster name, and that
it carries a k3s worker cloud-config with the join token. This gives the
worker join-material path integration coverage before P0 refactors it.

Signed-off-by: William Rizzo <[email protected]>
The k0s generator returned the bare error from getControlPlaneLBEndpoint,
while the k3s generator wrapped it as "failed to get control plane LB
endpoint: %w". Wrap the k0s arm identically so the user-visible message
(copied into conditions and status.failureMessage) no longer depends on
the distribution. This changes a user-visible message for the k0s CAPK
control-plane LB-lookup failure path; the only other effect is dropping
the OQ-A error-prefix asymmetry the P0 render seam would otherwise carry.

Updates the characterisation assertion added in PR 1 accordingly.

Signed-off-by: William Rizzo <[email protected]>
…ribution

Introduce api/bootstrap/v1beta2/distribution.go as the api-layer source
of truth for the accepted distribution names: the DistributionK0s/K3s and
DefaultDistribution constants, EffectiveDistribution / SupportedDistributions
/ IsSupportedDistribution, and a static per-distribution rules table
(bootstrapDistributions) with validateK0sConfig / validateK3sConfig.

The KairosConfig webhook's validate() now delegates its inline distribution
name-check and worker-token switch to r.validateDistribution(); the
defaulter writes DefaultDistribution. Behaviour is byte-identical: the same
enum message, the same k0s and k3s worker-token messages, the same
aggregate order (name error then token error), and an unknown name still
falls back to the default's rules (OQ-C).

TestDistributionTableMatchesCRDEnum pins the table to the kubebuilder enum
and default in the generated CRD, so a later kubeadm enum edit and the
table change must land together. No CRD/RBAC/deepcopy regeneration: make
manifests generate produces no diff. The KCP and KCPT webhooks are
untouched (M1: zero diff); api/ adds no import of internal/.

Signed-off-by: William Rizzo <[email protected]>
…ution

Introduce the renderer's single distribution table (distributionTemplates),
keyed by the api distribution constants so it cannot drift from admission:
each row carries the template.New name, the generic (CAPV/CAPM3/fleet) and
KubeVirt (CAPK) template paths, and the providerID marker.

Render(distribution, data) is now the only place a distribution name
selects templates; ProviderIDMarker(distribution) returns the regeneration
substring and an ok flag. RenderK0sCloudConfig / RenderK3sCloudConfig
become permanent one-line wrappers over Render (OQ-E), so the 179 renderer
test lines and the golden entry points are unchanged — the renderer
goldens are byte-identical.

Adds renderer-level seam tests: Render/wrapper byte-equality, the
unsupported-distribution error, ProviderIDMarker, the table-agrees-with-
SupportedDistributions check, and TestRenderContract (S2).

Signed-off-by: William Rizzo <[email protected]>
…ibution source

Add internal/controllers/bootstrap/join.go: the JoinMaterialSource
interface (WorkerJoin only), JoinRequest, the redacting WorkerJoinMaterial,
and the built-in k0sJoinSource / k3sJoinSource over a client.Reader. The
reconciler gains an optional JoinSources map (nil default), so a missing
key uses the built-in over r.Client and every existing construction site
is unchanged; P1 injects a kubeadm source here.

The tokens.go helpers (tokenFromWorkerRef, tokenFromLegacyRef,
resolveWorkerToken, plus the extracted resolveK3sWorkerToken) become
client.Reader functions without moving; resolveToken stays as a delegating
wrapper for the tokens_test.go precedence tables, and a transitional
tokenFromWorkerRef method shim keeps secretref_namespace_test.go compiling
until the final cleanup commit. The control-plane join stays outside the
seam (M2): applyControlPlaneRenderData now calls resolveControlPlaneJoinToken
directly.

Behaviour is unchanged: the empty-token and missing-serverAddress error
strings move verbatim into the sources, and the goldens are byte-identical.
Adds redaction, source-selection and injection tests.

Signed-off-by: William Rizzo <[email protected]>
…ribution table

Add internal/controllers/bootstrap/render.go: the controller distribution
table (bootstrapDistribution{fillTemplateData, builtinJoin}), the merged
renderCloudConfig generator, and renderCarriesProviderID. The ~190-line
generateK0sCloudConfig / generateK3sCloudConfig bodies collapse into one
renderCloudConfig that keeps today's read order and defers the
distribution-only fields (k0s: WorkerToken, PodCIDR, ServiceCIDR; k3s:
K3sToken, K3sServerURL) and template selection to the table and
bootstrap.Render. generateCloudConfig's switch becomes a table lookup with
the same unsupported-distribution error; the two generators become
transitional one-line wrappers (removed in the final commit).

renderCarriesProviderID replaces both inline regeneration checks
(:401-411 and :595-603), using bootstrap.ProviderIDMarker with the same
unknown-distribution fallback to the default marker. EffectiveDistribution
replaces the remaining ""->k0s literals in generateCloudConfig,
applyControlPlaneRenderData and resolveControlPlaneJoinToken (S7:
resolveControlPlaneJoinToken stays in tokens.go). joinSourceFor now selects
the built-in through the table's builtinJoin.

Behaviour is byte-identical: all controller and renderer goldens are
unchanged, and the characterisation tests (error strings, ordering,
regeneration) still pass. Adds the controller table-agreement and
renderCarriesProviderID tests.

Signed-off-by: William Rizzo <[email protected]>
…nd tokenKind

Final P0 cleanup: delete the now-dead code the seam replaced. The 16
controller-test calls move from generateK0sCloudConfig/generateK3sCloudConfig
to renderCloudConfig with an explicit distribution; the two generator
wrappers are removed. resolveToken, the tokenKind type and its constants,
and the transitional tokenFromWorkerRef method shim are deleted. The
tokens_test.go precedence tables now drive the k0s/k3s join sources
directly, the control-plane-join case drives resolveControlPlaneJoinToken,
and secretref_namespace_test.go calls the package-level tokenFromWorkerRef.

No behaviour change: all controller and renderer goldens are byte-identical
and the full suite passes.

Signed-off-by: William Rizzo <[email protected]>
…e table

The field-to-fixture table in render_golden_test.go listed spec.Manifests
twice with contradictory claims: once as set in "every fixture" and once as
"not in YAML goldens; see note". The NOTE on goldenBaseSpec is authoritative —
Manifests is deliberately not set in the YAML-validated goldens because the
manifest render path emits a column-0 heredoc terminator that breaks the
enclosing block scalar (a pre-existing defect, out of scope). Remove the stale
"every fixture" row so the table matches the fixtures. (ADR 0010 P1 N1.)

Signed-off-by: William Rizzo <[email protected]>
…lane

ADR 0010 P1 ships kubeadm as a bootstrap worker-join distribution only; a
kubeadm KairosControlPlane is deferred to P2, so the KCP enum stays [k0s, k3s].
The validating webhook already refuses an explicit spec.distribution=kubeadm.
This closes the inherit-path gap before the bootstrap enum flip admits kubeadm on
the webhook-less KairosConfigTemplate (ADR 0010 P1 item 2): a KairosControlPlane
that references a kubeadm KairosConfigTemplate would otherwise inherit kubeadm and
try to persist it into spec.distribution, which the webhook rejects on every
reconcile — an error loop.

The controller now refuses both the inherited and (defence-in-depth) the explicit
kubeadm case with a clear Available/Ready False(Warning, UnsupportedDistribution)
condition and latched failure fields, never writing kubeadm into spec.distribution
and never requeuing into a loop. A later k0s/k3s template edit re-enqueues through
the normal watches.

Signed-off-by: William Rizzo <[email protected]>
ADR 0010 P1 (worker join). This is the atomic enum-flip commit: the moment
kubeadm enters SupportedDistributions() the api validation table, the renderer
template table, and the controller distribution table must all carry a kubeadm
row, and the agreement tests enforce that — so they land together.

api:
- DistributionKubeadm constant + a validateKubeadmConfig admission row. Static
  refusals: role control-plane (kubeadm is worker-only in P1), inline bootstrap
  token, unsafeSkipCAVerification, discovery.file, any "{{" anywhere in the
  kubeadm block (Jinja is not expanded on Kairos and the block is written into an
  on-node file), and a non-DNS-1123 nodeRegistration.name.
- spec.kubeadm embeds the INDIVIDUAL CAPI kubeadm v1beta2 JoinConfiguration type
  (OQ-5), not KubeadmConfigSpec and not ClusterConfiguration. The embedding pulls
  k8s.io/cluster-bootstrap and k8s.io/component-base into go.sum — both already in
  the build list at the CAPI-pinned v0.35.4 (go mod tidy only records the sums).
- KairosConfigSpec.distribution enum flipped to [k0s, k3s, kubeadm] and
  status.bootstrapTokenID (non-secret token-id surface) added; make manifests +
  make generate run in this commit (TestDistributionTableMatchesCRDEnum enforces).
  The CRD grows ~36KB -> ~70KB, under the client-side-apply limit. The KCP enum is
  unchanged. A schema-diff guard (TestKubeadmSchemaDiff, OQ-5) hashes the served
  spec.kubeadm sub-schema so a CAPI bump that silently changes it fails loudly.

renderer (internal/bootstrap stays CAPI-type-free):
- a kubeadm templateSet row + a worker-only template that writes the marshalled
  JoinConfiguration, the providerID kubeadm patch, and a oneshot join unit named
  kairos-kubeadm-post-bootstrap.service (the providerID marker), and enables sshd
  (TestRenderContract / the R4 regeneration contract require both).
- TemplateData.Kubeadm is a plain-string struct; Render refuses a kubeadm
  control-plane render (the seam has no control-plane hook, item 5);
  validateTemplateData rejects "{{" in the block as a render-time last line.

controller:
- a kubeadm row {fillKubeadm, builtinJoin: nil}: the worker JoinMaterialSource is
  injected via JoinSources (dependencies only main.go has), so joinSourceFor now
  HARD-FAILS when builtinJoin==nil and no source is injected instead of silently
  falling back to the k0s built-in (N2). fillKubeadm copies the minted join
  material and builds the providerID patch from the resolved providerID.
- WorkerJoinMaterial gains a redacted kubeadm block (JoinConfiguration stays
  secret; only the non-secret token ID is surfaced in String/GoString).

The kubeadm JoinMaterialSource (trust check + token mint) lands in the next commit.

Signed-off-by: William Rizzo <[email protected]>
…rified trust check

ADR 0010 P1 security core. The kubeadm worker JoinMaterialSource
(internal/controllers/bootstrap/kubeadm.go), wired via JoinSources["kubeadm"] in
main.go with a clustercache-backed, ClusterFilter-gated workload client, an
uncached management reader, and the operator-level trust allowlist.

Trust check (ADR 0010 P1 step 1), every failure terminal with no token minted:
- resolve Cluster.spec.controlPlaneRef and match it on GROUP AND KIND against the
  allowlist (KubeadmControlPlane, KamajiControlPlane, and a KairosControlPlane whose
  spec.distribution is kubeadm); any other kind only via the operator-level
  --kubeadm-extra-controlplane-kinds flag, never a per-Cluster field (OQ-9);
- require the <cluster>-kubeconfig Secret to carry a controller ownerReference whose
  UID equals the resolved control-plane object (refuses node-pushed/planted
  kubeconfigs);
- require the kubeconfig server to equal https://<Cluster.spec.controlPlaneEndpoint>;
- take the CA pin from the kubeconfig's certificate-authority-data (SPKI sha256
  hashes) and, when <cluster>-ca exists, cross-check its tls.crt via an UNCACHED
  read, parsing only the cert (Kamaji's unowned default CA and an upstream-KCP
  user CA both pass; a mismatch is refused).

Token + render:
- mint a bootstrap token in the WORKLOAD cluster (clustercache), 15m TTL, CABPK
  groups/usages/keys; the token value is never logged (k8s.io/cluster-bootstrap, the
  canonical source, is already pulled in by the spec.kubeadm embedding);
- build the JoinConfiguration (nodeRegistration.name = Machine hostname, discovery
  bootstrapToken with the token + endpoint + caCertHashes, the
  node.cluster.x-k8s.io/uninitialized taint) and marshal it version-aware; providerID
  flows through a kubeadm kubeletconfiguration patch, not the JoinConfiguration;
- MachineDeployment workers only — a MachinePool owner is refused.

Terminal-vs-not-ready contract (item 6): a typed bootstrapNotReadyError wraps every
transient wait (control plane not initialized, kubeconfig absent, workload
unreachable/API timeout). reconcileBootstrapData maps it to a 10s requeue next to
errTokenNotReady and leaves conditions alone, so a watch/clustercache blip never
parks the KairosConfig.

Token refresh (item 7): the optional workerTokenRefresher, found by type assertion,
re-mints while the Machine has no nodeRef — rotating once the workload token Secret
(keyed by the stored non-secret ID) is gone or past half its TTL — with a 24h hard
cap that raises a BootstrapTokenRefreshCapExceeded condition and a documented MHC
nodeStartupTimeout backstop.

Status (item 8): the reconciler writes the NON-SECRET token ID to
status.bootstrapTokenID; the secret half lives only in the data Secret.
WorkerJoinMaterial's redaction stays total (kubeadm JoinConfiguration REDACTED, only
the token ID surfaced).

Signed-off-by: William Rizzo <[email protected]>
Exercises the kubeadm worker join against a real apiserver (ADR 0010 P1 test
plan). A Cluster whose controlPlaneRef is an owner-verified control plane and whose
<cluster>-kubeconfig carries a matching controller owner UID mints a bootstrap
token in a fake workload cluster and produces bootstrap data: status.dataSecretName
and the non-secret status.bootstrapTokenID are set, the rendered cloud-config
carries the JoinConfiguration and the providerID-marker unit, and the token Secret
is created in the workload kube-system. A kubeconfig whose controller owner UID does
NOT match the control plane is refused — no data Secret, no token id, and a
BootstrapDataSecretGenerationFailed condition whose message names the owner-check
failure.

The trust anchor is a throwaway schemaless TestControlPlane kind trusted via the
operator-level allowlist flag, which also covers the operator opt-in path (OQ-9)
without depending on the full KubeadmControlPlane schema.

Signed-off-by: William Rizzo <[email protected]>
…inds

The kubeadm worker-join trust check (ADR 0010 P1) does a get on the Cluster's
control-plane object to verify the <cluster>-kubeconfig's controller-owner UID, but
the bootstrap manager had no controlplane.cluster.x-k8s.io grant. On a bootstrap-only
(--controllers=bootstrap) Deployment — which has none of the control-plane manager's
RBAC — that Get returns Forbidden, which the source maps to not-ready and requeues
forever, so no kubeadm worker ever joins. envtest masked this because it runs with an
admin client and enforces no RBAC.

Add a least-privilege, read-only, enumerated grant on kamajicontrolplanes,
kubeadmcontrolplanes, and kairoscontrolplanes (the P1 trust anchors) in the bootstrap
package. make manifests lands it in both the aggregated config/rbac/role.yaml and the
bootstrap split overlay config/rbac/bootstrap/role.yaml (ADR 0007). An operator who
widens the allowlist via --kubeadm-extra-controlplane-kinds must supply RBAC for that
group themselves — we cannot grant kinds in groups unknown at build time.

TestBootstrapRoleGrantsControlPlaneRead parses the generated bootstrap role and
asserts the get so a future make-manifests cannot silently drop it (the
envtest-masks-RBAC guard the security review required).

Signed-off-by: William Rizzo <[email protected]>
…ane kinds flag

Add the user-facing documentation for KairosConfig distribution: kubeadm
(worker-only joins to a Kamaji or KubeadmControlPlane control plane).

- docs/KUBEADM_WORKERS.md: how the join works, prerequisites (control plane,
  node image at the exact Machine.spec.version, CNI), admission refusals, token
  lifetime and the refresh cap, MachineHealthCheck guidance, the
  KairosControlPlane UnsupportedDistribution refusal, and troubleshooting.
- config/samples/kubeadm: a worker KairosConfigTemplate, MachineDeployment and
  MachineHealthCheck. The password comes from a Secret created by command.
- docs/API_REFERENCE.md: spec.distribution, spec.kubeadm, status.bootstrapTokenID,
  the new condition reasons, and the persistence and token-requirement notes.
- docs/INSTALL.md: the --kubeadm-extra-controlplane-kinds manager flag (format,
  default allowlist, RBAC, how to set it on each install path).
- README.md, docs/TESTING.md: distribution list, target versions, and the
  verification status (unit and envtest only).

Signed-off-by: William Rizzo <[email protected]>
…ent state

We publicly promised in the #4920 reply that a Kairos kubeadm worker "runs only on
an installed (active/passive) boot, never on live/recovery/auto-reset/in-RAM" and
"a node that has joined never joins again on reboot; a failed join is not retried on
the node". The join unit previously gated only on
ConditionPathExists=!/etc/kubernetes/kubelet.conf, which fails open on a v4.2+ in-RAM
boot (/etc is a tmpfs overlay, KD-59).

Mirror the ADR 0010 "Node side" / C5 pattern in the join unit:
- a positive OR allowlist ConditionPathExists=|/run/cos/active_mode /
  |/run/cos/passive_mode; ConditionPathExists=!/run/cos/in_ram_mode;
- ConditionPathIsMountPoint for /etc/kubernetes and /var/lib/kubelet so a join never
  runs on ephemeral/tmpfs state;
and in the join script:
- a persistent "attempted" marker written on COS_PERSISTENT (/var/lib/kairos) BEFORE
  kubeadm runs and a "completed" marker after a verified join; a later boot that finds
  attempted-without-completion refuses to retry on-node (remediation replaces the
  Machine);
- the exact-version check runs before the attempted marker so a version mismatch does
  not burn the single attempt;
- on failure, credential cleanup only (rm the token-bearing bootstrap-kubelet.conf);
  a worker has no etcd/PKI to reset.

The render test asserts every gate condition and marker. k0s/k3s templates and
goldens are untouched (there are no kubeadm goldens; the kubeadm render is covered by
substring assertions).

Signed-off-by: William Rizzo <[email protected]>
…ct, not a pin

The kubeadm trust check hardcoded v1beta2 when reading the Cluster's control-plane
object (with a dead retry that pinned the same version). That silently breaks the
primary P1 target KamajiControlPlane, which is served at v1alpha1, and the retry did
nothing its comment claimed (security review NIT).

Resolve the ContractVersionedObjectReference through
external.GetObjectFromContractVersionedRef, which reads the referenced kind's CRD
contract label to pick the served API version — exactly how CAPI core resolves the
same ref, so Kamaji (v1alpha1) and KubeadmControlPlane (v1beta2) both resolve. The
resolver is injectable (default = the CAPI helper) so unit tests stub it and need no
installed CRD / contract labels; the envtest trust anchor carries the
cluster.x-k8s.io/v1beta2 contract label and exercises the real resolver.

Also document that TokenNeedsRefresh is read-only, reachable only after a prior
successful trust check (non-empty token ID in status), mints nothing, and still runs
behind the coarse clustercache ClusterFilter (security review NIT).

Signed-off-by: William Rizzo <[email protected]>
…ly joins, and neutralize user-facing messages

Three tech-writer/security cleanups on the kubeadm worker path:

1. Coarse cluster filter was over-broad. KubeadmTrustedClusterFilter matched
   group+kind and included KairosControlPlane, so the clustercache would connect to
   k0s/k3s KairosControlPlane clusters (the fine per-request check still refused them,
   so it was connect-only, never a mint). Split off coarseTrustedControlPlaneKinds
   (KubeadmControlPlane + KamajiControlPlane + operator-flag kinds) that the filter
   uses, excluding KairosControlPlane while kubeadm control planes are unsupported.
   The fine WorkerJoin check keeps the KairosControlPlane(distribution==kubeadm)
   branch for forward-compat. Adds TestKubeadmTrustedClusterFilter (none existed).

2. A control-plane join is now refused at admission. The joinConfiguration doc said
   controlPlane must be nil but nothing enforced it; validateKubeadmConfig now rejects
   a non-nil joinConfiguration.controlPlane (worker-only). Adds a webhook test case.

3. User-facing messages no longer cite internal decision records or hint at the
   embargoed topic. Reworded to self-contained guidance:
   - webhook role refusal (distribution.go): dropped the "(ADR 0010 P2)" tail.
   - KCP refusal failureMessage: dropped "(ADR 0010 P1)".
   - render control-plane refusal (template.go): dropped "(ADR 0010 P1)".
   - kubeadm control-plane distribution error: dropped "(ADR 0010)".
   - kubeconfig owner-mismatch error: "the <cluster>-kubeconfig Secret must be owned
     by the cluster's control plane; refusing to mint a token" (was "a node-pushed or
     planted kubeconfig is not trusted on the kubeadm path").
   Updated the unit + envtest assertions for the reworded owner-mismatch string, and
   corrected the Kamaji served-version comment (v1alpha2, not v1alpha1).

Signed-off-by: William Rizzo <[email protected]>
The distribution, kubeadm, and joinConfiguration field doc-comments cited internal
decision-record IDs (ADR 0010 P1/P2/OQ-5) and an unfollowable "see
validateKubeadmConfig" reference, which surface verbatim in `kubectl explain`.
Reworded noun-first with the enum values and defaults spelled out and no internal
IDs. Also corrected the pre-existing misleading KubernetesVersion comment ("version
to install"): it is informational; the authoritative version for a kubeadm worker is
the owning Machine's spec.version, which the rendered join checks against the image's
kubeadm. The status.bootstrapTokenID description lost its ADR tag too.

make manifests + make generate regenerate the two bootstrap CRDs; the only diff is
the reworded descriptions (no enum/field/type/default change). The kubeadm
schema-diff digest is updated to match (descriptions are part of the served schema it
guards).

Signed-off-by: William Rizzo <[email protected]>
A CA mismatch has a legitimate benign cause (CA rotation or re-pin), so the
message now points the operator at inspecting and re-pinning rather than
asserting tampering.

Signed-off-by: William Rizzo <[email protected]>
…d join credentials

Three findings from the pre-merge security review of the kubeadm worker join.

The endpoint-equality leg of the trust check compared strings. Cluster API
writes the kubeconfig's server from APIEndpoint.String, which is
net.JoinHostPort and therefore brackets an IPv6 literal, so a cluster with an
IPv6 control-plane endpoint published `https://[fd00::1]:6443` while the check
demanded `https://fd00::1:6443`. Every such cluster was refused, terminally,
with a message stating that the kubeconfig's server does not equal the Cluster
endpoint -- which reads like a planted-credential incident rather than a
formatting mismatch. A trailing slash, a host differing in case, and an omitted
default port failed the same way. The comparison now parses both sides and
compares scheme, host case-insensitively, and port with the scheme's default.

The same hand-formatting produced the JoinConfiguration's apiServerEndpoint, so
an IPv6 endpoint would have reached kubeadm unbracketed even once the check
passed. Both sites now go through APIEndpoint.String, so they cannot drift.

discovery.tlsBootstrapToken was neither refused by the webhook nor cleared by
the renderer, although its sibling discovery.bootstrapToken.token is refused on
the grounds that tokens never live in the spec. It defaults to the bootstrap
token but may be overridden, so a value there became the kubelet's TLS-bootstrap
credential in preference to the minted one: plaintext in the spec, copied into
the rendered userdata on /oem, and outside the short TTL and refresh the
controller's own token relies on. It is now forbidden at admission and cleared
at render time, which also makes the invariant in docs/KUBEADM_WORKERS.md true
as written.

joinConfiguration.controlPlane was enforced only at admission. Because no
webhook is registered for KairosConfigTemplate, every kubeadm refusal on a
template is caught one object later, when Cluster API core derives the
KairosConfig. That is fail-closed, but the repo rule is webhook and render-time,
so the renderer now nils it as it already does for file discovery.

The docs and sample claimed these fields are "rejected at admission" without
qualification. Both now say which object the webhook covers and which refusals
are additionally enforced when the cloud-config is rendered.

Tests: the four endpoint shapes that used to be refused are now accept cases, a
non-controller ownerRef is pinned as a terminal refusal, the webhook refusal for
tlsBootstrapToken is pinned, and a render-time test asserts a spec-supplied
tlsBootstrapToken and controlPlane never reach the rendered JoinConfiguration
while the minted token still does. Each was confirmed to fail with its fix
reverted.

go mod tidy promotes blang/semver/v4 and k8s.io/cluster-bootstrap to direct
requirements; both were already imported directly.

Signed-off-by: William Rizzo <[email protected]>
@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.90547% with 250 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.35%. Comparing base (bb7b742) to head (1a9a84e).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
internal/controllers/bootstrap/kubeadm.go 67.08% 65 Missing and 40 partials ⚠️
main.go 0.00% 54 Missing ⚠️
...l/controllers/bootstrap/kairosconfig_controller.go 47.61% 28 Missing and 5 partials ⚠️
internal/controllers/bootstrap/render.go 83.45% 17 Missing and 6 partials ⚠️
api/bootstrap/v1beta2/zz_generated.deepcopy.go 0.00% 16 Missing ⚠️
...lers/controlplane/kairoscontrolplane_controller.go 55.55% 5 Missing and 3 partials ⚠️
api/bootstrap/v1beta2/distribution.go 96.55% 3 Missing and 1 partial ⚠️
internal/bootstrap/validate.go 20.00% 2 Missing and 2 partials ⚠️
internal/controllers/bootstrap/join.go 94.73% 1 Missing and 1 partial ⚠️
internal/bootstrap/template.go 94.44% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #141      +/-   ##
==========================================
+ Coverage   30.29%   36.35%   +6.05%     
==========================================
  Files          52       60       +8     
  Lines        6982     7678     +696     
==========================================
+ Hits         2115     2791     +676     
+ Misses       4635     4607      -28     
- Partials      232      280      +48     
Flag Coverage Δ
unittests 36.35% <68.90%> (+6.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

patrostkowski and others added 2 commits October 12, 2026 12:34
… kubeadm joins

Drop the kubeconfig server == controlPlaneEndpoint check, which refuses
Kamaji clusters publishing admin.svc; trust stays on the owner UID and CA pin.
Use spec.serverAddress as the join endpoint when set.

Signed-off-by: Patryk Rostkowski <[email protected]>
The preceding commit drops the kubeconfig server comparison outright, on the
correct observation that a control-plane provider may publish an internal
address there: Kamaji publishes admin.svc when spec.network.advertiseAddress is
set, and our equality check refused those clusters terminally. That part of the
diagnosis stands and the live evidence behind it is the only way it could have
been found.

Dropping the check entirely goes further than the fix needs to, because the
owner reference is not an anchor. ownerReferences are user-settable and a UID is
readable, so anyone who can create Secrets in the Cluster's namespace can forge
the owner leg. That left the server equalling the Cluster endpoint and the CA
matching a separately-read <cluster>-ca as the only two independent anchors, and
ADR 0010 P1 requires the first of them. With neither, a planted
<cluster>-kubeconfig steers this controller's own client -- which carries
workload-cluster admin credentials -- at any address reachable from the
management network.

So require at least one of the two rather than both. The server may differ from
the Cluster endpoint when <cluster>-ca exists and its tls.crt matches the
kubeconfig's CA. Kamaji satisfies that: it publishes <cluster>-ca alongside the
kubeconfig, which is why the accept case added by the preceding commit already
carries one. A kubeconfig whose server differs with no <cluster>-ca to anchor it
is refused, with a message naming both ways out.

Also restores the IPv6 fix the rebase reverted: the no-serverAddress path of
kubeadmJoinEndpoint was hand-formatting host:port again, so an IPv6 endpoint
would have reached kubeadm unbracketed. It goes back through
APIEndpoint.String, which is what the serverAddress path already does via
net.JoinHostPort.

Tests: the Kamaji accept case stands unchanged, and its counterpart -- server
differs, no <cluster>-ca -- is pinned as a terminal refusal. Confirmed to fail
with the conditional disabled.

Signed-off-by: William Rizzo <[email protected]>
@patrostkowski

patrostkowski commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

thanx @wrkode! answering to your comment from this PR

The conditional makes sense, and I am ok with it. I have tested https://github.com/wrkode/cluster-api-provider-kairos/tree/feat/kubeadm-kamaji-serveraddress end to end on a live Kamaji management cluster.

Results

Case kubeconfig server Anchor that held Join endpoint CA pin vs <cluster>-ca Workers
advertiseAddress + serverAddress https://<name>.<ns>.svc:6443 (differs from the endpoint) <cluster>-ca serverAddress match 2/2 joined, Ready
Vanilla (no advertiseAddress, no serverAddress) equals Cluster.spec.controlPlaneEndpoint endpoint Cluster endpoint match 2/2 joined, Ready
CAPI-style BYO CA + advertiseAddress, no serverAddress https://<name>.<ns>.svc:6443 (differs from the endpoint) <cluster>-ca Cluster endpoint match 2/2 joined, Ready

Every KairosConfig went Ready, and the controller logged no CA mismatch or refusal. While the tenant API servers were coming up, it only waited on WorkloadClusterUnreachable.

To confirm your two points:

  • advertiseAddress + serverAddress is now accepted through the CA anchor. The bootstrap data carries the serverAddress endpoint, and the pin matches <cluster>-ca.
  • Vanilla behaves as before, accepted through the endpoint anchor.

can <cluster>-ca be absent or carry something else? - not in any configuration I could find? I checked the Kamaji control-plane provider source and confirmed its behaviour live:

  • The provider names the TenantControlPlane after the KamajiControlPlane. When that name equals the Cluster name (the common case), <cluster>-ca is Kamaji's own <tcp>-ca Secret, owned by the TenantControlPlane (cases 1 and 2).
  • When the names differ, the provider writes a copy into <cluster>-ca, owned by the KamajiControlPlane, with tls.crt/tls.key taken from the TCP CA (case 3).
  • Kamaji honours a BYO CA only as a pre-created <tcp>-ca with a valid ca.crt/ca.key pair. It adopts that pair and fills in tls.crt/tls.key from it, so <cluster>-ca still carries the tenant CA.
  • A CAPI-style BYO CA (a pre-created <cluster>-ca with only tls.crt/tls.key) is not honoured. In case 3 I pre-created exactly that, with the KCP name differing from the Cluster name. The provider overwrote it with Kamaji's generated CA, and the join pin was Kamaji's CA, not the one I supplied. That's Kamaji behaviour rather than something for this PR, and it keeps your anchor consistent.

Let me know if you need any additional assitance!

The trust check now accepts a kubeconfig whose server differs from the Cluster
endpoint when <cluster>-ca anchors its CA, so operators have a reason to care
what lives in that Secret. Under Kamaji they cannot choose it: a bring-your-own
CA is honoured only as a pre-created <tcp>-ca holding a ca.crt/ca.key pair, and
a CAPI-style <cluster>-ca carrying only tls.crt/tls.key is overwritten with
Kamaji's generated CA. The join then pins Kamaji's CA rather than the supplied
one, silently.

Found by @patrostkowski while validating the anchor against a live Kamaji
cluster: he pre-created exactly that Secret, with the KamajiControlPlane name
differing from the Cluster name, and watched the provider replace it. It is
Kamaji's behaviour rather than ours, and it leaves the anchor intact, because
<cluster>-ca still carries the tenant CA in every configuration.

Signed-off-by: William Rizzo <[email protected]>
ADR 0010 OQ-8 permits shipping the kubeadm worker path as a preview before C5
lands only if its docs forbid putting an upstream KubeadmControlPlane cluster in
the same namespace as a k0s or k3s cluster. That condition was agreed and then
never written down, so state it where someone enabling the feature will read it.

The reason is that upstream KCP adopts a pre-existing <cluster>-ca or
<cluster>-kubeconfig Secret instead of insisting on creating one. An adopted
Secret carries a legitimate controller owner reference, so the trust check
cannot tell it from the real thing: the check catches a planted Secret, not an
adopted one. What makes that reachable is the unscoped secrets create on the
Role the k0s/k3s control-plane kubeconfig push uses, so a node compromised in a
namespace shared with an upstream-KCP cluster can plant for its neighbour.
Separate namespaces remove the adjacency, which is the only mitigation until
that push path is fixed.

Kamaji is not exposed the same way: it overwrites planted data and sets itself
as controller owner, so the prohibition names upstream KCP specifically.

Also corrects the verification status, which still claimed nothing had been run
against a live cluster. The Kamaji path has now been validated end to end in
three configurations with real Kairos images; upstream KCP still has not. The
scope table said both kinds were equally unvalidated and now distinguishes them.

Signed-off-by: William Rizzo <[email protected]>
@wrkode

wrkode commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

@patrostkowski, changes rolled in with little adjustments and we're good to merge.
Thanks a million for your contribution and help

@wrkode
wrkode merged commit b19dd35 into kairos-io:main Oct 13, 2026
11 checks passed
lexfrei added a commit to lexfrei/cluster-api-provider-kairos that referenced this pull request Oct 13, 2026
runcmd started the join unit with a plain `systemctl enable --now`.
The unit is a oneshot whose discovery retries can run for minutes, and
systemctl waited for it to finish, so the cloud-config stage that runs
runcmd sat in "Still running stage" for the whole join. Users and SSH
keys from the same cloud-config were applied only after it, so the
node could not be reached while the join was failing.

`--no-block` queues the start job and returns. Nothing after runcmd
depends on the join having finished, and the unit's ordering and
conditions are evaluated by systemd either way.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
wrkode added a commit that referenced this pull request Oct 15, 2026
* fix(build): build the image with Go 1.26.9

The builder image was pinned to golang:1.26-alpine at Go 1.26.8. Go 1.26.9,
released 2026-10-08, fixes thirteen standard-library advisories, most of them
in code this provider runs: the webhook, metrics and health servers, and the
HTTP/2 client to the API server.

- net/http and HTTP/2: GO-2026-6603, GO-2026-6605, GO-2026-6609,
  GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6613 and GO-2026-6617
- crypto/tls: GO-2026-6607
- net/textproto and mime/multipart: GO-2026-6608
- html/template: GO-2026-6599 and GO-2026-6600
- os, Windows only: GO-2026-6604

#143 cleared the golang.org/x/net copies of the HTTP/2 issues, but the binary
also links the standard library's own copy, which only a new toolchain fixes.

The new pin is the multi-arch index digest that golang:1.26.9-alpine resolves
to; the linux/amd64 and linux/arm64 images in it both carry Go 1.26.9.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>

* docs(release): prepare v0.1.4

v0.1.4 is a fix release for the v0.1 series, cut from main just before the
kubeadm worker distribution (#141) merged. It ships the control-plane and
bootstrap fixes #109 to #120 and the Go security updates, and none of the
kubeadm work, which goes into v0.2.0.

- docs/release-notes/v0.1.4.md: the release body that release.yaml requires.
- CHANGELOG.md: the v0.1.4 entry.
- docs/UPGRADING.md: a v0.1.3 to v0.1.4 section. Its main content is the
  check to run before upgrading for references to another namespace, which
  #119 now rejects. It also covers the clusterctl.yaml URL fix from #111 and
  the behaviour changes to expect.
- README, INSTALL, the quickstarts and the CAPM3 samples: install URLs and
  the latest-release pointer move to v0.1.4.

The "Last verified against" headers stay at v0.1.3, because nothing in them
was re-verified for this release.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>

---------

Signed-off-by: William Rizzo <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
lexfrei added a commit to lexfrei/cluster-api-provider-kairos that referenced this pull request Oct 15, 2026
kubeadm refuses any flag next to --config that is not on its short
allowlist, and --patches is not on it. The join unit passed --patches
whenever a providerID patch was rendered, so every worker whose
providerID is known at render time failed with "can not mix '--config'
with arguments [patches]" and never joined.

The directory now travels in JoinConfiguration.patches.directory,
defaulting to /etc/kubernetes/kairos-patches and honouring a directory
set in the spec. The unit creates it before it writes the attempted
marker, because kubeadm fails on a missing directory and a failing
mkdir must not use up the node's only join attempt. The same directory
is where users place their own kubelet configuration patches through
spec.files. The renderer checks it with ValidatePatchesDirectory, the
rule admission applies, as the backstop for configs derived from a
KairosConfigTemplate.

The new test renders the worker cloud-config, runs the join script
against a kubeadm stub and checks the resulting command line against
kubeadm's flag rules. With KUBEADM_BIN pointing at a real kubeadm, the
stub hands the call to it with every join phase skipped and the
rendered configuration is also checked with kubeadm config validate.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
lexfrei added a commit to lexfrei/cluster-api-provider-kairos that referenced this pull request Oct 15, 2026
yip loads runcmd, write_files and users from a cloud-config as the
first step of the boot stage and appends the stages.boot steps after
it. The "Configure DNS resolvers" step lived in stages.boot, so it ran
only after runcmd had started the join or post-bootstrap unit. Since
that start waited for the unit, the custom resolvers never applied to
a kubeadm join, a k3s or k0s post-bootstrap run, or anything they
resolved; a control-plane endpoint resolvable only through those
servers could not be reached on the first boot.

The step now lives in boot.before. kairos-agent runs boot.before,
boot and boot.after one after another, so the resolvers are written
before any runcmd entry runs.

Running earlier exposes a race the old placement hid. On Hadron
v0.5.3 /etc/resolv.conf is a symlink to
/run/systemd/resolve/resolv.conf, which systemd-resolved regenerates
when a DHCP lease brings DNS, and the yip in that release writes
through the link. cos-setup-boot is ordered only after
systemd-journald.socket. On a live node the first lease rewrote the
file at 13:23:00.25 and cos-setup-boot started at 13:23:00.66, so a
slower lease would land after the resolvers and replace them. The
step now writes a staged file and the next step renames it over
/etc/resolv.conf, which replaces the link with a regular file that
resolved does not touch.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)")
Fixes: 51f3504 ("fix(controlplane): eliminate SSH from kubeconfig retrieval; extend CAPV with node-push pattern (KD-3b) (kairos-io#55)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
lexfrei added a commit to lexfrei/cluster-api-provider-kairos that referenced this pull request Oct 15, 2026
runcmd started the join unit with a plain `systemctl enable --now`.
The unit is a oneshot whose discovery retries can run for minutes, and
systemctl waited for it to finish, so the cloud-config stage that runs
runcmd sat in "Still running stage" for the whole join. Users and SSH
keys from the same cloud-config were applied only after it, so the
node could not be reached while the join was failing.

`--no-block` queues the start job and returns. Nothing after runcmd
depends on the join having finished, and the unit's ordering and
conditions are evaluated by systemd either way.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
wrkode added a commit that referenced this pull request Oct 16, 2026
* fix(build): build the image with Go 1.26.9

The builder image was pinned to golang:1.26-alpine at Go 1.26.8. Go 1.26.9,
released 2026-10-08, fixes thirteen standard-library advisories, most of them
in code this provider runs: the webhook, metrics and health servers, and the
HTTP/2 client to the API server.

- net/http and HTTP/2: GO-2026-6603, GO-2026-6605, GO-2026-6609,
  GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6613 and GO-2026-6617
- crypto/tls: GO-2026-6607
- net/textproto and mime/multipart: GO-2026-6608
- html/template: GO-2026-6599 and GO-2026-6600
- os, Windows only: GO-2026-6604

#143 cleared the golang.org/x/net copies of the HTTP/2 issues, but the binary
also links the standard library's own copy, which only a new toolchain fixes.

The new pin is the multi-arch index digest that golang:1.26.9-alpine resolves
to; the linux/amd64 and linux/arm64 images in it both carry Go 1.26.9.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>
(cherry picked from commit 008c6ae)

* docs(release): prepare v0.1.4

v0.1.4 is a fix release for the v0.1 series, cut from main just before the
kubeadm worker distribution (#141) merged. It ships the control-plane and
bootstrap fixes #109 to #120 and the Go security updates, and none of the
kubeadm work, which goes into v0.2.0.

- docs/release-notes/v0.1.4.md: the release body that release.yaml requires.
- CHANGELOG.md: the v0.1.4 entry.
- docs/UPGRADING.md: a v0.1.3 to v0.1.4 section. Its main content is the
  check to run before upgrading for references to another namespace, which
  #119 now rejects. It also covers the clusterctl.yaml URL fix from #111 and
  the behaviour changes to expect.
- README, INSTALL, the quickstarts and the CAPM3 samples: install URLs and
  the latest-release pointer move to v0.1.4.

The "Last verified against" headers stay at v0.1.3, because nothing in them
was re-verified for this release.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>
(cherry picked from commit 33e2d47)

* docs: say that the kubeadm worker distribution is not in v0.1.4

v0.1.4 was cut from release-0.1, which branched from main just before #141,
so it does not contain kubeadm workers. On main, the README now names v0.1.4
as the latest release and then describes `distribution: kubeadm` directly
after it, which reads as though that release supports it.

- README: mark the kubeadm paragraph as not in v0.1.4, planned for v0.2.0.
- INSTALL: say the same for `--kubeadm-extra-controlplane-kinds`. A v0.1.4
  manager parses its flags with the standard library, so given this flag it
  exits with "flag provided but not defined" instead of starting.
- KUBEADM_WORKERS: the version line no longer says the feature follows
  v0.1.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>

* fix(build): pin kustomize so the release job can render manifests

The v0.1.4 release job failed at "Render release manifests": the Makefile
installed kustomize with `@latest`, which now resolves to v5.8.3, and v5.8.3
requires go >= 1.27. The job runs the go.mod toolchain (1.26.4) with
GOTOOLCHAIN=local, so `go install` refused it and no GitHub release was
published, although the image had already been pushed.

Pin KUSTOMIZE_VERSION to v5.8.1, the version that rendered the v0.1.3
release artifacts, so release manifests are reproducible and no longer
depend on whatever kustomize was published last.

Verified with a clean bin/: `make release-manifests` renders all five
artifacts, sha256sums verify, every image ref is digest-pinned, and the
resource kinds and counts match v0.1.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>

---------

Signed-off-by: William Rizzo <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
wrkode added a commit that referenced this pull request Oct 17, 2026
…on (#148)

* fix(api): refuse an unsafe kubeadm patches.directory

kubeadm walks the patches directory recursively and applies every
patch it finds there as root during the join. Admission accepted any
value: `/` made kubeadm walk the whole root filesystem, a
world-writable directory such as /tmp or /dev/shm let any local user
drop a kubelet patch, and a value with a newline, CR or NUL was
written into the node's JoinConfiguration.

ValidatePatchesDirectory now requires an absolute, clean path under
/etc/kubernetes/ without control characters. It must not be
/etc/kubernetes/manifests, the kubelet's static pod directory, or a
directory under it. The function is exported so the renderer can
apply the same rule to configs derived from a KairosConfigTemplate,
which has no webhook, and the two checks cannot drift apart.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* fix(bootstrap): name the kubeadm patches directory in JoinConfiguration

kubeadm refuses any flag next to --config that is not on its short
allowlist, and --patches is not on it. The join unit passed --patches
whenever a providerID patch was rendered, so every worker whose
providerID is known at render time failed with "can not mix '--config'
with arguments [patches]" and never joined.

The directory now travels in JoinConfiguration.patches.directory,
defaulting to /etc/kubernetes/kairos-patches and honouring a directory
set in the spec. The unit creates it before it writes the attempted
marker, because kubeadm fails on a missing directory and a failing
mkdir must not use up the node's only join attempt. The same directory
is where users place their own kubelet configuration patches through
spec.files. The renderer checks it with ValidatePatchesDirectory, the
rule admission applies, as the backstop for configs derived from a
KairosConfigTemplate.

The new test renders the worker cloud-config, runs the join script
against a kubeadm stub and checks the resulting command line against
kubeadm's flag rules. With KUBEADM_BIN pointing at a real kubeadm, the
stub hands the call to it with every join phase skipped and the
rendered configuration is also checked with kubeadm config validate.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (#141)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* test(bootstrap): run the kubeadm worker join against a real kubeadm

A copy of kubeadm's flag rules only catches the mistakes it was written
for. make test-kubeadm downloads kubeadm and runs the join command test
with it, so kubeadm itself parses the rendered command line and loads
the rendered JoinConfiguration. Every join phase is skipped and
--dry-run is set, so nothing touches the host or the network even if a
later kubeadm adds a phase the help scrape misses. The fixture names the
CRI socket, so kubeadm does not probe the host and fail where several
CRI sockets exist.

The sha256 of each kubeadm version is pinned in hack/kubeadm.sha256.
A sum fetched from the same host as the binary would not catch a
replaced binary.

CI runs it in its own job against kubeadm v1.30.14, which takes
JoinConfiguration v1beta3, and v1.36.2, which takes v1beta4. The
workflow token is now read-only: the job runs a downloaded binary, and
the repository default token can write.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* ci: make /etc/kubernetes writable for the kubeadm-join job

kubeadm join --dry-run creates /etc/kubernetes/tmp on the host whatever
the test root is, so the job failed as the non-root runner user with
'mkdir /etc/kubernetes: permission denied'. Create the directory owned by
the runner user before the test, and document the same for local runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: William Rizzo <[email protected]>

---------

Signed-off-by: Aleksei Sviridkin <[email protected]>
Signed-off-by: William Rizzo <[email protected]>
Co-authored-by: William Rizzo <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
wrkode added a commit that referenced this pull request Oct 22, 2026
…#149)

* fix(bootstrap): apply spec.dnsServers before runcmd

yip loads runcmd, write_files and users from a cloud-config as the
first step of the boot stage and appends the stages.boot steps after
it. The "Configure DNS resolvers" step lived in stages.boot, so it ran
only after runcmd had started the join or post-bootstrap unit. Since
that start waited for the unit, the custom resolvers never applied to
a kubeadm join, a k3s or k0s post-bootstrap run, or anything they
resolved; a control-plane endpoint resolvable only through those
servers could not be reached on the first boot.

The step now lives in boot.before. kairos-agent runs boot.before,
boot and boot.after one after another, so the resolvers are written
before any runcmd entry runs.

Running earlier exposes a race the old placement hid. On Hadron
v0.5.3 /etc/resolv.conf is a symlink to
/run/systemd/resolve/resolv.conf, which systemd-resolved regenerates
when a DHCP lease brings DNS, and the yip in that release writes
through the link. cos-setup-boot is ordered only after
systemd-journald.socket. On a live node the first lease rewrote the
file at 13:23:00.25 and cos-setup-boot started at 13:23:00.66, so a
slower lease would land after the resolvers and replace them. The
step now writes a staged file and the next step renames it over
/etc/resolv.conf, which replaces the link with a regular file that
resolved does not touch.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (#141)")
Fixes: 51f3504 ("fix(controlplane): eliminate SSH from kubeconfig retrieval; extend CAPV with node-push pattern (KD-3b) (#55)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* fix(bootstrap): write the providerID drop-ins through a rename

The k3s, k3s fleet and k0s fleet providerID discovery scripts run twice
on a boot: as a cloud-config boot stage step and as ExecStartPre of the
k3s or k0s unit. Nothing orders the two runs against each other. On
the first boot runcmd waited for the post-bootstrap unit and held the
stage, which kept them apart; on later boots they could already
overlap. Each run truncated and rewrote the drop-in in place, so when
they overlap, k3s or the kubelet starting in that window could
read an empty or partial file and register without the providerID,
which cannot be changed afterwards.

Each script now writes a temporary file named after its own PID next
to the drop-in and renames it over the old one, so two runs never
share a file and the drop-in is always either the old or the new
content. The temporary names fall outside what k3s (config.yaml.d
*.yaml, *.yml) and the kubelet (--config-dir *.conf) load. The Metal3
script runs only from ExecStartPre but writes the same drop-in, so it
gets the same treatment. The new test refuses an in-place or shared
temporary write in the scripts. An ExecStartPre that writes a static
providerID is left as it is: the only other writer of that drop-in is
write_files, which lands before runcmd starts k3s.

The k0s fleet worker comment claimed the boot stage step puts the
drop-in in place before the kubelet registers. The ExecStartPre does
that; the step is a backstop that can run at the same time.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* fix(bootstrap): queue the kubeadm join unit instead of waiting for it

runcmd started the join unit with a plain `systemctl enable --now`.
The unit is a oneshot whose discovery retries can run for minutes, and
systemctl waited for it to finish, so the cloud-config stage that runs
runcmd sat in "Still running stage" for the whole join. Users and SSH
keys from the same cloud-config were applied only after it, so the
node could not be reached while the join was failing.

`--no-block` queues the start job and returns. Nothing after runcmd
depends on the join having finished, and the unit's ordering and
conditions are evaluated by systemd either way.

Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (#141)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* fix(bootstrap): queue the k3s post-bootstrap unit instead of waiting for it

runcmd started kairos-k3s-post-bootstrap.service with a plain
`systemctl enable --now`. The unit is a oneshot whose retry loops can
run for minutes, and systemctl waits for a oneshot to finish, so the
cloud-config stage running runcmd is held for the whole time, and the
users and SSH keys from the same cloud-config are applied only after
it. `--no-block` queues the start job and returns. Nothing after runcmd
depends on the unit having finished: no later runcmd entry, no unit
ordered after it, and the bootstrap sentinel it writes is polled
asynchronously.

The live-installer skip (ConditionKernelCommandLine=!cdroot) stays.
Its comment named holding up runcmd as the reason, which no longer
applies. The node still cannot register on the installer, and the
script would still run to its end: it writes the bootstrap-success
sentinel, and on an HA control plane it reports a healthy etcd member
under the installer's hostname, both for a node that does not exist.

Fixes: 51f3504 ("fix(controlplane): eliminate SSH from kubeconfig retrieval; extend CAPV with node-push pattern (KD-3b) (#55)")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

* revert(bootstrap): keep dnsServers in stages.boot for now

This reverts commit f1ac99f
("fix(bootstrap): apply spec.dnsServers before runcmd").

Moving the resolvers to boot.before and renaming a staged file over
/etc/resolv.conf changes what spec.dnsServers means, not only when it
is applied:

- on systemd-resolved images the host loses the DHCP-provided servers
  and search domains for the whole boot, not just early boot;
- the effect differs per distribution: none of the renders set the
  kubelet resolvConf, so k3s (and likely k0s) pods inherit only the
  custom servers, while kubeadm detects resolved and keeps DHCP DNS;
- on a bind-mounted /etc/resolv.conf (e.g. the docker render path) the
  rename fails with EBUSY and the resolvers are never applied, where
  writing through the mount works today.

The step goes back to stages.boot, the CRD description and
API_REFERENCE go back to main's wording, and the goldens and the
stages test that pinned the move are restored or removed. The other
commits of this PR (atomic providerID drop-ins, --no-block for the
kubeadm and k3s units) stand on their own. The DNS move is split out
for a separate decision.

Signed-off-by: William Rizzo <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Signed-off-by: Aleksei Sviridkin <[email protected]>
Signed-off-by: William Rizzo <[email protected]>
Co-authored-by: William Rizzo <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants