Repository navigation
feat(bootstrap): add kubeadm as a third distribution for worker joins - #141
Conversation
…tribution, role and infrastructure
Freeze the byte-for-byte output of generateCloudConfig across the full
controller assembly (token resolution, providerID, management-endpoint
resolver, LB lookup, applyControlPlaneRenderData) so the upcoming P0
distribution-seam refactor can prove it changes no rendered bytes.
Matrix: {k0s,k3s} x {worker on CAPV/CAPK/Metal3/fleet; single/init/join
on CAPV and CAPK}, plus the S5 control-plane edge cases: a nil resolver,
a resolver returning (nil,nil), a DockerMachine control plane (push
disabled), and a k3s control plane with an explicit spec.serverAddress.
Every golden is asserted to be valid YAML. spec.Manifests is held out of
these fixtures: the manifest render path emits content and the heredoc
terminator at column 0 inside a content:| block scalar, which is not
valid YAML. That is a pre-existing template characteristic, out of P0
scope; the controller's copy of spec.Manifests is characterised
separately.
Signed-off-by: William Rizzo <[email protected]>
…d the regeneration contracts Characterise behaviour the goldens cannot observe, before the P0 distribution-seam refactor: - Field-set isolation (I1): a k3s config with a control character in spec.podCIDR and a k0s control plane with a control character in serverAddress both render successfully, proving neither generator routes the other distribution's field into the validated TemplateData. - Exact error strings (M3b): one case per worker-token path on both the k0s and k3s chains (cross-namespace with the worker/k3s label, missing key, legacy keyless, hard Get failure), plus the controller's empty-token, missing-serverAddress and unsupported-distribution messages, asserted verbatim. - errTokenNotReady for a referenced-but-absent worker Secret, and the token-before-password read order (the token error wins). - The LB-endpoint error asymmetry as it stands today (OQ-A): k0s bare, k3s wrapped. The dedicated fix commit updates the k0s expectation. - Render contract (S2): every distribution/infra variant rendered with a providerID contains its post-bootstrap marker and an SSH-enable stage. - Regeneration decision (R4): an existing Secret with providerID, marker and SSH-enable stage is left alone; one missing either is regenerated. - Manifests are copied into the render by both generators (held out of the YAML goldens because the manifest heredoc is not valid YAML). Signed-off-by: William Rizzo <[email protected]>
Pin the user-visible distribution admission strings verbatim before the P0 seam moves KairosConfig's per-distribution validation into distribution.go, and before kubeadm P1 touches the enums: - KairosConfig: the distribution enum message, the k0s and k3s worker empty-token messages, and the two-error aggregate for an unknown distribution on a worker (enum error + default k0s worker-token error, OQ-C). Plus the defaulter writing k0s for an unset distribution. - KairosControlPlane and KairosControlPlaneTemplate: the distribution enum messages verbatim (KCPT including the spec.template.spec.* path). These webhook files must show zero diff across P0 (M1); the KCP/KCPT defaulters are already pinned by existing tests. Signed-off-by: William Rizzo <[email protected]>
Add a k3s-worker companion to TestBootstrapIntegration (which covers a k0s control plane): assert the bootstrap data Secret is created, named deterministically after the KairosConfig, owned by it (controller ref, resolvable GVK, matching UID), labelled with the cluster name, and that it carries a k3s worker cloud-config with the join token. This gives the worker join-material path integration coverage before P0 refactors it. Signed-off-by: William Rizzo <[email protected]>
The k0s generator returned the bare error from getControlPlaneLBEndpoint, while the k3s generator wrapped it as "failed to get control plane LB endpoint: %w". Wrap the k0s arm identically so the user-visible message (copied into conditions and status.failureMessage) no longer depends on the distribution. This changes a user-visible message for the k0s CAPK control-plane LB-lookup failure path; the only other effect is dropping the OQ-A error-prefix asymmetry the P0 render seam would otherwise carry. Updates the characterisation assertion added in PR 1 accordingly. Signed-off-by: William Rizzo <[email protected]>
…ribution Introduce api/bootstrap/v1beta2/distribution.go as the api-layer source of truth for the accepted distribution names: the DistributionK0s/K3s and DefaultDistribution constants, EffectiveDistribution / SupportedDistributions / IsSupportedDistribution, and a static per-distribution rules table (bootstrapDistributions) with validateK0sConfig / validateK3sConfig. The KairosConfig webhook's validate() now delegates its inline distribution name-check and worker-token switch to r.validateDistribution(); the defaulter writes DefaultDistribution. Behaviour is byte-identical: the same enum message, the same k0s and k3s worker-token messages, the same aggregate order (name error then token error), and an unknown name still falls back to the default's rules (OQ-C). TestDistributionTableMatchesCRDEnum pins the table to the kubebuilder enum and default in the generated CRD, so a later kubeadm enum edit and the table change must land together. No CRD/RBAC/deepcopy regeneration: make manifests generate produces no diff. The KCP and KCPT webhooks are untouched (M1: zero diff); api/ adds no import of internal/. Signed-off-by: William Rizzo <[email protected]>
…ution Introduce the renderer's single distribution table (distributionTemplates), keyed by the api distribution constants so it cannot drift from admission: each row carries the template.New name, the generic (CAPV/CAPM3/fleet) and KubeVirt (CAPK) template paths, and the providerID marker. Render(distribution, data) is now the only place a distribution name selects templates; ProviderIDMarker(distribution) returns the regeneration substring and an ok flag. RenderK0sCloudConfig / RenderK3sCloudConfig become permanent one-line wrappers over Render (OQ-E), so the 179 renderer test lines and the golden entry points are unchanged — the renderer goldens are byte-identical. Adds renderer-level seam tests: Render/wrapper byte-equality, the unsupported-distribution error, ProviderIDMarker, the table-agrees-with- SupportedDistributions check, and TestRenderContract (S2). Signed-off-by: William Rizzo <[email protected]>
…ibution source Add internal/controllers/bootstrap/join.go: the JoinMaterialSource interface (WorkerJoin only), JoinRequest, the redacting WorkerJoinMaterial, and the built-in k0sJoinSource / k3sJoinSource over a client.Reader. The reconciler gains an optional JoinSources map (nil default), so a missing key uses the built-in over r.Client and every existing construction site is unchanged; P1 injects a kubeadm source here. The tokens.go helpers (tokenFromWorkerRef, tokenFromLegacyRef, resolveWorkerToken, plus the extracted resolveK3sWorkerToken) become client.Reader functions without moving; resolveToken stays as a delegating wrapper for the tokens_test.go precedence tables, and a transitional tokenFromWorkerRef method shim keeps secretref_namespace_test.go compiling until the final cleanup commit. The control-plane join stays outside the seam (M2): applyControlPlaneRenderData now calls resolveControlPlaneJoinToken directly. Behaviour is unchanged: the empty-token and missing-serverAddress error strings move verbatim into the sources, and the goldens are byte-identical. Adds redaction, source-selection and injection tests. Signed-off-by: William Rizzo <[email protected]>
…ribution table
Add internal/controllers/bootstrap/render.go: the controller distribution
table (bootstrapDistribution{fillTemplateData, builtinJoin}), the merged
renderCloudConfig generator, and renderCarriesProviderID. The ~190-line
generateK0sCloudConfig / generateK3sCloudConfig bodies collapse into one
renderCloudConfig that keeps today's read order and defers the
distribution-only fields (k0s: WorkerToken, PodCIDR, ServiceCIDR; k3s:
K3sToken, K3sServerURL) and template selection to the table and
bootstrap.Render. generateCloudConfig's switch becomes a table lookup with
the same unsupported-distribution error; the two generators become
transitional one-line wrappers (removed in the final commit).
renderCarriesProviderID replaces both inline regeneration checks
(:401-411 and :595-603), using bootstrap.ProviderIDMarker with the same
unknown-distribution fallback to the default marker. EffectiveDistribution
replaces the remaining ""->k0s literals in generateCloudConfig,
applyControlPlaneRenderData and resolveControlPlaneJoinToken (S7:
resolveControlPlaneJoinToken stays in tokens.go). joinSourceFor now selects
the built-in through the table's builtinJoin.
Behaviour is byte-identical: all controller and renderer goldens are
unchanged, and the characterisation tests (error strings, ordering,
regeneration) still pass. Adds the controller table-agreement and
renderCarriesProviderID tests.
Signed-off-by: William Rizzo <[email protected]>
…nd tokenKind Final P0 cleanup: delete the now-dead code the seam replaced. The 16 controller-test calls move from generateK0sCloudConfig/generateK3sCloudConfig to renderCloudConfig with an explicit distribution; the two generator wrappers are removed. resolveToken, the tokenKind type and its constants, and the transitional tokenFromWorkerRef method shim are deleted. The tokens_test.go precedence tables now drive the k0s/k3s join sources directly, the control-plane-join case drives resolveControlPlaneJoinToken, and secretref_namespace_test.go calls the package-level tokenFromWorkerRef. No behaviour change: all controller and renderer goldens are byte-identical and the full suite passes. Signed-off-by: William Rizzo <[email protected]>
…e table The field-to-fixture table in render_golden_test.go listed spec.Manifests twice with contradictory claims: once as set in "every fixture" and once as "not in YAML goldens; see note". The NOTE on goldenBaseSpec is authoritative — Manifests is deliberately not set in the YAML-validated goldens because the manifest render path emits a column-0 heredoc terminator that breaks the enclosing block scalar (a pre-existing defect, out of scope). Remove the stale "every fixture" row so the table matches the fixtures. (ADR 0010 P1 N1.) Signed-off-by: William Rizzo <[email protected]>
…lane ADR 0010 P1 ships kubeadm as a bootstrap worker-join distribution only; a kubeadm KairosControlPlane is deferred to P2, so the KCP enum stays [k0s, k3s]. The validating webhook already refuses an explicit spec.distribution=kubeadm. This closes the inherit-path gap before the bootstrap enum flip admits kubeadm on the webhook-less KairosConfigTemplate (ADR 0010 P1 item 2): a KairosControlPlane that references a kubeadm KairosConfigTemplate would otherwise inherit kubeadm and try to persist it into spec.distribution, which the webhook rejects on every reconcile — an error loop. The controller now refuses both the inherited and (defence-in-depth) the explicit kubeadm case with a clear Available/Ready False(Warning, UnsupportedDistribution) condition and latched failure fields, never writing kubeadm into spec.distribution and never requeuing into a loop. A later k0s/k3s template edit re-enqueues through the normal watches. Signed-off-by: William Rizzo <[email protected]>
ADR 0010 P1 (worker join). This is the atomic enum-flip commit: the moment
kubeadm enters SupportedDistributions() the api validation table, the renderer
template table, and the controller distribution table must all carry a kubeadm
row, and the agreement tests enforce that — so they land together.
api:
- DistributionKubeadm constant + a validateKubeadmConfig admission row. Static
refusals: role control-plane (kubeadm is worker-only in P1), inline bootstrap
token, unsafeSkipCAVerification, discovery.file, any "{{" anywhere in the
kubeadm block (Jinja is not expanded on Kairos and the block is written into an
on-node file), and a non-DNS-1123 nodeRegistration.name.
- spec.kubeadm embeds the INDIVIDUAL CAPI kubeadm v1beta2 JoinConfiguration type
(OQ-5), not KubeadmConfigSpec and not ClusterConfiguration. The embedding pulls
k8s.io/cluster-bootstrap and k8s.io/component-base into go.sum — both already in
the build list at the CAPI-pinned v0.35.4 (go mod tidy only records the sums).
- KairosConfigSpec.distribution enum flipped to [k0s, k3s, kubeadm] and
status.bootstrapTokenID (non-secret token-id surface) added; make manifests +
make generate run in this commit (TestDistributionTableMatchesCRDEnum enforces).
The CRD grows ~36KB -> ~70KB, under the client-side-apply limit. The KCP enum is
unchanged. A schema-diff guard (TestKubeadmSchemaDiff, OQ-5) hashes the served
spec.kubeadm sub-schema so a CAPI bump that silently changes it fails loudly.
renderer (internal/bootstrap stays CAPI-type-free):
- a kubeadm templateSet row + a worker-only template that writes the marshalled
JoinConfiguration, the providerID kubeadm patch, and a oneshot join unit named
kairos-kubeadm-post-bootstrap.service (the providerID marker), and enables sshd
(TestRenderContract / the R4 regeneration contract require both).
- TemplateData.Kubeadm is a plain-string struct; Render refuses a kubeadm
control-plane render (the seam has no control-plane hook, item 5);
validateTemplateData rejects "{{" in the block as a render-time last line.
controller:
- a kubeadm row {fillKubeadm, builtinJoin: nil}: the worker JoinMaterialSource is
injected via JoinSources (dependencies only main.go has), so joinSourceFor now
HARD-FAILS when builtinJoin==nil and no source is injected instead of silently
falling back to the k0s built-in (N2). fillKubeadm copies the minted join
material and builds the providerID patch from the resolved providerID.
- WorkerJoinMaterial gains a redacted kubeadm block (JoinConfiguration stays
secret; only the non-secret token ID is surfaced in String/GoString).
The kubeadm JoinMaterialSource (trust check + token mint) lands in the next commit.
Signed-off-by: William Rizzo <[email protected]>
…rified trust check ADR 0010 P1 security core. The kubeadm worker JoinMaterialSource (internal/controllers/bootstrap/kubeadm.go), wired via JoinSources["kubeadm"] in main.go with a clustercache-backed, ClusterFilter-gated workload client, an uncached management reader, and the operator-level trust allowlist. Trust check (ADR 0010 P1 step 1), every failure terminal with no token minted: - resolve Cluster.spec.controlPlaneRef and match it on GROUP AND KIND against the allowlist (KubeadmControlPlane, KamajiControlPlane, and a KairosControlPlane whose spec.distribution is kubeadm); any other kind only via the operator-level --kubeadm-extra-controlplane-kinds flag, never a per-Cluster field (OQ-9); - require the <cluster>-kubeconfig Secret to carry a controller ownerReference whose UID equals the resolved control-plane object (refuses node-pushed/planted kubeconfigs); - require the kubeconfig server to equal https://<Cluster.spec.controlPlaneEndpoint>; - take the CA pin from the kubeconfig's certificate-authority-data (SPKI sha256 hashes) and, when <cluster>-ca exists, cross-check its tls.crt via an UNCACHED read, parsing only the cert (Kamaji's unowned default CA and an upstream-KCP user CA both pass; a mismatch is refused). Token + render: - mint a bootstrap token in the WORKLOAD cluster (clustercache), 15m TTL, CABPK groups/usages/keys; the token value is never logged (k8s.io/cluster-bootstrap, the canonical source, is already pulled in by the spec.kubeadm embedding); - build the JoinConfiguration (nodeRegistration.name = Machine hostname, discovery bootstrapToken with the token + endpoint + caCertHashes, the node.cluster.x-k8s.io/uninitialized taint) and marshal it version-aware; providerID flows through a kubeadm kubeletconfiguration patch, not the JoinConfiguration; - MachineDeployment workers only — a MachinePool owner is refused. Terminal-vs-not-ready contract (item 6): a typed bootstrapNotReadyError wraps every transient wait (control plane not initialized, kubeconfig absent, workload unreachable/API timeout). reconcileBootstrapData maps it to a 10s requeue next to errTokenNotReady and leaves conditions alone, so a watch/clustercache blip never parks the KairosConfig. Token refresh (item 7): the optional workerTokenRefresher, found by type assertion, re-mints while the Machine has no nodeRef — rotating once the workload token Secret (keyed by the stored non-secret ID) is gone or past half its TTL — with a 24h hard cap that raises a BootstrapTokenRefreshCapExceeded condition and a documented MHC nodeStartupTimeout backstop. Status (item 8): the reconciler writes the NON-SECRET token ID to status.bootstrapTokenID; the secret half lives only in the data Secret. WorkerJoinMaterial's redaction stays total (kubeadm JoinConfiguration REDACTED, only the token ID surfaced). Signed-off-by: William Rizzo <[email protected]>
Exercises the kubeadm worker join against a real apiserver (ADR 0010 P1 test plan). A Cluster whose controlPlaneRef is an owner-verified control plane and whose <cluster>-kubeconfig carries a matching controller owner UID mints a bootstrap token in a fake workload cluster and produces bootstrap data: status.dataSecretName and the non-secret status.bootstrapTokenID are set, the rendered cloud-config carries the JoinConfiguration and the providerID-marker unit, and the token Secret is created in the workload kube-system. A kubeconfig whose controller owner UID does NOT match the control plane is refused — no data Secret, no token id, and a BootstrapDataSecretGenerationFailed condition whose message names the owner-check failure. The trust anchor is a throwaway schemaless TestControlPlane kind trusted via the operator-level allowlist flag, which also covers the operator opt-in path (OQ-9) without depending on the full KubeadmControlPlane schema. Signed-off-by: William Rizzo <[email protected]>
…inds The kubeadm worker-join trust check (ADR 0010 P1) does a get on the Cluster's control-plane object to verify the <cluster>-kubeconfig's controller-owner UID, but the bootstrap manager had no controlplane.cluster.x-k8s.io grant. On a bootstrap-only (--controllers=bootstrap) Deployment — which has none of the control-plane manager's RBAC — that Get returns Forbidden, which the source maps to not-ready and requeues forever, so no kubeadm worker ever joins. envtest masked this because it runs with an admin client and enforces no RBAC. Add a least-privilege, read-only, enumerated grant on kamajicontrolplanes, kubeadmcontrolplanes, and kairoscontrolplanes (the P1 trust anchors) in the bootstrap package. make manifests lands it in both the aggregated config/rbac/role.yaml and the bootstrap split overlay config/rbac/bootstrap/role.yaml (ADR 0007). An operator who widens the allowlist via --kubeadm-extra-controlplane-kinds must supply RBAC for that group themselves — we cannot grant kinds in groups unknown at build time. TestBootstrapRoleGrantsControlPlaneRead parses the generated bootstrap role and asserts the get so a future make-manifests cannot silently drop it (the envtest-masks-RBAC guard the security review required). Signed-off-by: William Rizzo <[email protected]>
…ane kinds flag Add the user-facing documentation for KairosConfig distribution: kubeadm (worker-only joins to a Kamaji or KubeadmControlPlane control plane). - docs/KUBEADM_WORKERS.md: how the join works, prerequisites (control plane, node image at the exact Machine.spec.version, CNI), admission refusals, token lifetime and the refresh cap, MachineHealthCheck guidance, the KairosControlPlane UnsupportedDistribution refusal, and troubleshooting. - config/samples/kubeadm: a worker KairosConfigTemplate, MachineDeployment and MachineHealthCheck. The password comes from a Secret created by command. - docs/API_REFERENCE.md: spec.distribution, spec.kubeadm, status.bootstrapTokenID, the new condition reasons, and the persistence and token-requirement notes. - docs/INSTALL.md: the --kubeadm-extra-controlplane-kinds manager flag (format, default allowlist, RBAC, how to set it on each install path). - README.md, docs/TESTING.md: distribution list, target versions, and the verification status (unit and envtest only). Signed-off-by: William Rizzo <[email protected]>
…ent state We publicly promised in the #4920 reply that a Kairos kubeadm worker "runs only on an installed (active/passive) boot, never on live/recovery/auto-reset/in-RAM" and "a node that has joined never joins again on reboot; a failed join is not retried on the node". The join unit previously gated only on ConditionPathExists=!/etc/kubernetes/kubelet.conf, which fails open on a v4.2+ in-RAM boot (/etc is a tmpfs overlay, KD-59). Mirror the ADR 0010 "Node side" / C5 pattern in the join unit: - a positive OR allowlist ConditionPathExists=|/run/cos/active_mode / |/run/cos/passive_mode; ConditionPathExists=!/run/cos/in_ram_mode; - ConditionPathIsMountPoint for /etc/kubernetes and /var/lib/kubelet so a join never runs on ephemeral/tmpfs state; and in the join script: - a persistent "attempted" marker written on COS_PERSISTENT (/var/lib/kairos) BEFORE kubeadm runs and a "completed" marker after a verified join; a later boot that finds attempted-without-completion refuses to retry on-node (remediation replaces the Machine); - the exact-version check runs before the attempted marker so a version mismatch does not burn the single attempt; - on failure, credential cleanup only (rm the token-bearing bootstrap-kubelet.conf); a worker has no etcd/PKI to reset. The render test asserts every gate condition and marker. k0s/k3s templates and goldens are untouched (there are no kubeadm goldens; the kubeadm render is covered by substring assertions). Signed-off-by: William Rizzo <[email protected]>
…ct, not a pin The kubeadm trust check hardcoded v1beta2 when reading the Cluster's control-plane object (with a dead retry that pinned the same version). That silently breaks the primary P1 target KamajiControlPlane, which is served at v1alpha1, and the retry did nothing its comment claimed (security review NIT). Resolve the ContractVersionedObjectReference through external.GetObjectFromContractVersionedRef, which reads the referenced kind's CRD contract label to pick the served API version — exactly how CAPI core resolves the same ref, so Kamaji (v1alpha1) and KubeadmControlPlane (v1beta2) both resolve. The resolver is injectable (default = the CAPI helper) so unit tests stub it and need no installed CRD / contract labels; the envtest trust anchor carries the cluster.x-k8s.io/v1beta2 contract label and exercises the real resolver. Also document that TokenNeedsRefresh is read-only, reachable only after a prior successful trust check (non-empty token ID in status), mints nothing, and still runs behind the coarse clustercache ClusterFilter (security review NIT). Signed-off-by: William Rizzo <[email protected]>
…ly joins, and neutralize user-facing messages
Three tech-writer/security cleanups on the kubeadm worker path:
1. Coarse cluster filter was over-broad. KubeadmTrustedClusterFilter matched
group+kind and included KairosControlPlane, so the clustercache would connect to
k0s/k3s KairosControlPlane clusters (the fine per-request check still refused them,
so it was connect-only, never a mint). Split off coarseTrustedControlPlaneKinds
(KubeadmControlPlane + KamajiControlPlane + operator-flag kinds) that the filter
uses, excluding KairosControlPlane while kubeadm control planes are unsupported.
The fine WorkerJoin check keeps the KairosControlPlane(distribution==kubeadm)
branch for forward-compat. Adds TestKubeadmTrustedClusterFilter (none existed).
2. A control-plane join is now refused at admission. The joinConfiguration doc said
controlPlane must be nil but nothing enforced it; validateKubeadmConfig now rejects
a non-nil joinConfiguration.controlPlane (worker-only). Adds a webhook test case.
3. User-facing messages no longer cite internal decision records or hint at the
embargoed topic. Reworded to self-contained guidance:
- webhook role refusal (distribution.go): dropped the "(ADR 0010 P2)" tail.
- KCP refusal failureMessage: dropped "(ADR 0010 P1)".
- render control-plane refusal (template.go): dropped "(ADR 0010 P1)".
- kubeadm control-plane distribution error: dropped "(ADR 0010)".
- kubeconfig owner-mismatch error: "the <cluster>-kubeconfig Secret must be owned
by the cluster's control plane; refusing to mint a token" (was "a node-pushed or
planted kubeconfig is not trusted on the kubeadm path").
Updated the unit + envtest assertions for the reworded owner-mismatch string, and
corrected the Kamaji served-version comment (v1alpha2, not v1alpha1).
Signed-off-by: William Rizzo <[email protected]>
The distribution, kubeadm, and joinConfiguration field doc-comments cited internal
decision-record IDs (ADR 0010 P1/P2/OQ-5) and an unfollowable "see
validateKubeadmConfig" reference, which surface verbatim in `kubectl explain`.
Reworded noun-first with the enum values and defaults spelled out and no internal
IDs. Also corrected the pre-existing misleading KubernetesVersion comment ("version
to install"): it is informational; the authoritative version for a kubeadm worker is
the owning Machine's spec.version, which the rendered join checks against the image's
kubeadm. The status.bootstrapTokenID description lost its ADR tag too.
make manifests + make generate regenerate the two bootstrap CRDs; the only diff is
the reworded descriptions (no enum/field/type/default change). The kubeadm
schema-diff digest is updated to match (descriptions are part of the served schema it
guards).
Signed-off-by: William Rizzo <[email protected]>
A CA mismatch has a legitimate benign cause (CA rotation or re-pin), so the message now points the operator at inspecting and re-pinning rather than asserting tampering. Signed-off-by: William Rizzo <[email protected]>
…d join credentials Three findings from the pre-merge security review of the kubeadm worker join. The endpoint-equality leg of the trust check compared strings. Cluster API writes the kubeconfig's server from APIEndpoint.String, which is net.JoinHostPort and therefore brackets an IPv6 literal, so a cluster with an IPv6 control-plane endpoint published `https://[fd00::1]:6443` while the check demanded `https://fd00::1:6443`. Every such cluster was refused, terminally, with a message stating that the kubeconfig's server does not equal the Cluster endpoint -- which reads like a planted-credential incident rather than a formatting mismatch. A trailing slash, a host differing in case, and an omitted default port failed the same way. The comparison now parses both sides and compares scheme, host case-insensitively, and port with the scheme's default. The same hand-formatting produced the JoinConfiguration's apiServerEndpoint, so an IPv6 endpoint would have reached kubeadm unbracketed even once the check passed. Both sites now go through APIEndpoint.String, so they cannot drift. discovery.tlsBootstrapToken was neither refused by the webhook nor cleared by the renderer, although its sibling discovery.bootstrapToken.token is refused on the grounds that tokens never live in the spec. It defaults to the bootstrap token but may be overridden, so a value there became the kubelet's TLS-bootstrap credential in preference to the minted one: plaintext in the spec, copied into the rendered userdata on /oem, and outside the short TTL and refresh the controller's own token relies on. It is now forbidden at admission and cleared at render time, which also makes the invariant in docs/KUBEADM_WORKERS.md true as written. joinConfiguration.controlPlane was enforced only at admission. Because no webhook is registered for KairosConfigTemplate, every kubeadm refusal on a template is caught one object later, when Cluster API core derives the KairosConfig. That is fail-closed, but the repo rule is webhook and render-time, so the renderer now nils it as it already does for file discovery. The docs and sample claimed these fields are "rejected at admission" without qualification. Both now say which object the webhook covers and which refusals are additionally enforced when the cloud-config is rendered. Tests: the four endpoint shapes that used to be refused are now accept cases, a non-controller ownerRef is pinned as a terminal refusal, the webhook refusal for tlsBootstrapToken is pinned, and a render-time test asserts a spec-supplied tlsBootstrapToken and controlPlane never reach the rendered JoinConfiguration while the minted token still does. Each was confirmed to fail with its fix reverted. go mod tidy promotes blang/semver/v4 and k8s.io/cluster-bootstrap to direct requirements; both were already imported directly. Signed-off-by: William Rizzo <[email protected]>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #141 +/- ##
==========================================
+ Coverage 30.29% 36.35% +6.05%
==========================================
Files 52 60 +8
Lines 6982 7678 +696
==========================================
+ Hits 2115 2791 +676
+ Misses 4635 4607 -28
- Partials 232 280 +48
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… kubeadm joins Drop the kubeconfig server == controlPlaneEndpoint check, which refuses Kamaji clusters publishing admin.svc; trust stays on the owner UID and CA pin. Use spec.serverAddress as the join endpoint when set. Signed-off-by: Patryk Rostkowski <[email protected]>
The preceding commit drops the kubeconfig server comparison outright, on the correct observation that a control-plane provider may publish an internal address there: Kamaji publishes admin.svc when spec.network.advertiseAddress is set, and our equality check refused those clusters terminally. That part of the diagnosis stands and the live evidence behind it is the only way it could have been found. Dropping the check entirely goes further than the fix needs to, because the owner reference is not an anchor. ownerReferences are user-settable and a UID is readable, so anyone who can create Secrets in the Cluster's namespace can forge the owner leg. That left the server equalling the Cluster endpoint and the CA matching a separately-read <cluster>-ca as the only two independent anchors, and ADR 0010 P1 requires the first of them. With neither, a planted <cluster>-kubeconfig steers this controller's own client -- which carries workload-cluster admin credentials -- at any address reachable from the management network. So require at least one of the two rather than both. The server may differ from the Cluster endpoint when <cluster>-ca exists and its tls.crt matches the kubeconfig's CA. Kamaji satisfies that: it publishes <cluster>-ca alongside the kubeconfig, which is why the accept case added by the preceding commit already carries one. A kubeconfig whose server differs with no <cluster>-ca to anchor it is refused, with a message naming both ways out. Also restores the IPv6 fix the rebase reverted: the no-serverAddress path of kubeadmJoinEndpoint was hand-formatting host:port again, so an IPv6 endpoint would have reached kubeadm unbracketed. It goes back through APIEndpoint.String, which is what the serverAddress path already does via net.JoinHostPort. Tests: the Kamaji accept case stands unchanged, and its counterpart -- server differs, no <cluster>-ca -- is pinned as a terminal refusal. Confirmed to fail with the conditional disabled. Signed-off-by: William Rizzo <[email protected]>
|
thanx @wrkode! answering to your comment from this PR The conditional makes sense, and I am ok with it. I have tested https://github.com/wrkode/cluster-api-provider-kairos/tree/feat/kubeadm-kamaji-serveraddress end to end on a live Kamaji management cluster. Results
Every KairosConfig went To confirm your two points:
Let me know if you need any additional assitance! |
The trust check now accepts a kubeconfig whose server differs from the Cluster endpoint when <cluster>-ca anchors its CA, so operators have a reason to care what lives in that Secret. Under Kamaji they cannot choose it: a bring-your-own CA is honoured only as a pre-created <tcp>-ca holding a ca.crt/ca.key pair, and a CAPI-style <cluster>-ca carrying only tls.crt/tls.key is overwritten with Kamaji's generated CA. The join then pins Kamaji's CA rather than the supplied one, silently. Found by @patrostkowski while validating the anchor against a live Kamaji cluster: he pre-created exactly that Secret, with the KamajiControlPlane name differing from the Cluster name, and watched the provider replace it. It is Kamaji's behaviour rather than ours, and it leaves the anchor intact, because <cluster>-ca still carries the tenant CA in every configuration. Signed-off-by: William Rizzo <[email protected]>
…raddress # Conflicts: # go.sum
ADR 0010 OQ-8 permits shipping the kubeadm worker path as a preview before C5 lands only if its docs forbid putting an upstream KubeadmControlPlane cluster in the same namespace as a k0s or k3s cluster. That condition was agreed and then never written down, so state it where someone enabling the feature will read it. The reason is that upstream KCP adopts a pre-existing <cluster>-ca or <cluster>-kubeconfig Secret instead of insisting on creating one. An adopted Secret carries a legitimate controller owner reference, so the trust check cannot tell it from the real thing: the check catches a planted Secret, not an adopted one. What makes that reachable is the unscoped secrets create on the Role the k0s/k3s control-plane kubeconfig push uses, so a node compromised in a namespace shared with an upstream-KCP cluster can plant for its neighbour. Separate namespaces remove the adjacency, which is the only mitigation until that push path is fixed. Kamaji is not exposed the same way: it overwrites planted data and sets itself as controller owner, so the prohibition names upstream KCP specifically. Also corrects the verification status, which still claimed nothing had been run against a live cluster. The Kamaji path has now been validated end to end in three configurations with real Kairos images; upstream KCP still has not. The scope table said both kinds were equally unvalidated and now distinguishes them. Signed-off-by: William Rizzo <[email protected]>
|
@patrostkowski, changes rolled in with little adjustments and we're good to merge. |
runcmd started the join unit with a plain `systemctl enable --now`. The unit is a oneshot whose discovery retries can run for minutes, and systemctl waited for it to finish, so the cloud-config stage that runs runcmd sat in "Still running stage" for the whole join. Users and SSH keys from the same cloud-config were applied only after it, so the node could not be reached while the join was failing. `--no-block` queues the start job and returns. Nothing after runcmd depends on the join having finished, and the unit's ordering and conditions are evaluated by systemd either way. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
* fix(build): build the image with Go 1.26.9 The builder image was pinned to golang:1.26-alpine at Go 1.26.8. Go 1.26.9, released 2026-10-08, fixes thirteen standard-library advisories, most of them in code this provider runs: the webhook, metrics and health servers, and the HTTP/2 client to the API server. - net/http and HTTP/2: GO-2026-6603, GO-2026-6605, GO-2026-6609, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6613 and GO-2026-6617 - crypto/tls: GO-2026-6607 - net/textproto and mime/multipart: GO-2026-6608 - html/template: GO-2026-6599 and GO-2026-6600 - os, Windows only: GO-2026-6604 #143 cleared the golang.org/x/net copies of the HTTP/2 issues, but the binary also links the standard library's own copy, which only a new toolchain fixes. The new pin is the multi-arch index digest that golang:1.26.9-alpine resolves to; the linux/amd64 and linux/arm64 images in it both carry Go 1.26.9. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> * docs(release): prepare v0.1.4 v0.1.4 is a fix release for the v0.1 series, cut from main just before the kubeadm worker distribution (#141) merged. It ships the control-plane and bootstrap fixes #109 to #120 and the Go security updates, and none of the kubeadm work, which goes into v0.2.0. - docs/release-notes/v0.1.4.md: the release body that release.yaml requires. - CHANGELOG.md: the v0.1.4 entry. - docs/UPGRADING.md: a v0.1.3 to v0.1.4 section. Its main content is the check to run before upgrading for references to another namespace, which #119 now rejects. It also covers the clusterctl.yaml URL fix from #111 and the behaviour changes to expect. - README, INSTALL, the quickstarts and the CAPM3 samples: install URLs and the latest-release pointer move to v0.1.4. The "Last verified against" headers stay at v0.1.3, because nothing in them was re-verified for this release. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> --------- Signed-off-by: William Rizzo <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
kubeadm refuses any flag next to --config that is not on its short allowlist, and --patches is not on it. The join unit passed --patches whenever a providerID patch was rendered, so every worker whose providerID is known at render time failed with "can not mix '--config' with arguments [patches]" and never joined. The directory now travels in JoinConfiguration.patches.directory, defaulting to /etc/kubernetes/kairos-patches and honouring a directory set in the spec. The unit creates it before it writes the attempted marker, because kubeadm fails on a missing directory and a failing mkdir must not use up the node's only join attempt. The same directory is where users place their own kubelet configuration patches through spec.files. The renderer checks it with ValidatePatchesDirectory, the rule admission applies, as the backstop for configs derived from a KairosConfigTemplate. The new test renders the worker cloud-config, runs the join script against a kubeadm stub and checks the resulting command line against kubeadm's flag rules. With KUBEADM_BIN pointing at a real kubeadm, the stub hands the call to it with every join phase skipped and the rendered configuration is also checked with kubeadm config validate. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
yip loads runcmd, write_files and users from a cloud-config as the first step of the boot stage and appends the stages.boot steps after it. The "Configure DNS resolvers" step lived in stages.boot, so it ran only after runcmd had started the join or post-bootstrap unit. Since that start waited for the unit, the custom resolvers never applied to a kubeadm join, a k3s or k0s post-bootstrap run, or anything they resolved; a control-plane endpoint resolvable only through those servers could not be reached on the first boot. The step now lives in boot.before. kairos-agent runs boot.before, boot and boot.after one after another, so the resolvers are written before any runcmd entry runs. Running earlier exposes a race the old placement hid. On Hadron v0.5.3 /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf, which systemd-resolved regenerates when a DHCP lease brings DNS, and the yip in that release writes through the link. cos-setup-boot is ordered only after systemd-journald.socket. On a live node the first lease rewrote the file at 13:23:00.25 and cos-setup-boot started at 13:23:00.66, so a slower lease would land after the resolvers and replace them. The step now writes a staged file and the next step renames it over /etc/resolv.conf, which replaces the link with a regular file that resolved does not touch. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)") Fixes: 51f3504 ("fix(controlplane): eliminate SSH from kubeconfig retrieval; extend CAPV with node-push pattern (KD-3b) (kairos-io#55)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
runcmd started the join unit with a plain `systemctl enable --now`. The unit is a oneshot whose discovery retries can run for minutes, and systemctl waited for it to finish, so the cloud-config stage that runs runcmd sat in "Still running stage" for the whole join. Users and SSH keys from the same cloud-config were applied only after it, so the node could not be reached while the join was failing. `--no-block` queues the start job and returns. Nothing after runcmd depends on the join having finished, and the unit's ordering and conditions are evaluated by systemd either way. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (kairos-io#141)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
* fix(build): build the image with Go 1.26.9 The builder image was pinned to golang:1.26-alpine at Go 1.26.8. Go 1.26.9, released 2026-10-08, fixes thirteen standard-library advisories, most of them in code this provider runs: the webhook, metrics and health servers, and the HTTP/2 client to the API server. - net/http and HTTP/2: GO-2026-6603, GO-2026-6605, GO-2026-6609, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6613 and GO-2026-6617 - crypto/tls: GO-2026-6607 - net/textproto and mime/multipart: GO-2026-6608 - html/template: GO-2026-6599 and GO-2026-6600 - os, Windows only: GO-2026-6604 #143 cleared the golang.org/x/net copies of the HTTP/2 issues, but the binary also links the standard library's own copy, which only a new toolchain fixes. The new pin is the multi-arch index digest that golang:1.26.9-alpine resolves to; the linux/amd64 and linux/arm64 images in it both carry Go 1.26.9. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> (cherry picked from commit 008c6ae) * docs(release): prepare v0.1.4 v0.1.4 is a fix release for the v0.1 series, cut from main just before the kubeadm worker distribution (#141) merged. It ships the control-plane and bootstrap fixes #109 to #120 and the Go security updates, and none of the kubeadm work, which goes into v0.2.0. - docs/release-notes/v0.1.4.md: the release body that release.yaml requires. - CHANGELOG.md: the v0.1.4 entry. - docs/UPGRADING.md: a v0.1.3 to v0.1.4 section. Its main content is the check to run before upgrading for references to another namespace, which #119 now rejects. It also covers the clusterctl.yaml URL fix from #111 and the behaviour changes to expect. - README, INSTALL, the quickstarts and the CAPM3 samples: install URLs and the latest-release pointer move to v0.1.4. The "Last verified against" headers stay at v0.1.3, because nothing in them was re-verified for this release. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> (cherry picked from commit 33e2d47) * docs: say that the kubeadm worker distribution is not in v0.1.4 v0.1.4 was cut from release-0.1, which branched from main just before #141, so it does not contain kubeadm workers. On main, the README now names v0.1.4 as the latest release and then describes `distribution: kubeadm` directly after it, which reads as though that release supports it. - README: mark the kubeadm paragraph as not in v0.1.4, planned for v0.2.0. - INSTALL: say the same for `--kubeadm-extra-controlplane-kinds`. A v0.1.4 manager parses its flags with the standard library, so given this flag it exits with "flag provided but not defined" instead of starting. - KUBEADM_WORKERS: the version line no longer says the feature follows v0.1.3. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> * fix(build): pin kustomize so the release job can render manifests The v0.1.4 release job failed at "Render release manifests": the Makefile installed kustomize with `@latest`, which now resolves to v5.8.3, and v5.8.3 requires go >= 1.27. The job runs the go.mod toolchain (1.26.4) with GOTOOLCHAIN=local, so `go install` refused it and no GitHub release was published, although the image had already been pushed. Pin KUSTOMIZE_VERSION to v5.8.1, the version that rendered the v0.1.3 release artifacts, so release manifests are reproducible and no longer depend on whatever kustomize was published last. Verified with a clean bin/: `make release-manifests` renders all five artifacts, sha256sums verify, every image ref is digest-pinned, and the resource kinds and counts match v0.1.3. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> --------- Signed-off-by: William Rizzo <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
…on (#148) * fix(api): refuse an unsafe kubeadm patches.directory kubeadm walks the patches directory recursively and applies every patch it finds there as root during the join. Admission accepted any value: `/` made kubeadm walk the whole root filesystem, a world-writable directory such as /tmp or /dev/shm let any local user drop a kubelet patch, and a value with a newline, CR or NUL was written into the node's JoinConfiguration. ValidatePatchesDirectory now requires an absolute, clean path under /etc/kubernetes/ without control characters. It must not be /etc/kubernetes/manifests, the kubelet's static pod directory, or a directory under it. The function is exported so the renderer can apply the same rule to configs derived from a KairosConfigTemplate, which has no webhook, and the two checks cannot drift apart. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * fix(bootstrap): name the kubeadm patches directory in JoinConfiguration kubeadm refuses any flag next to --config that is not on its short allowlist, and --patches is not on it. The join unit passed --patches whenever a providerID patch was rendered, so every worker whose providerID is known at render time failed with "can not mix '--config' with arguments [patches]" and never joined. The directory now travels in JoinConfiguration.patches.directory, defaulting to /etc/kubernetes/kairos-patches and honouring a directory set in the spec. The unit creates it before it writes the attempted marker, because kubeadm fails on a missing directory and a failing mkdir must not use up the node's only join attempt. The same directory is where users place their own kubelet configuration patches through spec.files. The renderer checks it with ValidatePatchesDirectory, the rule admission applies, as the backstop for configs derived from a KairosConfigTemplate. The new test renders the worker cloud-config, runs the join script against a kubeadm stub and checks the resulting command line against kubeadm's flag rules. With KUBEADM_BIN pointing at a real kubeadm, the stub hands the call to it with every join phase skipped and the rendered configuration is also checked with kubeadm config validate. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (#141)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * test(bootstrap): run the kubeadm worker join against a real kubeadm A copy of kubeadm's flag rules only catches the mistakes it was written for. make test-kubeadm downloads kubeadm and runs the join command test with it, so kubeadm itself parses the rendered command line and loads the rendered JoinConfiguration. Every join phase is skipped and --dry-run is set, so nothing touches the host or the network even if a later kubeadm adds a phase the help scrape misses. The fixture names the CRI socket, so kubeadm does not probe the host and fail where several CRI sockets exist. The sha256 of each kubeadm version is pinned in hack/kubeadm.sha256. A sum fetched from the same host as the binary would not catch a replaced binary. CI runs it in its own job against kubeadm v1.30.14, which takes JoinConfiguration v1beta3, and v1.36.2, which takes v1beta4. The workflow token is now read-only: the job runs a downloaded binary, and the repository default token can write. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * ci: make /etc/kubernetes writable for the kubeadm-join job kubeadm join --dry-run creates /etc/kubernetes/tmp on the host whatever the test root is, so the job failed as the non-root runner user with 'mkdir /etc/kubernetes: permission denied'. Create the directory owned by the runner user before the test, and document the same for local runs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: William Rizzo <[email protected]> --------- Signed-off-by: Aleksei Sviridkin <[email protected]> Signed-off-by: William Rizzo <[email protected]> Co-authored-by: William Rizzo <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
…#149) * fix(bootstrap): apply spec.dnsServers before runcmd yip loads runcmd, write_files and users from a cloud-config as the first step of the boot stage and appends the stages.boot steps after it. The "Configure DNS resolvers" step lived in stages.boot, so it ran only after runcmd had started the join or post-bootstrap unit. Since that start waited for the unit, the custom resolvers never applied to a kubeadm join, a k3s or k0s post-bootstrap run, or anything they resolved; a control-plane endpoint resolvable only through those servers could not be reached on the first boot. The step now lives in boot.before. kairos-agent runs boot.before, boot and boot.after one after another, so the resolvers are written before any runcmd entry runs. Running earlier exposes a race the old placement hid. On Hadron v0.5.3 /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf, which systemd-resolved regenerates when a DHCP lease brings DNS, and the yip in that release writes through the link. cos-setup-boot is ordered only after systemd-journald.socket. On a live node the first lease rewrote the file at 13:23:00.25 and cos-setup-boot started at 13:23:00.66, so a slower lease would land after the resolvers and replace them. The step now writes a staged file and the next step renames it over /etc/resolv.conf, which replaces the link with a regular file that resolved does not touch. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (#141)") Fixes: 51f3504 ("fix(controlplane): eliminate SSH from kubeconfig retrieval; extend CAPV with node-push pattern (KD-3b) (#55)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * fix(bootstrap): write the providerID drop-ins through a rename The k3s, k3s fleet and k0s fleet providerID discovery scripts run twice on a boot: as a cloud-config boot stage step and as ExecStartPre of the k3s or k0s unit. Nothing orders the two runs against each other. On the first boot runcmd waited for the post-bootstrap unit and held the stage, which kept them apart; on later boots they could already overlap. Each run truncated and rewrote the drop-in in place, so when they overlap, k3s or the kubelet starting in that window could read an empty or partial file and register without the providerID, which cannot be changed afterwards. Each script now writes a temporary file named after its own PID next to the drop-in and renames it over the old one, so two runs never share a file and the drop-in is always either the old or the new content. The temporary names fall outside what k3s (config.yaml.d *.yaml, *.yml) and the kubelet (--config-dir *.conf) load. The Metal3 script runs only from ExecStartPre but writes the same drop-in, so it gets the same treatment. The new test refuses an in-place or shared temporary write in the scripts. An ExecStartPre that writes a static providerID is left as it is: the only other writer of that drop-in is write_files, which lands before runcmd starts k3s. The k0s fleet worker comment claimed the boot stage step puts the drop-in in place before the kubelet registers. The ExecStartPre does that; the step is a backstop that can run at the same time. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * fix(bootstrap): queue the kubeadm join unit instead of waiting for it runcmd started the join unit with a plain `systemctl enable --now`. The unit is a oneshot whose discovery retries can run for minutes, and systemctl waited for it to finish, so the cloud-config stage that runs runcmd sat in "Still running stage" for the whole join. Users and SSH keys from the same cloud-config were applied only after it, so the node could not be reached while the join was failing. `--no-block` queues the start job and returns. Nothing after runcmd depends on the join having finished, and the unit's ordering and conditions are evaluated by systemd either way. Fixes: b19dd35 ("feat(bootstrap): add kubeadm as a third distribution for worker joins (#141)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * fix(bootstrap): queue the k3s post-bootstrap unit instead of waiting for it runcmd started kairos-k3s-post-bootstrap.service with a plain `systemctl enable --now`. The unit is a oneshot whose retry loops can run for minutes, and systemctl waits for a oneshot to finish, so the cloud-config stage running runcmd is held for the whole time, and the users and SSH keys from the same cloud-config are applied only after it. `--no-block` queues the start job and returns. Nothing after runcmd depends on the unit having finished: no later runcmd entry, no unit ordered after it, and the bootstrap sentinel it writes is polled asynchronously. The live-installer skip (ConditionKernelCommandLine=!cdroot) stays. Its comment named holding up runcmd as the reason, which no longer applies. The node still cannot register on the installer, and the script would still run to its end: it writes the bootstrap-success sentinel, and on an HA control plane it reports a healthy etcd member under the installer's hostname, both for a node that does not exist. Fixes: 51f3504 ("fix(controlplane): eliminate SSH from kubeconfig retrieval; extend CAPV with node-push pattern (KD-3b) (#55)") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> * revert(bootstrap): keep dnsServers in stages.boot for now This reverts commit f1ac99f ("fix(bootstrap): apply spec.dnsServers before runcmd"). Moving the resolvers to boot.before and renaming a staged file over /etc/resolv.conf changes what spec.dnsServers means, not only when it is applied: - on systemd-resolved images the host loses the DHCP-provided servers and search domains for the whole boot, not just early boot; - the effect differs per distribution: none of the renders set the kubelet resolvConf, so k3s (and likely k0s) pods inherit only the custom servers, while kubeadm detects resolved and keeps DHCP DNS; - on a bind-mounted /etc/resolv.conf (e.g. the docker render path) the rename fails with EBUSY and the resolvers are never applied, where writing through the mount works today. The step goes back to stages.boot, the CRD description and API_REFERENCE go back to main's wording, and the goldens and the stages test that pinned the move are restored or removed. The other commits of this PR (atomic providerID drop-ins, --no-block for the kubeadm and k3s units) stand on their own. The DNS move is split out for a separate decision. Signed-off-by: William Rizzo <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Signed-off-by: Aleksei Sviridkin <[email protected]> Signed-off-by: William Rizzo <[email protected]> Co-authored-by: William Rizzo <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
Draft, opened for visibility. This is the work described in kairos-io/kairos#4920 on 25 September:
kubeadmas a third distribution, starting with the hosted worker case. It should have been visible from the moment it started, and it was not. See the note on #135 below.Setting
spec.distribution: kubeadmon aKairosConfigorKairosConfigTemplatebootstraps a Kairos worker that joins a control plane this provider does not manage: a KamajiKamajiControlPlane, or a Cluster APIKubeadmControlPlane. Kamaji needs no changes on its side.How the join material is obtained
It follows the same contract CABPK does. The controller mints a short-lived bootstrap token in the workload cluster and pins the cluster CA, both reached through
<cluster>-kubeconfig.That Secret is only trusted after a check, because its contents decide which CA a node pins and which API server it joins:
KubeadmControlPlane,KamajiControlPlaneandKairosControlPlane; anything else needs--kubeadm-extra-controlplane-kinds;<cluster>-cawhere that exists;ClusterFilter, so the manager neither dials nor caches arbitrary clusters.Tokens use a 15-minute TTL and rotate at half-TTL, the same rule as CABPK's
shouldRotate, with a 24-hour cap. A machine that is slow to install does not get stranded by an expired token.What runs on the node
The controller marshals a kubeadm
JoinConfigurationwith the CAPI version-aware marshaller and the node runskubeadm join --config, with the provider ID applied through--patches. There is no CABPK, no node-side hook to add to a template, and no Jinja.The join is gated so it happens once and only where it should:
/var/lib/kairosplus kubeadm's ownkubelet.confpreflight, because thekubelet.confcheck alone fails open on affected Kairos versions;The Kubernetes version is resolved through the CAPI contract rather than pinned, and checked against the kubeadm the image actually ships before joining.
API
spec.distributiongainskubeadm, andspec.kubeadm.joinConfigurationtakes the upstream CAPI kubeadm v1beta2 type rather than a hand-rolled subset, so the served CRD does not carry the whole kubeadm bootstrap surface.KairosControlPlanerefusesdistribution: kubeadm; running a kubeadm control plane on Kairos nodes is a separate design and not in this change.Scope
Supported: Kairos workers from a
MachineDeployment. Not supported, and refused: a kubeadmKairosControlPlane,MachinePoolworkers, and joining a k0s or k3sKairosControlPlane. CNI installation and image building are out of scope;docs/KUBEADM_WORKERS.mdcovers the prerequisites.The diff is larger than the feature because the k0s and k3s generators were merged behind one distribution table first, so a third distribution adds a row instead of a third copy. The refactor commits come before the feature commits.
Verification status, stated plainly
Unit tests, golden fixtures and envtest only. This has not been run against a live Kamaji or
KubeadmControlPlanecluster, or against a specific Kairos image. That is why this is a draft. The provider-kubernetes images are the reference for a node image that ships kubeadm, kubelet and containerd, and their Kubernetes range overlaps our validated workload band only at 1.35 and 1.36.Relationship to #135
#135 implements the same
distribution: kubeadmenum value for the same use case, independently, and arrived while this branch was sitting unpublished. The overlap is this repository's fault, not the contributor's. The two differ in where the join executes: #135 renders the Kairoscluster:block and lets provider-kubernetes perform the join, which is a route the internal decision records evaluated and set aside over lifecycle ownership (in-placekubeadm upgrade, its own reset, node-side minting), whereas this change keeps the lifecycle and the PKI on the management side. The specific review points are on #135.