Skip to content

Commit 556c1a9

Browse files
feat(kubernetes-nodes): boot worker disks from a shared golden Talos image (CDI clone)
Rebase of #3294 onto main after the Phase 2 KubernetesNodes split (#3315). The worker disk source and the in-guest installer pin move from packages/apps/kubernetes to packages/apps/kubernetes-nodes, where the pool objects now live; the per-node-group `nodeGroups.<name>.image` union becomes the pool-level `image` value of the KubernetesNodes chart. Adds packages/system/kubernetes-worker-image, an opt-in catalog that imports a golden Talos worker image into cozy-public once per (schematicID, version). A pool that sets image.builtin CDI-clones that golden instead of streaming the raw image over HTTP per worker, which is the per-worker Image Factory dependency behind the kubernetes-* node-join flake (#3231). image.factory keeps the HTTP path and lets a pool point at its own mirror/schematic/version. With `image` omitted the render is byte-identical to before -- the stored render snapshot is unchanged, so existing workers do not roll on upgrade. The e2e suites switch to the clone path: the per-worker talos-image-cache mirror and its manifest, helper and tests are removed, the catalog is enabled in the sandbox, and the run waits for the golden to import before creating the tenant. Neither tenant CR carries a spec.talos override any more. The ghcr.io pull-through that used to share that block was already removed with the QEMU merge-gating lane (#4020), so both CRs now take the chart defaults. Two things the cache took with it. The successful-join timing report loses its cache-transfer arm: a cloned disk is populated in the storage layer, so there is no compressed-byte service time to report per worker, and DataVolume Pending -> Succeeded is the whole of the disk's own cost. And the node-join failure path loses the cache re-probe, which was the collector its diagnostic budget gave up first; what replaced it, the golden's own DataVolume, is read inside the budget at (a2) rather than at its mercy. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 9e577a5 commit 556c1a9

32 files changed

Lines changed: 1104 additions & 1687 deletions

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 30 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go‎

Lines changed: 56 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Ties the kubernetes-nodes chart's default Talos image to the kubernetes-worker-image
4+
# catalog's default golden. The two live in independently edited files:
5+
#
6+
# packages/apps/kubernetes-nodes/values.yaml talos.{schematicID,version}
7+
# packages/system/kubernetes-worker-image/values.yaml images[], storageClass
8+
#
9+
# A worker pool that opts in with `image.builtin: {}` resolves the golden it
10+
# clones from the POOL chart's talos defaults, then requires the catalog to hold a
11+
# matching (schematicID, version) — packages/apps/kubernetes-nodes/templates/
12+
# nodegroup.yaml fails the whole render when it does not. So a one-sided Talos bump
13+
# silently breaks every fresh install using image.builtin, and nothing catches it
14+
# until a ~95-minute e2e run does.
15+
#
16+
# The StorageClass is the same trap from the other direction: CDI cannot
17+
# CSI-clone across StorageClasses (it would silently fall back to a host-assisted
18+
# copy over the pod network), so the render also rejects a group whose
19+
# storageClass differs from its golden's. Bumping one file's default alone brings
20+
# that guard down on the default path.
21+
#
22+
# Both checks are cheap and exact, which is the point — they turn a slow, remote
23+
# e2e failure into a local one-second unit test.
24+
#
25+
# Compatible with both `bats` directly and the in-repo cozytest.sh runner, which
26+
# runs each @test in a fresh subshell with `set -u` and does not honor bats
27+
# setup()/teardown().
28+
29+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
30+
APP_VALUES="$REPO_ROOT/packages/apps/kubernetes-nodes/values.yaml"
31+
CATALOG_VALUES="$REPO_ROOT/packages/system/kubernetes-worker-image/values.yaml"
32+
33+
@test "worker image catalog holds a golden for the pool chart's default Talos (schematicID, version)" {
34+
sid=$(yq '.talos.schematicID' "$APP_VALUES")
35+
ver=$(yq '.talos.version' "$APP_VALUES")
36+
[ -n "$sid" ] && [ "$sid" != "null" ]
37+
[ -n "$ver" ] && [ "$ver" != "null" ]
38+
39+
# Any entry may match — the catalog is a list of flavors, not an ordered pair.
40+
if ! yq '.images[] | .schematicID + " " + .version' "$CATALOG_VALUES" \
41+
| grep -qxF "$sid $ver"; then
42+
echo "packages/apps/kubernetes-nodes/values.yaml defaults to Talos:" >&2
43+
echo " schematicID: $sid" >&2
44+
echo " version: $ver" >&2
45+
echo "but the kubernetes-worker-image catalog has no matching images[] entry:" >&2
46+
yq '.images[] | " - " + .schematicID + " " + .version' "$CATALOG_VALUES" >&2
47+
echo "Every worker pool using 'image.builtin: {}' would fail to render." >&2
48+
echo "Add the pair to the catalog, or bump both files together." >&2
49+
return 1
50+
fi
51+
}
52+
53+
@test "worker image catalog golden shares the pool chart's default StorageClass" {
54+
sid=$(yq '.talos.schematicID' "$APP_VALUES")
55+
ver=$(yq '.talos.version' "$APP_VALUES")
56+
app_sc=$(yq '.storageClass' "$APP_VALUES")
57+
58+
# The golden's effective class is its own override when set, else the catalog
59+
# default — mirroring how the pool chart resolves its class.
60+
entry_sc=$(SID="$sid" VER="$ver" yq \
61+
'.images[] | select(.schematicID == strenv(SID) and .version == strenv(VER)) | .storageClass' \
62+
"$CATALOG_VALUES")
63+
if [ -z "$entry_sc" ] || [ "$entry_sc" = "null" ]; then
64+
entry_sc=$(yq '.storageClass' "$CATALOG_VALUES")
65+
fi
66+
67+
if [ "$app_sc" != "$entry_sc" ]; then
68+
echo "StorageClass mismatch on the default image.builtin path:" >&2
69+
echo " packages/apps/kubernetes-nodes/values.yaml storageClass: $app_sc" >&2
70+
echo " kubernetes-worker-image golden effective storageClass: $entry_sc" >&2
71+
echo "CDI cannot CSI-clone across StorageClasses, so the tenant render" >&2
72+
echo "rejects this outright. Keep the two defaults in step." >&2
73+
return 1
74+
fi
75+
}

‎hack/e2e-chainsaw/_lib/etcd-probe.sh‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,12 @@
2626
#
2727
# The recovery is that the container's stdout is not actually gone: kubelet
2828
# wrote it to the Pod's log. So a lost stream is re-read rather than believed.
29-
# hack/e2e-chainsaw/_lib/talos-image-cache.sh solves the same race inline for
30-
# the image-cache probe (root cause from the same class of bug); that inline
31-
# copy is what identified this call site as still carrying the race. This is
32-
# the extracted, unit-tested form — see hack/etcd-probe_test.bats. It is not
33-
# yet shared: talos-image-cache.sh still carries its own copy, and folding it
34-
# onto this helper would need a name that is not etcd-specific.
29+
# The pattern was first written inline in the Talos image-cache probe, and that
30+
# inline copy is what identified this call site as carrying the same race; the
31+
# image-cache probe has since been removed, so this extracted, unit-tested form
32+
# — see hack/etcd-probe_test.bats — is the only copy left. Renaming it to
33+
# something that is not etcd-specific is worth doing when a second caller
34+
# appears, not before.
3535
#
3636
# Three conditions make the re-read safe rather than a way to mask real
3737
# failures. The marker encodes the outcome, so a `code=503` recovered from the

0 commit comments

Comments
 (0)