Skip to content

Commit 829edfa

Browse files
committed
test(e2e): warm the ghcr.io mirror before the tenant suites need it
The in-sandbox pull-through cache fetches a blob only when a client asks for it, so the first tenant worker's kubelet pull is a live fetch from ghcr.io. That leg is what misses the node-Ready budget: measured at 40-165 KB/s for a 61 MiB image, with two blob streams dying mid-transfer after 7-8 MiB on an upstream HTTP/2 reset. Talos holds the kubelet service in PreFunc until the pull completes, so the node never registers, and the tenant cilium HelmRelease then fails for want of a schedulable node rather than for any fault of its own. Fill the cache from a Job started in the same install step that deploys the mirror, tens of minutes before the kubernetes suites run. The Job waits for the mirror to serve, then asks it for each kubelet tag's index, walks the manifests it gets back, and discards the blob bodies; distribution populates its blobstore on the way past. It warms one platform per tag -- the one the workers run -- picked out of the index by record, which keeps the walk a grep. Walking both platforms was a true addition to the same throttled leg: arm64 is bytes nothing in the sandbox pulls, and the digest ordering put it first for both tags, so on the slow runs this exists for the deadline expired with the second tag still cold. It warms the two minors the suites select, not the whole version map. The map would be ~300 MiB at the one-platform walk this ships (~61 MiB per tag), three fifths of it for minors no suite can resolve, spent on the same throttled egress the rest of the install is using. The tag list still comes from the chart's map rather than a copy, ordered the way the suites resolve it, so the set is theirs without restating their expression. The readiness wait is a wall clock, not a per-attempt count. Cilium answers a Service with no ready endpoints by rejecting, so a probe costs nothing when the mirror is absent and the full timeout when it is merely slow; a constant was wrong in both directions across two attempts at picking one. Tags are refused by character class before the pattern is applied, so a value cannot pass by having a valid first line: grep matches per line and the pattern's anchor is end-of-line, not end-of-input. The Job exits non-zero on any miss, so backoffLimit is a real bound and a warm-up that reached nothing cannot report success. It names its own requests, because the node-join diagnostic counts kubelet GETs in the access log to tell a worker that pulled through the mirror from one that fell back to public ghcr.io. Naming them is only half of that. The same diagnostic now reports the Job's status and result and the Job outlives the suites, because excluding the warm-up without reporting it would leave a red run unable to distinguish a warm cache from a warm-up that never started. That counter also moved onto the access-line shape: registry:2.8.3 writes three lines carrying the path per request, so matching the path counted one pull as three. It reads the repository from the same variable the warm-up does, so a rename cannot leave the diagnostic counting zero while the workers pull under the new name. GET and HEAD both count: containerd resolves a manifest with a HEAD first, so a GET-anchored count reports a pull that died before its first GET as no request at all. The failure path also prints how long the mirror took to answer the worker. A warm cache and a cold one are otherwise indistinguishable in the artifacts -- the Job reports success either way, because the proxy stores the blob in the background and a completed GET is a different event from a stored blob -- and server-side duration separates them by orders of magnitude. It filters the log in the shell that read it: a single argv element is capped at 131072 bytes on Linux regardless of ARG_MAX, and probe traffic carries this log past that within about twenty minutes of mirror uptime, so handing it to a helper would empty the section on every run worth reading. Nothing is printed when the log could not be read at all, because a heading with no lines under it reads as a finding about the workers. Measured against a local pull-through registry using the script the builder emits: 32m47s cold, 1.1s warm for the same tags, and the blobs it fetches are the same digests the failing runs stalled on. A warm-up that fails, times out or never starts does not fail the step, and the suites behave as they did before it existed. It is not free, though: it pulls across the same throttled egress the install is using, which is why the warm set is two tags and not five. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent abd331b commit 829edfa

8 files changed

Lines changed: 1547 additions & 19 deletions

File tree

‎hack/e2e-chainsaw/_lib/ghcr-mirror.sh‎

Lines changed: 431 additions & 5 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1908,8 +1908,8 @@ cozy_report_node_join_failure() {
19081908
# and this one waits, whichever of them is cheaper. Cheaper than the
19091909
# talos-image-cache re-probe below, which creates a Pod and waits on curl
19101910
# retries, so it goes ahead of it.
1911-
if cozy_diag_phase_has_time 'ghcr-mirror state + access log'; then
1912-
echo "--- ghcr-mirror state + access log ---"
1911+
if cozy_diag_phase_has_time 'ghcr-mirror state, access log and warm-up Job'; then
1912+
echo "--- ghcr-mirror state, access log and warm-up Job ---"
19131913
ghcr_mirror_diagnose || true
19141914
fi
19151915

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ spec:
3535
# worker CPU throttling counters
3636
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3737
# guest Talos capture
38-
# ghcr-mirror state + access log dump
38+
# ghcr-mirror state, access log and warm-up Job
3939
# talos-image-cache diagnosis
4040
# and then, after the phase rather than inside it, the in-process tenant
4141
# crust-gather snapshot.

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ spec:
2929
# worker CPU throttling counters
3030
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3131
# guest Talos capture
32-
# ghcr-mirror state + access log dump
32+
# ghcr-mirror state, access log and warm-up Job
3333
# talos-image-cache diagnosis
3434
# and then, after the phase rather than inside it, the in-process tenant
3535
# crust-gather snapshot.

‎hack/e2e-ghcr-mirror.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,10 @@
5050
# context here (non-root 65532, read-only rootfs, dropped caps) and to serve
5151
# ghcr.io/siderolabs/kubelet manifests and blobs as an anonymous pull-through. It is
5252
# deliberately NOT in the pre-pull set: hack/e2e-install-cozystack.bats applies this
53-
# manifest before the pre-pull DaemonSet runs, and one replica pulls it once, so
54-
# adding it there would cache it after the only pull that matters.
53+
# manifest before the pre-pull DaemonSet runs, so adding it there would cache it
54+
# after the pulls that matter. Two Pods now pull it in that window -- this
55+
# Deployment and the warm-up Job the same install step starts -- and the Job carries
56+
# no affinity, so it may land on a second node and pull it again.
5557
apiVersion: v1
5658
kind: Service
5759
metadata:

‎hack/e2e-install-cozystack.bats‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,25 @@
9393
else
9494
echo "WARNING: ghcr-mirror Deployment NOT created — tenant workers will pull ghcr.io directly"
9595
fi
96+
# Start filling the cache now, not when the first tenant worker asks for it.
97+
# An empty pull-through cache makes the first pull a live fetch from ghcr.io, and
98+
# that leg is what the tenant worker cannot afford: measured at 40-165 KB/s for a
99+
# 61 MiB image against an 18m node-Ready budget, with some blob streams dying
100+
# mid-transfer. The kubernetes suites start tens of
101+
# minutes after this step, so a background Job has time to finish.
102+
#
103+
# What is guaranteed: a warm-up that fails, times out or never starts does not
104+
# fail this step, and the suites behave as they did before it existed.
105+
#
106+
# What is NOT guaranteed, because it would be untrue: that this is free. The Job
107+
# pulls the two selected kubelet tags across the same throttled egress the rest of
108+
# the install and the pre-pull step below are using, and if a worker cannot reach
109+
# the mirror (assumption 2 in hack/e2e-ghcr-mirror.yaml) Talos falls back to public
110+
# ghcr.io and the two then compete for it. That is why the warm set is the suites'
111+
# two minors (~61 MiB each, one platform) rather than the whole version map, which
112+
# would be ~300 MiB at the same one-platform walk.
113+
. hack/e2e-chainsaw/_lib/ghcr-mirror.sh
114+
warm_ghcr_mirror || true
96115
}
97116

98117
@test "Required installer chart exists" {

‎hack/ghcr-mirror_test.bats‎

Lines changed: 1088 additions & 7 deletions
Large diffs are not rendered by default.

‎hack/run-kubernetes-node-join_test.bats‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -516,7 +516,7 @@ assert_file_contains() {
516516
assert_file_contains '(b) in-guest Talos dmesg + kubelet logs: not collected' "$tmp/out"
517517
assert_file_contains 're-probe talos-image-cache ClusterIP + cacher debug bundle: not collected' "$tmp/out"
518518
assert_file_contains '(d) tenant worker CPU throttling: not collected' "$tmp/out"
519-
assert_file_contains 'ghcr-mirror state + access log: not collected' "$tmp/out"
519+
assert_file_contains 'ghcr-mirror state, access log and warm-up Job: not collected' "$tmp/out"
520520
rm -rf "$tmp"
521521
}
522522

0 commit comments

Comments
 (0)