Skip to content

Commit ce6bea2

Browse files
lexfreilexfrei
committed
test(e2e): warm the ghcr.io mirror before the tenant suites need it
The in-sandbox pull-through cache fetches a blob only when a client asks for it, so the first tenant worker's kubelet pull is a live fetch from ghcr.io. That leg is what misses the node-Ready budget: measured at 40-165 KB/s for a 61 MiB image, with two blob streams dying mid-transfer after 7-8 MiB on an upstream HTTP/2 reset. Talos holds the kubelet service in PreFunc until the pull completes, so the node never registers, and the tenant cilium HelmRelease then fails for want of a schedulable node rather than for any fault of its own. Fill the cache from a Job started in the same install step that deploys the mirror, tens of minutes before the kubernetes suites run. The Job waits for the mirror to serve, then asks it for each kubelet tag's index, walks the manifests it gets back, and discards the blob bodies; distribution populates its blobstore on the way past. It warms one platform per tag -- the one the workers run -- picked out of the index by record, which keeps the walk a grep. Walking both platforms was a true addition to the same throttled leg: arm64 is bytes nothing in the sandbox pulls, and the digest ordering put it first for both tags, so on the slow runs this exists for the deadline expired with the second tag still cold. It warms the two minors the suites select, not the whole version map. The map would be ~300 MiB at the one-platform walk this ships (~61 MiB per tag), three fifths of it for minors no suite can resolve, spent on the same throttled egress the rest of the install is using. The tag list still comes from the chart's map rather than a copy, ordered the way the suites resolve it, so the set is theirs without restating their expression. The readiness wait is a wall clock, not a per-attempt count. Cilium answers a Service with no ready endpoints by rejecting, so a probe costs nothing when the mirror is absent and the full timeout when it is merely slow; a constant was wrong in both directions across two attempts at picking one. Tags are refused by character class before the pattern is applied, so a value cannot pass by having a valid first line: grep matches per line and the pattern's anchor is end-of-line, not end-of-input. The Job exits non-zero on any miss, so backoffLimit is a real bound and a warm-up that reached nothing cannot report success. It names its own requests, because the node-join diagnostic counts kubelet GETs in the access log to tell a worker that pulled through the mirror from one that fell back to public ghcr.io. Naming them is only half of that. The same diagnostic now reports the Job's status and result and the Job outlives the suites, because excluding the warm-up without reporting it would leave a red run unable to distinguish a warm cache from a warm-up that never started. That counter also moved onto the access-line shape: registry:2.8.3 writes three lines carrying the path per request, so matching the path counted one pull as three. It reads the repository from the same variable the warm-up does, so a rename cannot leave the diagnostic counting zero while the workers pull under the new name. GET and HEAD both count: containerd resolves a manifest with a HEAD first, so a GET-anchored count reports a pull that died before its first GET as no request at all. The failure path also prints how long the mirror took to answer the worker. A warm cache and a cold one are otherwise indistinguishable in the artifacts -- the Job reports success either way, because the proxy stores the blob in the background and a completed GET is a different event from a stored blob -- and server-side duration separates them by orders of magnitude. It filters the log in the shell that read it: a single argv element is capped at 131072 bytes on Linux regardless of ARG_MAX, and probe traffic carries this log past that within about twenty minutes of mirror uptime, so handing it to a helper would empty the section on every run worth reading. Nothing is printed when the log could not be read at all, because a heading with no lines under it reads as a finding about the workers. Measured against a local pull-through registry using the script the builder emits: 32m47s cold, 1.1s warm for the same tags, and the blobs it fetches are the same digests the failing runs stalled on. A warm-up that fails, times out or never starts does not fail the step, and the suites behave as they did before it existed. It is not free, though: it pulls across the same throttled egress the install is using, which is why the warm set is two tags and not five. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent cfaf2e0 commit ce6bea2

7 files changed

Lines changed: 1546 additions & 18 deletions

File tree

‎hack/e2e-chainsaw/_lib/ghcr-mirror.sh‎

Lines changed: 431 additions & 5 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1467,8 +1467,8 @@ cozy_report_node_join_failure() {
14671467
# irreplaceable, and the mirror's state is partly recoverable from the reads
14681468
# above. Cheaper than the talos-image-cache re-probe below, which creates a
14691469
# Pod and waits on curl retries, so it goes ahead of it.
1470-
if cozy_diag_phase_has_time 'ghcr-mirror state + access log'; then
1471-
echo "--- ghcr-mirror state + access log ---"
1470+
if cozy_diag_phase_has_time 'ghcr-mirror state, access log and warm-up Job'; then
1471+
echo "--- ghcr-mirror state, access log and warm-up Job ---"
14721472
ghcr_mirror_diagnose || true
14731473
fi
14741474

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ spec:
3434
# node-join diagnostics reads
3535
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3636
# guest Talos capture
37-
# ghcr-mirror state + access log dump
37+
# ghcr-mirror state, access log and warm-up Job
3838
# talos-image-cache diagnosis
3939
# and then, after the phase rather than inside it, the in-process tenant
4040
# crust-gather snapshot.

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ spec:
2828
# node-join diagnostics reads
2929
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3030
# guest Talos capture
31-
# ghcr-mirror state + access log dump
31+
# ghcr-mirror state, access log and warm-up Job
3232
# talos-image-cache diagnosis
3333
# and then, after the phase rather than inside it, the in-process tenant
3434
# crust-gather snapshot.

‎hack/e2e-ghcr-mirror.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,10 @@
5050
# context here (non-root 65532, read-only rootfs, dropped caps) and to serve
5151
# ghcr.io/siderolabs/kubelet manifests and blobs as an anonymous pull-through. It is
5252
# deliberately NOT in the pre-pull set: hack/e2e-install-cozystack.bats applies this
53-
# manifest before the pre-pull DaemonSet runs, and one replica pulls it once, so
54-
# adding it there would cache it after the only pull that matters.
53+
# manifest before the pre-pull DaemonSet runs, so adding it there would cache it
54+
# after the pulls that matter. Two Pods now pull it in that window -- this
55+
# Deployment and the warm-up Job the same install step starts -- and the Job carries
56+
# no affinity, so it may land on a second node and pull it again.
5557
apiVersion: v1
5658
kind: Service
5759
metadata:

‎hack/e2e-install-cozystack.bats‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,25 @@
9393
else
9494
echo "WARNING: ghcr-mirror Deployment NOT created — tenant workers will pull ghcr.io directly"
9595
fi
96+
# Start filling the cache now, not when the first tenant worker asks for it.
97+
# An empty pull-through cache makes the first pull a live fetch from ghcr.io, and
98+
# that leg is what the tenant worker cannot afford: measured at 40-165 KB/s for a
99+
# 61 MiB image against an 18m node-Ready budget, with some blob streams dying
100+
# mid-transfer. The kubernetes suites start tens of
101+
# minutes after this step, so a background Job has time to finish.
102+
#
103+
# What is guaranteed: a warm-up that fails, times out or never starts does not
104+
# fail this step, and the suites behave as they did before it existed.
105+
#
106+
# What is NOT guaranteed, because it would be untrue: that this is free. The Job
107+
# pulls the two selected kubelet tags across the same throttled egress the rest of
108+
# the install and the pre-pull step below are using, and if a worker cannot reach
109+
# the mirror (assumption 2 in hack/e2e-ghcr-mirror.yaml) Talos falls back to public
110+
# ghcr.io and the two then compete for it. That is why the warm set is the suites'
111+
# two minors (~61 MiB each, one platform) rather than the whole version map, which
112+
# would be ~300 MiB at the same one-platform walk.
113+
. hack/e2e-chainsaw/_lib/ghcr-mirror.sh
114+
warm_ghcr_mirror || true
96115
}
97116

98117
@test "Required installer chart exists" {

0 commit comments

Comments
 (0)