Skip to content

Commit 1b4b6ff

Browse files
authored
test(e2e): warm the ghcr.io mirror before the tenant suites need it (#3768)
## What this PR does The most expensive e2e flake, #3513, is a tenant worker missing the 18m node-Ready budget with zero Nodes registered. The in-guest capture in #3548 diagnosed one variant: the worker's kubelet image pull crawls to ghcr.io at 40-165 KB/s over the runner egress and does not finish inside the budget, Talos holds the kubelet service until its image arrives, so the node never registers and the tenant cilium HelmRelease then fails for want of a schedulable node rather than for any fault of its own. The sandbox already runs an in-cluster pull-through mirror for ghcr.io, but a pull-through cache fetches a blob only when a client first asks for it, so the first tenant worker pays the cold fetch inside its node-Ready budget. Measured with the same fetch script against a local instance of the same registry image: 32m47s cold, 1.1s warm, for the same tags the failing runs stalled on. This PR fills the cache from a Job started by the same install step that deploys the mirror, tens of minutes before the kubernetes suites run. It warms the two kubelet tags the suites actually select, resolved from the chart's version map in the same order the suites resolve it, and one platform per tag (the one the workers run, ~61 MiB each) - warming what nobody pulls would be a true addition to the same throttled egress the install itself is using. The warm-up is best-effort by construction: a Job that fails, times out or never starts does not fail the install, and the suites behave exactly as they did before it existed. The Job itself exits non-zero on any missed tag, so its status stays honest. The node-join failure diagnostics now answer the questions a red run used to leave open: whether a worker pulled through the mirror at all (counting GET and HEAD access lines per request, excluding the warm-up's own requests, which name themselves via user-agent), how long the mirror took to answer (server-side `http.response.duration` is the only after-the-fact signal separating a warm cache from a cold one, since the proxy stores blobs in the background and the Job reports success either way), and whether the warm-up itself got anywhere. The diagnostic section grows from five to seven bounded reads, moving its worst case from roughly 100s to roughly 150s inside the existing phase budget gate; unbudgeted diagnostic residuals stay tracked in #3666. This addresses one variant of #3513 and does not close it: the zero-CSR class remains untouched, and a green suite after this change is not evidence the flake is gone. The 18m node-Ready budget is not raised. Covered by 65 unit tests in `hack/ghcr-mirror_test.bats`: tag selection and ordering against the suites' own expressions, rejection of malformed and injected tag values, Job shape and restricted security context, the readiness wait as a wall clock, every failure path of the new diagnostics including an over-1-MiB log fixture that pins the duration filter against real log sizes, and the exclusion of the warm-up's own traffic from both the request count and the duration report. ### Screenshots No UI changes. ### Downstream repositories The diff is confined to e2e infrastructure under `hack/`: the mirror manifest, the warm-up library and its tests, the install step, and diagnostic labels. Walked the trigger map file by file - no chart values, no APIs, no documentation content, nothing any downstream repository consumes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added best-effort GHCR mirror warm-up during installation. * Preloads selected Kubernetes image manifests and blobs to improve cache readiness. * Warm-up failures safely fall back to direct image pulls. * **Diagnostics** * Improved request classification, timing, repository identification, and reporting of warm-up status and logs. * **Tests** * Expanded coverage for warm-up behavior, validation, retries, security settings, caching, logging, installation, and diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents abd331b + 829edfa commit 1b4b6ff

8 files changed

Lines changed: 1547 additions & 19 deletions

File tree

‎hack/e2e-chainsaw/_lib/ghcr-mirror.sh‎

Lines changed: 431 additions & 5 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1908,8 +1908,8 @@ cozy_report_node_join_failure() {
19081908
# and this one waits, whichever of them is cheaper. Cheaper than the
19091909
# talos-image-cache re-probe below, which creates a Pod and waits on curl
19101910
# retries, so it goes ahead of it.
1911-
if cozy_diag_phase_has_time 'ghcr-mirror state + access log'; then
1912-
echo "--- ghcr-mirror state + access log ---"
1911+
if cozy_diag_phase_has_time 'ghcr-mirror state, access log and warm-up Job'; then
1912+
echo "--- ghcr-mirror state, access log and warm-up Job ---"
19131913
ghcr_mirror_diagnose || true
19141914
fi
19151915

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ spec:
3535
# worker CPU throttling counters
3636
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3737
# guest Talos capture
38-
# ghcr-mirror state + access log dump
38+
# ghcr-mirror state, access log and warm-up Job
3939
# talos-image-cache diagnosis
4040
# and then, after the phase rather than inside it, the in-process tenant
4141
# crust-gather snapshot.

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ spec:
2929
# worker CPU throttling counters
3030
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3131
# guest Talos capture
32-
# ghcr-mirror state + access log dump
32+
# ghcr-mirror state, access log and warm-up Job
3333
# talos-image-cache diagnosis
3434
# and then, after the phase rather than inside it, the in-process tenant
3535
# crust-gather snapshot.

‎hack/e2e-ghcr-mirror.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,10 @@
5050
# context here (non-root 65532, read-only rootfs, dropped caps) and to serve
5151
# ghcr.io/siderolabs/kubelet manifests and blobs as an anonymous pull-through. It is
5252
# deliberately NOT in the pre-pull set: hack/e2e-install-cozystack.bats applies this
53-
# manifest before the pre-pull DaemonSet runs, and one replica pulls it once, so
54-
# adding it there would cache it after the only pull that matters.
53+
# manifest before the pre-pull DaemonSet runs, so adding it there would cache it
54+
# after the pulls that matter. Two Pods now pull it in that window -- this
55+
# Deployment and the warm-up Job the same install step starts -- and the Job carries
56+
# no affinity, so it may land on a second node and pull it again.
5557
apiVersion: v1
5658
kind: Service
5759
metadata:

‎hack/e2e-install-cozystack.bats‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,25 @@
9393
else
9494
echo "WARNING: ghcr-mirror Deployment NOT created — tenant workers will pull ghcr.io directly"
9595
fi
96+
# Start filling the cache now, not when the first tenant worker asks for it.
97+
# An empty pull-through cache makes the first pull a live fetch from ghcr.io, and
98+
# that leg is what the tenant worker cannot afford: measured at 40-165 KB/s for a
99+
# 61 MiB image against an 18m node-Ready budget, with some blob streams dying
100+
# mid-transfer. The kubernetes suites start tens of
101+
# minutes after this step, so a background Job has time to finish.
102+
#
103+
# What is guaranteed: a warm-up that fails, times out or never starts does not
104+
# fail this step, and the suites behave as they did before it existed.
105+
#
106+
# What is NOT guaranteed, because it would be untrue: that this is free. The Job
107+
# pulls the two selected kubelet tags across the same throttled egress the rest of
108+
# the install and the pre-pull step below are using, and if a worker cannot reach
109+
# the mirror (assumption 2 in hack/e2e-ghcr-mirror.yaml) Talos falls back to public
110+
# ghcr.io and the two then compete for it. That is why the warm set is the suites'
111+
# two minors (~61 MiB each, one platform) rather than the whole version map, which
112+
# would be ~300 MiB at the same one-platform walk.
113+
. hack/e2e-chainsaw/_lib/ghcr-mirror.sh
114+
warm_ghcr_mirror || true
96115
}
97116

98117
@test "Required installer chart exists" {

‎hack/ghcr-mirror_test.bats‎

Lines changed: 1088 additions & 7 deletions
Large diffs are not rendered by default.

‎hack/run-kubernetes-node-join_test.bats‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -516,7 +516,7 @@ assert_file_contains() {
516516
assert_file_contains '(b) in-guest Talos dmesg + kubelet logs: not collected' "$tmp/out"
517517
assert_file_contains 're-probe talos-image-cache ClusterIP + cacher debug bundle: not collected' "$tmp/out"
518518
assert_file_contains '(d) tenant worker CPU throttling: not collected' "$tmp/out"
519-
assert_file_contains 'ghcr-mirror state + access log: not collected' "$tmp/out"
519+
assert_file_contains 'ghcr-mirror state, access log and warm-up Job: not collected' "$tmp/out"
520520
rm -rf "$tmp"
521521
}
522522

0 commit comments

Comments
 (0)