Skip to content

Commit 51dfcef

Browse files
test(e2e): drop the ghcr.io pull-through mirror
The mirror was added on the hypothesis that tenant Kubernetes workers missed the node-join deadline because pulling ghcr.io/siderolabs/kubelet from public ghcr.io was rate-limited from the CI runner, and that caching those blobs in-sandbox would take that leg out of the budget. Work on #3513 since then has disproved it. The discriminator between a run whose workers join and a run that stalls is the host's kvm_amd vgif flag, not registry egress, and the stall reproduces with the mirror deployed and its warm-up Job complete. The mirror therefore bought nothing, and it was not free. It is a Deployment, a warm-up Job and a CiliumClusterwideNetworkPolicy applied to every e2e cluster before install, a registry:2 image pulled across the same throttled egress it exists to avoid, a tenant CR knob wired through talos_spec_block, and a failure-path collector competing for the node-join diagnostic budget against the guest serial console -- which is the capture that actually answers why a worker did not join. Removed whole rather than left dormant. The durable knob it exercised is the chart's own spec.talos.registryMirrors, which a production operator points at their own registry, so nothing here was a product artifact and a disabled copy would preserve only the hypothesis. talos_spec_block keeps its Talos OS image cache half (#3231). That is a separate mechanism against a separate failure and stays. The node-join deadline sweep in run-kubernetes-node-join_test.bats has its floor lowered from 18 to 15 because the removed collector quoted the deadline in several comments; the floor is there to catch a pattern that has gone stale, not to pin a count. Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent e84a446 commit 51dfcef

10 files changed

Lines changed: 61 additions & 2618 deletions

‎hack/e2e-chainsaw/_lib/ghcr-mirror.sh‎

Lines changed: 0 additions & 713 deletions
This file was deleted.

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 12 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,22 @@
22
# Sourced by the chainsaw kubernetes-latest/previous Tests after cd to repo root.
33
. hack/e2e-chainsaw/_lib/remediation-guard.sh
44
. hack/e2e-chainsaw/_lib/talos-image-cache.sh
5-
. hack/e2e-chainsaw/_lib/ghcr-mirror.sh
65

7-
# talos_spec_block: emit the tenant Kubernetes CR `spec.talos` block combining the
8-
# Talos OS image cache (imageFactoryURL) and the ghcr.io worker image-pull mirror
9-
# (registryMirrors), each included only when its in-sandbox mirror is up. Prints
10-
# nothing when both fall back to the public defaults, so the chart defaults apply.
6+
# talos_spec_block: emit the tenant Kubernetes CR `spec.talos` block with the Talos
7+
# OS image cache (imageFactoryURL) when the in-sandbox cache is up. Prints nothing
8+
# when it falls back to the public default, so the chart default applies.
119
# Indented for insertion directly under `spec:` in the heredoc below.
1210
#
1311
# No trailing newline, unlike the single-key helper this replaced: the caller reads
1412
# it through `$(...)`, which strips one anyway, and `${talos_block}` sits on a line
1513
# of its own in the heredoc, which supplies the break before the next `spec` key.
1614
# An empty result therefore leaves a blank line, which is valid YAML.
1715
talos_spec_block() {
18-
local url ghcr mirrors
16+
local url
1917
url=$(resolve_talos_image_factory_url)
20-
ghcr=$(resolve_ghcr_mirror_endpoint)
21-
mirrors=$(ghcr_registry_mirrors_block "$ghcr")
22-
[ -n "$url" ] || [ -n "$mirrors" ] || return 0
18+
[ -n "$url" ] || return 0
2319
printf ' talos:\n'
24-
[ -n "$url" ] && printf ' imageFactoryURL: %s\n' "$url"
25-
[ -n "$mirrors" ] && printf '%s' "$mirrors"
20+
printf ' imageFactoryURL: %s\n' "$url"
2621
return 0
2722
}
2823

@@ -3745,11 +3740,8 @@ COZY_DIAG_RATE_INTERVAL_DEFAULT=12
37453740
# snapshot past the end of the window.
37463741
#
37473742
# What a budget of this size gives up first is what is gated last --
3748-
# ghcr_mirror_diagnose and talos_image_cache_diagnose. Neither loses its subject
3749-
# entirely: the mirror's state is partly recoverable from the request counts and
3750-
# response durations the mirror capture records against its own log, and from
3751-
# the kube-system snapshot the host report takes. The image cache's re-probe has
3752-
# no substitute, and it is the collector this budget gives up first.
3743+
# talos_image_cache_diagnose. Its re-probe has no substitute, so it is the
3744+
# collector this budget gives up first.
37533745
#
37543746
# Only part of what runs ahead of the console is held mechanically, and the
37553747
# boundary is worth naming rather than leaving a reader to trust the whole of
@@ -3905,7 +3897,7 @@ cozy_diag_phase_start() {
39053897
# an unchecked list sitting beside a checked one, drifting from it at whatever
39063898
# rate collectors are added.
39073899
command -v timeout >/dev/null 2>&1 || \
3908-
echo "» WARNING: timeout is not on PATH; the bounded reads below run UNBOUNDED, so one that hangs can still take the op and the tenant snapshot with it, and the collectors that call timeout directly (wedge check, serial console, guest Talos) exit 127 and collect nothing; the ones that guard the call with command -v -- the worker CPU throttling, worker network counter, worker block IO counter, sandbox node CPU time, sandbox kernel KVM counters, runner kernel CPU time, sandbox QEMU per-thread CPU time, runner fixed-work canary, worker per-thread CPU time, ghcr-mirror and talos-image-cache captures -- keep collecting instead, unbounded" >&2
3900+
echo "» WARNING: timeout is not on PATH; the bounded reads below run UNBOUNDED, so one that hangs can still take the op and the tenant snapshot with it, and the collectors that call timeout directly (wedge check, serial console, guest Talos) exit 127 and collect nothing; the ones that guard the call with command -v -- the worker CPU throttling, worker network counter, worker block IO counter, sandbox node CPU time, sandbox kernel KVM counters, runner kernel CPU time, sandbox QEMU per-thread CPU time, runner fixed-work canary, worker per-thread CPU time and the talos-image-cache capture -- keep collecting instead, unbounded" >&2
39093901
# Re-checked here, not only at assignment: a value set after this file is sourced
39103902
# -- which is how a test sets it -- would otherwise reach the arithmetic below
39113903
# unvalidated, and that is the one failure that costs the whole block.
@@ -4473,27 +4465,6 @@ cozy_report_node_join_failure() {
44734465
cozy_capture_tenant_talos "${test_name}" || true
44744466
fi
44754467

4476-
# The OS-image cache and the ghcr.io mirror fail independently and produce the
4477-
# same symptom from outside the guest, so both get dumped: this one answers
4478-
# whether the worker's kubelet-image pull reached the mirror or fell back to
4479-
# public ghcr.io, which the node-join failure alone cannot distinguish. Gated
4480-
# like its neighbours, and after the guest captures. Bounded read by read
4481-
# like the collector at (d2), but five of them at COZY_DIAG_READ_TIMEOUT plus
4482-
# grace, so it can spend a quarter of the phase budget -- and time is the
4483-
# only thing the gate rations, so a quarter spent here is a quarter the
4484-
# guest captures do not get. Cost is not what settles the order, though, or
4485-
# (d2) would sit here too: what settles it is whether the answer survives
4486-
# being declined. The console evidence this would starve is irreplaceable
4487-
# and (d2)'s question has no other answer in the tree, while the mirror's
4488-
# state is partly recoverable from the reads above -- so those two go first
4489-
# and this one waits, whichever of them is cheaper. Cheaper than the
4490-
# talos-image-cache re-probe below, which creates a Pod and waits on curl
4491-
# retries, so it goes ahead of it.
4492-
if cozy_diag_phase_has_time 'ghcr-mirror state, access log and warm-up Job'; then
4493-
echo "--- ghcr-mirror state, access log and warm-up Job ---"
4494-
ghcr_mirror_diagnose || true
4495-
fi
4496-
44974468
# Last, and gated on the phase as well as bounded per read, because it is the
44984469
# collector that most needs both: its reachability re-probe creates a Pod, waits
44994470
# on curl retries, and makes seven unbounded management-cluster calls of its own,
@@ -4669,11 +4640,9 @@ YAML
46694640
)
46704641
fi
46714642

4672-
# Point worker DataVolume imports at the in-sandbox Talos OS image cache and
4673-
# worker image pulls at the in-sandbox ghcr.io mirror when each is up (falls back
4674-
# to the public defaults otherwise). Emitted right under spec: as
4675-
# `talos: { imageFactoryURL: ..., registryMirrors: {...} }`, or nothing when both
4676-
# defaults apply.
4643+
# Point worker DataVolume imports at the in-sandbox Talos OS image cache when it
4644+
# is up (falls back to the public default otherwise). Emitted right under spec:
4645+
# as `talos: { imageFactoryURL: ... }`, or nothing when the default applies.
46774646
local talos_block
46784647
talos_block=$(talos_spec_block)
46794648

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,6 @@ spec:
4848
# (those alternate rather than run to completion in turn: each takes
4949
# its first reading, one wait passes, each takes its second)
5050
# guest Talos capture
51-
# ghcr-mirror state, access log and warm-up Job
5251
# talos-image-cache diagnosis
5352
# The four collectors that bracket the node-join wait -- the sandbox
5453
# kernel KVM counters, the runner kernel CPU time, the sandbox QEMU

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,6 @@ spec:
3535
# (those alternate rather than run to completion in turn: each takes
3636
# its first reading, one wait passes, each takes its second)
3737
# guest Talos capture
38-
# ghcr-mirror state, access log and warm-up Job
3938
# talos-image-cache diagnosis
4039
# The four collectors that bracket the node-join wait -- the sandbox
4140
# kernel KVM counters, the runner kernel CPU time, the sandbox QEMU

‎hack/e2e-ghcr-mirror.yaml‎

Lines changed: 0 additions & 195 deletions
This file was deleted.

‎hack/e2e-install-cozystack.bats‎

Lines changed: 0 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -69,56 +69,6 @@
6969
fi
7070
}
7171

72-
@test "Deploy ghcr.io pull-through mirror (best-effort e2e mirror)" {
73-
# Tenant Kubernetes worker Talos nodes pull ghcr.io/siderolabs/kubelet straight
74-
# from public ghcr.io. That egress is intermittently rate-limited from the CI
75-
# runner, the pull dies on a TLS handshake timeout, the kubelet service never
76-
# starts and no tenant node joins (cozystack/cozystack#3513). This Deployment
77-
# caches ghcr.io locally; tenant CRs point spec.talos.registryMirrors at its
78-
# Service. Readiness is gated at point-of-use in
79-
# hack/e2e-chainsaw/_lib/ghcr-mirror.sh, which falls back to direct pulls.
80-
#
81-
# Its own @test, not folded into the Talos image factory cache above: that step
82-
# returns early when it cannot read talos.schematicID/version, and this manifest
83-
# substitutes no placeholders and needs neither value. Sharing a block would let
84-
# a values.yaml restructure silently stop deploying this mirror.
85-
#
86-
# The CiliumClusterwideNetworkPolicy in the manifest is deferred to point-of-use
87-
# for the same reason as above: Cilium's CRDs do not exist until Cozystack is
88-
# installed.
89-
yq 'select(.kind != "CiliumClusterwideNetworkPolicy")' hack/e2e-ghcr-mirror.yaml \
90-
| kubectl apply -f - || echo "WARNING: failed to apply ghcr-mirror (fallback to direct ghcr.io pulls)"
91-
if kubectl -n kube-system get deploy ghcr-mirror >/dev/null 2>&1; then
92-
echo "ghcr-mirror Deployment created (pull-through cache for ghcr.io)"
93-
else
94-
echo "WARNING: ghcr-mirror Deployment NOT created — tenant workers will pull ghcr.io directly"
95-
fi
96-
# Start filling the cache now, not when the first tenant worker asks for it.
97-
# An empty pull-through cache makes the first pull a live fetch from ghcr.io, and
98-
# that leg is what the tenant worker cannot afford: measured at 40-165 KB/s for a
99-
# 61 MiB image, with some blob streams dying mid-transfer. At the low end that
100-
# single pull runs about twenty-six minutes, and it is one leg of a join that
101-
# also has to boot Talos and get a CNI onto the node inside the same node-Ready
102-
# budget. Deliberately not stated as a fraction of that budget: the figure is a
103-
# measured transfer rate and stays true whatever the deadline is set to, while a
104-
# ratio would have to be re-derived every time the deadline moves and would read
105-
# as comfortable the moment it did. The kubernetes suites start tens of minutes
106-
# after this step, so a background Job has time to finish.
107-
#
108-
# What is guaranteed: a warm-up that fails, times out or never starts does not
109-
# fail this step, and the suites behave as they did before it existed.
110-
#
111-
# What is NOT guaranteed, because it would be untrue: that this is free. The Job
112-
# pulls the two selected kubelet tags across the same throttled egress the rest of
113-
# the install and the pre-pull step below are using, and if a worker cannot reach
114-
# the mirror (assumption 2 in hack/e2e-ghcr-mirror.yaml) Talos falls back to public
115-
# ghcr.io and the two then compete for it. That is why the warm set is the suites'
116-
# two minors (~61 MiB each, one platform) rather than the whole version map, which
117-
# would be ~300 MiB at the same one-platform walk.
118-
. hack/e2e-chainsaw/_lib/ghcr-mirror.sh
119-
warm_ghcr_mirror || true
120-
}
121-
12272
@test "Required installer chart exists" {
12373
if [ ! -f packages/core/installer/Chart.yaml ]; then
12474
echo "Missing: packages/core/installer/Chart.yaml" >&2

0 commit comments

Comments
 (0)