Skip to content

Commit 677d80a

Browse files
authored
feat(kubernetes): route tenant worker image pulls through a registry mirror (#3575)
## What this PR does Refs #3513. It addresses one variant of that flake and does not close it; see "Scope against #3513" below. ### Root cause of the variant this addresses The chainsaw `kubernetes-latest` / `kubernetes-previous` suites fail with `node-join failed: fewer than 2 tenant nodes Ready within 18m` and zero registered tenant nodes. The in-guest Talos capture on #3548 shows where the time goes on one class of those runs: the worker loops trying to pull `ghcr.io/siderolabs/kubelet` from public ghcr.io and fails with `Head "https://ghcr.io/v2/siderolabs/kubelet/manifests/...": net/http: TLS handshake timeout host=ghcr.io`. The Talos `kubelet` service never starts (`kubelet was not registered`), so the node never registers, cilium-operator stays `Pending` for lack of a node to schedule on, and the budget expires. On those runs the tenant apiserver is healthy and the worker VMs boot with kube-ovn IPs, so the failing link is the worker's public image-pull egress, and the worker machine config has no `machine.registries.mirrors`, so there is no way to route around it. Same class as the Talos OS image, which #3231 solved with the `talos.imageFactoryURL` cache. The thread also carries a later analysis putting worker-VM CPU starvation from nested-virt oversubscription at the root, with serial-console RCU stalls showing the guest never reaching Talos userland, and reading the TLS-handshake variant as a symptom of the same contention. Nothing here contradicts that. Removing a live dependency on public egress from worker bring-up is worth doing on its own terms, and if the contention account is the dominant one, this narrows what is left to explain rather than competing with it. ### The change **Chart (`packages/apps/kubernetes`, `packages/apps/kubernetes-nodes`)**, a `talos.registryMirrors` passthrough, a map of upstream registry host to `{ endpoints: [ ... ] }`, rendered into `machine.registries.mirrors` of the worker `TalosConfigTemplate` that the MachineDeployment clones from, so it applies at first boot rather than after it. Empty by default: with the default values the rendered chart is byte-identical to main, both content-hash object names included, so no existing worker rolls. This is the durable half; an air-gapped, rate-limited or flaky-egress environment can point worker image pulls at its own mirror, which is the `machine.registries.mirrors` knob the Talos rollover left unfilled. Talos still falls back to the upstream registry unless a host also sets `skipFallback`, so a mirror on its own is not air-gap enforcement, and the field doc says so. The value lands inside the reconcile Job's unquoted heredoc, so it is escaped for backslash, dollar and backtick and renders as a literal. `talos.installerRepository`, `talos.schematicID` and the Talos version render into that same heredoc one block above it and now get the same treatment, so the file is consistent about it rather than escaping only the newest field. Escaping the rendered value covers every byte by construction, which enumerating known-bad shapes does not. The chain is a no-op for values carrying none of those characters, so the default render, content-hash Job name included, is unchanged and no existing worker is replaced. **e2e harness**, an in-sandbox `registry:2` pull-through cache for ghcr.io (`hack/e2e-ghcr-mirror.yaml`), applied at install time from its own step, with `hack/e2e-chainsaw/_lib/ghcr-mirror.sh` setting `spec.talos.registryMirrors` on the tenant CR when the mirror is up and its egress allow is in place, and emitting nothing otherwise so workers pull directly. A `CiliumClusterwideNetworkPolicy` opens one tightly-scoped hole for worker VM (`kubevirt.io: virt-launcher`) Pods in `tenant-test` to reach the mirror, mirroring the existing `talos-image-cache` policy. Two properties of the helper worth calling out, because both are easy to get wrong and both are now pinned by tests. A failed API call is never cached: the decision is reused by every later suite in the shared sandbox, so caching one blip would disable the mirror for the whole run. Readiness is decided once and acted on once. `rollout status` is the cheap way to learn it, but it also exits non-zero on a broken watch, so when it fails the Deployment's own `Available` condition is read rather than the watch's exit code taken as the answer. Exactly one failure is cached, a definite `Available=False`, and that is a trade rather than an invariant: it avoids re-paying the wait in every later suite, and costs a mirror that recovers mid-run. On a node-join failure the suite dumps the mirror's state alongside the Talos image cache it already dumped. Workers being pointed at the mirror and then joining does not establish that the mirror served anything, because Talos falls back to public ghcr.io on its own; the registry's access log is what separates the two, so the dump counts kubelet-image requests across the whole log rather than tailing it. The readiness probe writes an access line every five seconds, so by the time a run has spent its 18m budget a fixed tail window is hundreds of lines short of the request worth finding. ### Testing - `helm unittest packages/apps/kubernetes`, 205 tests, 22 suites; `helm unittest packages/apps/kubernetes-nodes`, 14 tests, 4 suites. New cases: the default omits `machine.registries.mirrors`, and a set value renders it with the endpoint. - `hack/ghcr-mirror_test.bats`, 23 tests. Beyond the manifest's two-phase split, egress identities, pull-through config and the pure YAML builder, this covers every outcome of `resolve_ghcr_mirror_endpoint` against a stubbed `kubectl`: an API NotFound caches, a transient query failure does not, a shell reporting a missing binary is not mistaken for either, a failed egress allow does not cache and its reason reaches the log, a rollout watch that failed for a reason the API does not confirm does not cache while a definite `Available=False` does, and a committed endpoint is cached and short-circuits the next call. The stub honours `--tail`, so the diagnostic's tests tell a filtered read from a truncated one instead of trusting the flag, and it reports errors on stderr as the real binary does. The suite also pins that the install step excludes the Cilium policy from its pre-Cilium apply, not only what that exclusion produces. - `hack/talos-reconcile-heredoc_test.bats`, 5 tests, which render both charts and run the heredoc through a real shell with hostile values rather than regexing the rendered string. A `matchRegex` cannot catch a heredoc the shell refuses to emit; this runs it. - `hack/run-kubernetes-talos-spec_test.bats`, 5 tests over `talos_spec_block`, the point where the two optional fragments are merged. The assertions splice the result under `spec:` and read it back with yq, so a block indented into the wrong parent fails instead of passing. - `packages/apps/kubernetes-nodes/tests/render-parity.sh` byte-identical; per-chart `make generate` in both charts with no drift; both `cozyrds` `openAPISchema` blocks equal to their `values.schema.json`. - The pinned `mirror.gcr.io/library/registry:2.8.3@sha256:a3d8aaa6...` is the digest Docker Hub serves for that tag, and `mirror.gcr.io` serves the same OCI index with a linux/amd64 manifest. Run under exactly the security context the Deployment sets, non-root 65532, read-only root filesystem, all capabilities dropped, writable state only on the mounted volumes, it starts, answers `/v2/` with 200, and serves `ghcr.io/siderolabs/kubelet` manifests and blobs as an anonymous pull-through. - `shellcheck` and `git diff --check` clean on the new and edited shell. ### What a CI run still has to answer Two runtime facts the harness half rests on, both recorded in the manifest: 1. Talos honours a plain-http mirror endpoint via `machine.registries.mirrors[ghcr.io].endpoints=[http://...]`. If a scheme or TLS knob is additionally needed, `machine.registries.config` has to be added too. 2. The containerd pull from inside the worker VM leaves with the virt-launcher Pod's own Cilium identity, which is what the policy selects. The label half is settled: `run-kubernetes.sh` already reads tenant worker Pods by `kubevirt.io=virt-launcher`, but whether the in-guest pull is subject to that Pod's egress policy at all is not. The second one bounds how strong a claim the harness can make. The gate is a `rollout status` plus an accepted egress-allow object, and neither observes the tenant-side datapath, so the fallback paths cannot make CI worse while the committed path is only as safe as that assumption. The in-repo comments say this rather than claiming the mirror can only ever help. ### Named improvements left out A tenant-scoped reachability probe, like the byte-level 206 check `talos-image-cache.sh` runs from a Pod carrying the consumer's own label, is what turns assumption 2 from an assertion into a measurement, and would also let the suite assert the mirror served the kubelet pull instead of the worker quietly falling back. Left out because a probe that does not exercise the tenant network path would pass while real workers stayed blocked, and the one that does is a change of its own size. The same gap has a cheaper edge: both the install step and the resolver key on the Deployment alone, so an apply that made the Deployment but not the Service still reads as success. That one costs little in practice, since the guest gets a fast NXDOMAIN instead of a dropped SYN, but the gate does check less than it commits to. `registryMirrors` could be a typed `map[string]struct{ endpoints []string }`, the way `nodeGroups` already is in this chart, so malformed input is rejected at admission instead of failing late inside Talos. The helm-unittest escape cases assert the `$` path; the backtick path is covered by the execution-level heredoc bats and the pure-backslash path by the escape chain's construction, so the security contract is fully exercised across the two layers. A backtick assertion at the helm-unittest layer would express it at the fast layer too. The escape chain is spelled out at five interpolation sites across the two reconcile-Job templates. A named helper (an `include` template) would make the escape a single reusable primitive and shrink the surface where the heredoc invariant can be forgotten; the current per-site form is correct and covered, so this is a readability refactor rather than a fix. Only `machine.registries.mirrors` is passed through. `machine.registries.config` is not, so a mirror behind a private CA or one needing credentials still has no knob, and the platform-wide `registries` value does not reach tenant workers. The chart's breaking-change note says so, and also says which knob covers which artifact: the mirror routes registry pulls, while the worker OS disk image arrives as a raw HTTP artifact under `talos.imageFactoryURL` and no registry mirror can touch it. The helper's failure model is worth revisiting as a whole rather than per call site. Every call it makes to the cluster has three outcomes (the world says no, the world says yes, the call could not be made) and a shell exit code carries two, so the third is re-derived at each site from a different signal: stderr text for a NotFound, a field value for readiness, the returned message for a failed apply. Since the resulting decision is cached for the whole sandbox, a misclassification is permanent for the run rather than a blip, which makes each of those sites load-bearing. A single prober answering `yes`/`no`/`unknown`, with the caching rule stated once over that answer, would move the question out of the branches; the current shape reaches the same behaviour but re-establishes it five times. Left out here because it is a visible rework of code this PR only extends, and it would need its own verification of every site; tracked in #3682. One nicety left in the same helper: the diagnostic reads the mirror log twice, once with `--tail=-1` to count kubelet-image requests across the whole log and once with `--tail=50 --prefix` for a bounded context tail. The second read exists for `--prefix`, which attributes each line to its pod; collapsing to one read drops that. The attribution is worthless while the mirror is a single replica and load-bearing the moment it is not, in the one artifact read after a failure, and the second read is itself wall-clock bounded, so it is left in place. ### Merge-order dependency (resolved) #3676 landed first and this branch is rebased onto it. `ghcr_mirror_diagnose` now runs in the node-join failure block under the same phase gate as its neighbours, and each of its reads is wall-clock bounded by the same `COZY_DIAG_READ_TIMEOUT`/`COZY_DIAG_READ_GRACE` the block validates, so no single read can hold the op open. It is placed after the guest captures and before the image-cache re-probe: the console evidence is irreplaceable and must not be starved of budget, and the dump is still cheaper than the Pod-creating re-probe, so the spend order puts it between the two. When the diagnostics phase runs out of budget the dump is declined out loud instead of attempted. An ungated dump would spend that time anyway and take it from the captures behind it, while the mirror's state is partly recoverable from the reads that already ran, so gating costs less than it saves. The known cost is #3686: cheap reads ahead of the gate can drain the whole budget on a viscous apiserver, and the dump is then declined in exactly the runs where the mirror is a suspect. That is a defect of the budget, not of this dump, and it is tracked there. ### Scope against #3513 #3513 asks for three things: capture the guest side, act on the evidence, and do not raise the 18m budget again. The capture landed in #3548 and this consumes its output; the budget is untouched. What this acts on is the TLS-handshake variant only. The same thread documents runs that issue no apid CSR at all inside the whole budget, time lost before Talos reaches the kubelet image, which no registry mirror can affect. So #3513 stays open after this merges, and a green suite afterwards is not evidence the flake is closed. The manifest header says so too, for whoever reads it next. ### Both tenant suites are covered `kubernetes-previous` selects the previous Kubernetes *minor* out of `packages/apps/kubernetes/files/versions.yaml`, `run_kubernetes_test 'keys | sort_by(.) | .[-2]'`, not a previously-released chart. Both suites build their tenant CR from the same in-tree chart through the same helper, so both get the mirror. ### Screenshots Not applicable; no UI changes. ### Downstream repositories Walking the trigger map in `docs/agents/contributing.md` against the diff: **cozystack/terraform-provider-cozystack is affected**. The provider is hand-written, and the map lists "add, remove or rename a field in an app's `values.schema.json`" and "change a default in an app's `values.yaml`" as triggers for its schema, model and expand/flatten pair. This adds `talos.registryMirrors` with a `{}` default to both the `kubernetes` and `kubernetes-nodes` schemas. No other repository matches a trigger. The website's managed-app reference pages regenerate from each package's `README.md` on a stable tag, and this only changes an existing package's README. I have not opened the provider follow-up and have not ticked the box, because a ticked box with no link claims work that does not exist. Leaving it to a maintainer to decide whether the provider models this field and who files it. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(kubernetes): add `talos.registryMirrors` to route tenant worker node image pulls through a registry mirror for air-gapped or flaky-egress environments ```
2 parents 02a3fda + 3ae1e10 commit 677d80a

26 files changed

Lines changed: 1482 additions & 44 deletions

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 2 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go‎

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
# shellcheck shell=bash
2+
# Shared helper: point tenant Kubernetes e2e worker nodes at the in-sandbox
3+
# ghcr.io pull-through registry (hack/e2e-ghcr-mirror.yaml) when it is up, via the
4+
# chart's `talos.registryMirrors` knob, otherwise emit nothing so workers fall back
5+
# to pulling ghcr.io directly.
6+
#
7+
# Why: tenant worker Talos nodes pull `ghcr.io/siderolabs/kubelet` directly; that
8+
# egress is flaky/rate-limited from the CI runner and the pull times out with a TLS
9+
# handshake timeout, so the kubelet service never starts and no tenant node joins.
10+
# Diagnosed in cozystack/cozystack#3548 (in-guest Talos dmesg), tracked by #3513.
11+
# Same flaky-public-egress class the talos-image-cache (#3231) fixed for the OS image.
12+
# One variant of that flake, not the whole of it: #3513 also records failures that
13+
# lose the budget before Talos ever reaches the kubelet pull, and those this cannot
14+
# touch.
15+
#
16+
# WHAT THE GATE BELOW ESTABLISHES, precisely, because it is easy to overstate: the
17+
# Deployment went Available, and the API accepted the egress allow. Neither observes
18+
# the tenant-side datapath. If the allow selects an identity the worker's pull does
19+
# not actually carry (see assumption 2 in hack/e2e-ghcr-mirror.yaml), workers are
20+
# pointed at a ClusterIP whose SYNs are dropped, and Talos burns a TCP connect
21+
# timeout per pull before falling back to ghcr.io -- slower than not mirroring at
22+
# all, on exactly the runs this exists to protect. So: the fallback paths cannot
23+
# make CI worse, the committed path is only as safe as the assumptions listed
24+
# there.
25+
#
26+
# talos-image-cache.sh gates the equivalent decision on a byte-level 206 from a Pod
27+
# carrying the consumer's own label, which is what turns that assumption into a
28+
# measurement. Doing the same here is the hardening follow-up, and it is also what
29+
# would let the suite assert the mirror served the pull rather than the worker
30+
# quietly falling back.
31+
32+
GHCR_MIRROR_SVC_URL="${GHCR_MIRROR_SVC_URL:-http://ghcr-mirror.kube-system.svc}"
33+
_GHCR_MIRROR_DECISION_FILE="${_GHCR_MIRROR_DECISION_FILE:-/tmp/e2e-ghcr-mirror-endpoint}"
34+
GHCR_MIRROR_MANIFEST="${GHCR_MIRROR_MANIFEST:-hack/e2e-ghcr-mirror.yaml}"
35+
36+
# ghcr_registry_mirrors_block: pure string builder. Given a mirror endpoint URL
37+
# ($1, may be empty), print the `registryMirrors:` sub-block for a tenant Kubernetes
38+
# CR `spec.talos` (4-space indented, i.e. nested under ` talos:`), or nothing when
39+
# the endpoint is empty. Kept separate so it can be unit-tested: an indentation or
40+
# quoting slip here would emit an invalid CR and every kubernetes-* test would fail.
41+
ghcr_registry_mirrors_block() {
42+
local endpoint="$1"
43+
[ -n "$endpoint" ] || return 0
44+
printf ' registryMirrors:\n ghcr.io:\n endpoints:\n - %s\n' "$endpoint"
45+
}
46+
47+
# _apply_ghcr_mirror_egress_policy: install the CiliumClusterwideNetworkPolicy that
48+
# lets tenant worker VM (virt-launcher) Pods egress to the mirror. It ships in the
49+
# manifest but is applied here, after Cilium's CRDs exist. Idempotent, best-effort.
50+
#
51+
# On failure it PRINTS the reason to stdout for the caller to surface. The causes
52+
# are not all alike -- a missing yq, an apiVersion the cluster does not serve, an
53+
# RBAC denial are permanent, while an apiserver blip is not -- and the exit code
54+
# cannot tell them apart. Discarding the message would leave the caller announcing
55+
# a cause it does not know, in the subsystem this exists to make legible.
56+
_apply_ghcr_mirror_egress_policy() {
57+
if ! command -v yq >/dev/null 2>&1; then
58+
echo "yq is not installed, so the policy could not be extracted from ${GHCR_MIRROR_MANIFEST}"
59+
return 1
60+
fi
61+
# Braces, not a bare `2>&1 >/dev/null`: the goal is to keep stderr (the reason)
62+
# on stdout for the caller and drop kubectl's success chatter, and the braced
63+
# form says that unambiguously.
64+
{ yq 'select(.kind == "CiliumClusterwideNetworkPolicy")' "$GHCR_MIRROR_MANIFEST" \
65+
| kubectl apply -f - >/dev/null; } 2>&1
66+
}
67+
68+
# resolve_ghcr_mirror_endpoint: print the mirror endpoint URL to use, or an empty
69+
# string to signal "pull ghcr.io directly". Resolved once and cached in a /tmp file
70+
# that persists across bats files (same sandbox container), so only the first tenant
71+
# test pays the readiness wait.
72+
resolve_ghcr_mirror_endpoint() {
73+
if [ -f "$_GHCR_MIRROR_DECISION_FILE" ]; then
74+
cat "$_GHCR_MIRROR_DECISION_FILE"
75+
return 0
76+
fi
77+
local endpoint="" cache=1 out rc avail ready apply_err
78+
# One `get deploy` serves both the existence check and the not-deployed-vs-error
79+
# split: a genuine NotFound is a stable decision worth caching, but any other
80+
# failure (transient apiserver blip) must NOT be cached, or one blip would
81+
# disable the mirror for every later test in the shared sandbox.
82+
#
83+
# Capture inside `if` (not a bare `out=$(...); rc=$?`): the chainsaw suites run
84+
# this script under `sh -c` with `set -eu`, and dash inherits errexit into a
85+
# command substitution, so a non-zero kubectl in a bare assignment would kill
86+
# the script before either fallback branch runs. bash does not inherit it,
87+
# which is why a bare assignment survives locally but not in CI.
88+
if out=$(kubectl -n kube-system get deploy ghcr-mirror 2>&1); then
89+
rc=0
90+
else
91+
rc=$?
92+
fi
93+
if [ "$rc" -ne 0 ]; then
94+
case "$out" in
95+
# The API's own NotFound only. A bare "not found" also matches a shell
96+
# reporting a missing kubectl and a bad --context, neither of which says
97+
# anything about whether the mirror was applied -- and both would then be
98+
# cached as the stable answer for every later suite in the sandbox.
99+
*NotFound*)
100+
echo "ghcr-mirror not deployed -- tenant workers pull ghcr.io directly" >&2 ;;
101+
*)
102+
echo "WARNING: could not query ghcr-mirror Deployment (transient); not caching, will retry" >&2
103+
cache=0 ;;
104+
esac
105+
else
106+
# Readiness is decided once, then acted on once. The watch is the cheap way to
107+
# learn it, but `rollout status` also exits non-zero on a broken or aborted
108+
# watch, and its output goes to /dev/null -- so when it fails, ask the API what
109+
# the Deployment's own condition says instead of reading the watch's exit code
110+
# as the answer.
111+
#
112+
# 2m, not longer: the wait is serial with the rest of the 50m e2e step, whose
113+
# budget inventory has to stay above the sum of its legs, and spending more here
114+
# eats the margin the failure-path diagnostics need in order to be collected at
115+
# all. It is already generous -- the Deployment pulls one small digest-pinned
116+
# image and was applied at install time, tens of minutes earlier.
117+
if kubectl -n kube-system rollout status deploy/ghcr-mirror --timeout=2m >/dev/null 2>&1; then
118+
ready=yes
119+
else
120+
if avail=$(kubectl -n kube-system get deploy ghcr-mirror \
121+
-o jsonpath='{.status.conditions[?(@.type=="Available")].status}' 2>/dev/null); then
122+
:
123+
else
124+
avail=""
125+
fi
126+
case "$avail" in
127+
True) ready=yes ;;
128+
False) ready=no ;;
129+
*) ready=unknown ;;
130+
esac
131+
fi
132+
case "$ready" in
133+
yes)
134+
# Only commit the endpoint once the tenant egress allow is actually in
135+
# place. The mirror is a kube-system ClusterIP that the tenant default-deny
136+
# egress drops without this allow; pointing workers at it when the policy
137+
# did not apply just adds dial-timeout latency before Talos falls back.
138+
# Leaving the endpoint empty is what keeps that cost off the run -- it
139+
# removes the one failure mode this function can see, not every one of them
140+
# (see the header on what the gate does and does not establish).
141+
if apply_err=$(_apply_ghcr_mirror_egress_policy); then
142+
endpoint="$GHCR_MIRROR_SVC_URL"
143+
echo "ghcr-mirror ready and egress allow applied -- tenant workers mirror ghcr.io via ${endpoint}" >&2
144+
else
145+
# Not cached, for the same reason as the failed query above: a failed
146+
# apply says nothing about whether the next attempt would succeed. The
147+
# reason is printed rather than characterised -- some causes here are
148+
# permanent and retrying will not help, and only the message says which.
149+
echo "WARNING: ghcr-mirror ready but its egress allow failed to apply; not caching, will retry. Reason: ${apply_err:-none reported}" >&2
150+
cache=0
151+
fi
152+
;;
153+
no)
154+
# Cached, and this is a trade rather than an invariant: a Deployment can
155+
# still go Available later on its own -- a kubelet image-pull retry or a
156+
# Pending Pod finally scheduling would both do it, with nothing re-applying
157+
# the manifest. What caching buys is not re-paying the wait in every later
158+
# suite, serial with the rest of the step; what it costs is missing a mirror
159+
# that recovers mid-run. For a best-effort accelerator that trade is worth
160+
# taking, and re-reading the condition cheaply on a cached miss would take
161+
# it back if it ever bites.
162+
echo "WARNING: ghcr-mirror not Available -- tenant workers pull ghcr.io directly" >&2
163+
;;
164+
*)
165+
echo "WARNING: could not confirm ghcr-mirror readiness (transient); not caching, will retry" >&2
166+
cache=0
167+
;;
168+
esac
169+
fi
170+
# Cache atomically (temp + mv) so a concurrent reader in the shared sandbox never
171+
# sees a half-written decision file.
172+
if [ "$cache" -eq 1 ]; then
173+
printf '%s' "$endpoint" > "${_GHCR_MIRROR_DECISION_FILE}.tmp" \
174+
&& mv "${_GHCR_MIRROR_DECISION_FILE}.tmp" "$_GHCR_MIRROR_DECISION_FILE"
175+
fi
176+
printf '%s' "$endpoint"
177+
}
178+
179+
# _ghcr_mirror_bounded_read <label> <command...>: run one diagnostic read under a
180+
# wall-clock bound and name the outcome when it does not finish, so a read cut off
181+
# mid-flight is not misread as the cluster being silent. Always returns 0: the only
182+
# caller sits before the node-join block's `exit 1`, and a non-zero here would
183+
# replace that exit under `set -e`.
184+
#
185+
# Bounded by the caller's own COZY_DIAG_READ_TIMEOUT/COZY_DIAG_READ_GRACE, the same
186+
# knobs cozy_diag_read uses, so lowering the node-join block's budget lowers these
187+
# too -- a private knob here would sit at its own default while the caller's budget
188+
# dropped. The block validates those values before this runs; standalone use
189+
# (hack/ghcr-mirror_test.bats exercises this without run-kubernetes.sh) falls back
190+
# to the block's own 20s/5s defaults. The bound lives here rather than in a call to
191+
# cozy_diag_read because that function is defined in run-kubernetes.sh and this file
192+
# is sourced on its own, the same reason talos-image-cache.sh carries its own copy.
193+
_ghcr_mirror_bounded_read() {
194+
local label="$1"
195+
shift
196+
local to="${COZY_DIAG_READ_TIMEOUT:-20}" grace="${COZY_DIAG_READ_GRACE:-5}" rc=0
197+
if command -v timeout >/dev/null 2>&1; then
198+
timeout -k "$grace" "$to" "$@" || rc=$?
199+
else
200+
"$@" || rc=$?
201+
fi
202+
case "$rc" in
203+
0) ;;
204+
124|137)
205+
echo "=== ${label}: read did not finish within ${to}s and was cut off; what it had not printed is absent from this log, not from the cluster ===" >&2 ;;
206+
*)
207+
echo "=== ${label}: read failed (exit ${rc}); what it would have shown was not observed ===" >&2 ;;
208+
esac
209+
return 0
210+
}
211+
212+
# ghcr_mirror_diagnose: on-demand diagnostic for the node-join failure path.
213+
#
214+
# Pointing workers at the mirror and their nodes joining does not establish that
215+
# the mirror served anything -- Talos falls back to public ghcr.io on its own, so
216+
# a pass is consistent with the mirror never being touched, and a failure is
217+
# consistent with it being unreachable. registry:2's access log is what tells the
218+
# two apart: a `GET /v2/siderolabs/kubelet/...` line means a worker really came
219+
# through here. Without it #3513 gains another run that has to be reasoned about
220+
# from the outside, which is how it survived four attempts.
221+
#
222+
# All output to stderr; never fails the caller. Every read is wall-clock bounded
223+
# by the caller's COZY_DIAG_READ_TIMEOUT/COZY_DIAG_READ_GRACE.
224+
ghcr_mirror_diagnose() {
225+
local hits log gate_rc=0
226+
local to="${COZY_DIAG_READ_TIMEOUT:-20}" grace="${COZY_DIAG_READ_GRACE:-5}"
227+
# The existence gate is bounded too: a wedged apiserver hangs `get deploy` just
228+
# as it hangs the reads below.
229+
if command -v timeout >/dev/null 2>&1; then
230+
timeout -k "$grace" "$to" \
231+
kubectl -n kube-system get deploy ghcr-mirror >/dev/null 2>&1 || gate_rc=$?
232+
else
233+
kubectl -n kube-system get deploy ghcr-mirror >/dev/null 2>&1 || gate_rc=$?
234+
fi
235+
if [ "$gate_rc" -ne 0 ]; then
236+
echo "ghcr-mirror not deployed -- tenant workers pulled ghcr.io directly; nothing to diagnose" >&2
237+
return 0
238+
fi
239+
{
240+
echo "--- ghcr-mirror deploy/pod/svc/endpointslice ---"
241+
_ghcr_mirror_bounded_read 'ghcr-mirror deploy/pod/svc/endpointslice' \
242+
kubectl -n kube-system get deploy,pod,svc,endpointslice \
243+
-l app.kubernetes.io/name=ghcr-mirror -o wide
244+
echo "--- worker egress allow policy ---"
245+
_ghcr_mirror_bounded_read 'worker egress allow policy' \
246+
kubectl get ciliumclusterwidenetworkpolicy e2e-ghcr-mirror-worker-egress -o wide
247+
# The answer to "did a worker actually pull through the mirror".
248+
#
249+
# Filter the whole log rather than tailing it. The readinessProbe requests /v2/
250+
# every 5s and registry:2 writes an access line per request, so ~12 lines a
251+
# minute pile up after the worker's pull; by the time this runs -- past an 18m
252+
# node-join budget -- that is 200+ probe lines on top of the one request worth
253+
# finding, and any fixed tail window reports "no kubelet pull" for a mirror that
254+
# served it. --tail=-1 is explicit because kubectl defaults a label-selected
255+
# read to the last 10 lines, not to the whole log. It is the wall clock, not the
256+
# line count, that bounds this read: --tail=-1 stays, timeout caps the time.
257+
echo "--- did a worker pull the kubelet image through the mirror? ---"
258+
# Read and filter as separate steps. Piping them would make an unreadable log
259+
# and a log with no kubelet request produce the same 0, and the second is a
260+
# finding about the workers while the first is a statement about nothing.
261+
local log_rc=0
262+
if command -v timeout >/dev/null 2>&1; then
263+
log=$(timeout -k "$grace" "$to" \
264+
kubectl -n kube-system logs -l app.kubernetes.io/name=ghcr-mirror \
265+
--tail=-1 2>/dev/null) || log_rc=$?
266+
else
267+
log=$(kubectl -n kube-system logs -l app.kubernetes.io/name=ghcr-mirror \
268+
--tail=-1 2>/dev/null) || log_rc=$?
269+
fi
270+
if [ "$log_rc" -eq 0 ]; then
271+
hits=$(printf '%s\n' "$log" | grep -cF '/v2/siderolabs/kubelet') || hits=0
272+
if [ "${hits:-0}" -gt 0 ]; then
273+
echo "the mirror served ${hits} kubelet-image request(s), so workers did reach it"
274+
else
275+
echo "no kubelet-image request reached the mirror: workers either fell back to public ghcr.io or never got as far as the pull"
276+
fi
277+
else
278+
echo "could not read the mirror's log; this says nothing either way about whether workers used it"
279+
fi
280+
echo "--- registry access log (last 50 lines, for context) ---"
281+
_ghcr_mirror_bounded_read 'registry access log (last 50 lines)' \
282+
kubectl -n kube-system logs -l app.kubernetes.io/name=ghcr-mirror \
283+
--tail=50 --prefix
284+
} >&2
285+
return 0
286+
}

0 commit comments

Comments
 (0)