Skip to content

Commit 1c779cf

Browse files
committed
test(e2e): allow tenant CDI importers to reach the image cache
Tenant namespaces run under a default-deny Cilium egress installed by the tenant chart, which lets a worker's CDI importer reach the outside world and kube-dns but nothing in kube-system. The in-sandbox image mirror is a kube-system Service, so without an explicit allow the importer's connect to its ClusterIP is dropped and the worker disk import never starts. Ship a tightly-scoped CiliumClusterwideNetworkPolicy in the mirror manifest that lets importer Pods in tenant-test egress to the mirror Pod in kube-system. Cilium unions allow rules, so this only adds a hole; scoping it to tenant-test (already under default-deny egress) means it never flips another namespace's posture, and the mirror Pod needs no ingress rule because it is selected by no ingress policy. The policy is kept out of the pre-install apply because that runs before Cilium's CRDs exist; the resolver in hack/e2e-apps/talos-image-cache.sh applies it at point-of-use once Cilium is up. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 012c512 commit 1c779cf

3 files changed

Lines changed: 205 additions & 7 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -42,19 +42,25 @@
4242
# (see cozystack/cozystack#3231). This Deployment fetches the image ONCE with a
4343
# hard curl retry loop and serves it locally; tenant CRs then point
4444
# spec.talos.imageFactoryURL at its Service. Deployed here (before the long
45-
# install) so the seed download overlaps the install churn — readiness is gated
46-
# at point-of-use in hack/e2e-apps/run-kubernetes.sh, which falls back to the
47-
# public factory if the mirror never becomes Available, so this can only help.
48-
# Best-effort: never fail the suite on the band-aid. Remove once tenant workers
49-
# no longer bulk-pull the OS image from the public internet in CI.
45+
# install) so the seed download overlaps the install churn — reachability is
46+
# gated at point-of-use in hack/e2e-apps/run-kubernetes.sh, which falls back to
47+
# the public factory unless a tenant-scoped probe confirms a worker can actually
48+
# reach the ClusterIP, so this can only help. Best-effort: never fail the suite
49+
# on the band-aid. Remove once tenant workers no longer bulk-pull the OS image
50+
# from the public internet in CI.
5051
local sid ver
5152
sid=$(yq '.talos.schematicID' packages/apps/kubernetes/values.yaml 2>/dev/null)
5253
ver=$(yq '.talos.version' packages/apps/kubernetes/values.yaml 2>/dev/null)
5354
if [ -z "$sid" ] || [ "$sid" = "null" ] || [ -z "$ver" ] || [ "$ver" = "null" ]; then
5455
echo "WARNING: could not read talos.schematicID/version from values.yaml — skipping mirror (fallback to public factory)"
5556
return 0
5657
fi
58+
# The CiliumClusterwideNetworkPolicy in the manifest is intentionally NOT applied
59+
# here: Cilium's CRDs do not exist until Cozystack is installed (below), so
60+
# applying it now would error. It is applied at point-of-use by
61+
# hack/e2e-apps/talos-image-cache.sh once Cilium is up.
5762
sed -e "s|__SCHEMATIC_ID__|${sid}|g" -e "s|__TALOS_VERSION__|${ver}|g" hack/e2e-talos-image-cache.yaml \
63+
| yq 'select(.kind != "CiliumClusterwideNetworkPolicy")' \
5864
| kubectl apply -f - || echo "WARNING: failed to apply talos-image-cache (fallback to public factory)"
5965
if kubectl -n kube-system get deploy talos-image-cache >/dev/null 2>&1; then
6066
echo "talos-image-cache Deployment created (seeding ${sid}/${ver} in background)"

‎hack/e2e-talos-image-cache.yaml‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,15 @@
1616
# per-worker imports), then serves it locally over plain HTTP with range support.
1717
# Tenant `Kubernetes` CRs point `spec.talos.imageFactoryURL` at this Service
1818
# (see hack/e2e-apps/run-kubernetes.sh) so every worker imports from the warm
19-
# local copy instead of the public internet. If this Deployment never becomes
20-
# Available the e2e harness falls back to the public factory (status quo), so the
19+
# local copy instead of the public internet. Tenant namespaces run under a
20+
# default-deny Cilium egress installed by the tenant chart, whose allow rules cover
21+
# same-namespace endpoints, the tenant tree and the `world` entity (which is why the
22+
# public factory works) but NOT an arbitrary kube-system Service, so a worker's CDI
23+
# importer would resolve this Service yet have its TCP connect to the ClusterIP
24+
# silently dropped. The CiliumClusterwideNetworkPolicy at the end of this file punches a
25+
# single tightly-scoped hole for that path. The e2e harness only points tenants
26+
# here after a tenant-scoped probe confirms the importer path actually reaches the
27+
# ClusterIP with a 206; otherwise it falls back to the public factory, so the
2128
# mirror can only help, never make CI worse.
2229
#
2330
# The __SCHEMATIC_ID__ / __TALOS_VERSION__ placeholders are substituted at apply
@@ -221,3 +228,51 @@ data:
221228
with http.server.ThreadingHTTPServer(("0.0.0.0", PORT), RangeHandler) as httpd:
222229
print("serving %s on :%d" % (ROOT, PORT), flush=True)
223230
httpd.serve_forever()
231+
---
232+
# CI-only: allow the tenant CDI importer Pods to reach the mirror.
233+
#
234+
# Tenant namespaces run under a default-deny Cilium egress installed by the tenant
235+
# chart (allow-internal-communication, allow-external-communication and the
236+
# `<tenant>-egress` CiliumClusterwideNetworkPolicy). Their union allows egress to
237+
# same-namespace endpoints, the tenant tree and the `world` entity (so the public
238+
# factory works), but nothing permits egress to an arbitrary kube-system Service —
239+
# so a worker's CDI importer resolves talos-image-cache.kube-system.svc yet its TCP
240+
# connect to the Service ClusterIP is silently dropped (dial ... i/o timeout), the
241+
# disk import never starts, and the tenant node never joins. Cilium unions allow
242+
# rules across policies, so this adds one tightly-scoped allow: importer Pods in
243+
# tenant-test may egress to the mirror Pod in kube-system.
244+
#
245+
# Scoped to tenant-test (which the tenant chart already puts under default-deny
246+
# egress) so it only ever ADDS an allow — it never flips another namespace from
247+
# default-allow to default-deny. The mirror Pod is selected by no ingress policy
248+
# (default-allow ingress), so no ingress rule is needed; replies return on the
249+
# established conntrack entry.
250+
#
251+
# Applied at point-of-use by hack/e2e-apps/talos-image-cache.sh, not with the rest
252+
# of this manifest in hack/e2e-install-cozystack.bats: that runs before Cozystack
253+
# (and thus before Cilium's CRDs) is installed, so this kind does not yet exist
254+
# when the manifest is first applied.
255+
apiVersion: cilium.io/v2
256+
kind: CiliumClusterwideNetworkPolicy
257+
metadata:
258+
name: e2e-talos-image-cache-importer-egress
259+
labels:
260+
app.kubernetes.io/name: talos-image-cache
261+
spec:
262+
endpointSelector:
263+
matchLabels:
264+
"k8s:cdi.kubevirt.io": importer
265+
"k8s:io.kubernetes.pod.namespace": tenant-test
266+
# No toPorts: the allow is already scoped to a single kube-system pod identity
267+
# (CI-only, torn down with the sandbox), so an L4 port restriction adds no
268+
# meaningful blast-radius reduction — only risk. With kube-proxy-less Cilium the
269+
# importer's connect to the Service ClusterIP is socket-LB translated to the
270+
# backend pod port BEFORE egress policy is evaluated, so a port literal here
271+
# would have to track the container port (8080), not the Service port (80); a
272+
# wrong guess silently drops the traffic and reintroduces the exact public-factory
273+
# fallback this policy exists to prevent.
274+
egress:
275+
- toEndpoints:
276+
- matchLabels:
277+
"k8s:io.kubernetes.pod.namespace": kube-system
278+
"k8s:app.kubernetes.io/name": talos-image-cache

‎hack/talos-image-cache_test.bats‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for the in-sandbox Talos image cache manifest split and the
4+
# importer-reachability contract it relies on.
5+
#
6+
# hack/e2e-talos-image-cache.yaml bundles four documents but they are applied in
7+
# two phases: hack/e2e-install-cozystack.bats applies everything EXCEPT the
8+
# CiliumClusterwideNetworkPolicy (its CRD does not exist before Cozystack is
9+
# installed), and hack/e2e-apps/talos-image-cache.sh applies ONLY that policy
10+
# later, once Cilium is up. If a future document is added to the manifest and
11+
# silently dropped from the pre-Cilium apply, or the Cilium document leaks into
12+
# it (and errors on the missing CRD), the mirror breaks and e2e falls back to
13+
# the flaky public factory. These tests pin that split.
14+
#
15+
# They also pin the load-bearing reachability invariant: the throwaway probe pod
16+
# is a faithful proxy for a real CDI importer only if it carries the exact label
17+
# and namespace the egress policy selects. If either side drifts, the probe can
18+
# pass while real importers stay blocked (a false positive that makes CI worse).
19+
#
20+
# Finally they cover the two pure fragments of the decision itself, sourced from
21+
# hack/e2e-apps/talos-image-cache.sh (sourcing it has no cluster side effects — it
22+
# only sets parameter defaults and defines functions): the strict 206 gate that
23+
# decides whether tenants are pointed at the mirror at all, and the --overrides JSON
24+
# builder whose silent breakage would drop every probe to the public factory. The
25+
# kubectl orchestration around them needs a live cluster and is covered by the e2e
26+
# run, matching how the other hack/ helpers are tested.
27+
#
28+
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
29+
# own line; there is no bats `run` or `$status`, and setup()/teardown() are not
30+
# honored. Each test runs under `set -eu -x`; assertions are direct shell tests
31+
# that exit non-zero on failure. mikefarah yq prints `---` between matched
32+
# documents, so document streams are compared with those separators stripped.
33+
#
34+
# Run with: hack/cozytest.sh hack/talos-image-cache_test.bats
35+
# -----------------------------------------------------------------------------
36+
37+
@test "manifest documents partition into pre-Cilium apply plus the Cilium policy" {
38+
manifest=hack/e2e-talos-image-cache.yaml
39+
total=$(yq '.kind' "$manifest" | grep -vc '^---$')
40+
excluded=$(yq 'select(.kind != "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -vc '^---$')
41+
selected=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -vc '^---$')
42+
[ "$total" -eq 4 ]
43+
[ "$excluded" -eq 3 ]
44+
[ "$selected" -eq 1 ]
45+
[ $((excluded + selected)) -eq "$total" ]
46+
}
47+
48+
@test "pre-Cilium apply keeps Service, Deployment, ConfigMap and drops the Cilium policy" {
49+
manifest=hack/e2e-talos-image-cache.yaml
50+
kinds=$(yq 'select(.kind != "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -v '^---$')
51+
for want in Service Deployment ConfigMap; do
52+
printf '%s\n' "$kinds" | grep -qx "$want" || { echo "pre-Cilium apply is missing $want" >&2; exit 1; }
53+
done
54+
if printf '%s\n' "$kinds" | grep -qx CiliumClusterwideNetworkPolicy; then
55+
echo "CiliumClusterwideNetworkPolicy leaked into the pre-Cilium apply" >&2
56+
exit 1
57+
fi
58+
}
59+
60+
@test "point-of-use apply selects exactly the Cilium policy" {
61+
manifest=hack/e2e-talos-image-cache.yaml
62+
kinds=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .kind' "$manifest" | grep -v '^---$')
63+
[ "$kinds" = "CiliumClusterwideNetworkPolicy" ]
64+
}
65+
66+
@test "egress policy selects the importer label and namespace the probe pod uses" {
67+
manifest=hack/e2e-talos-image-cache.yaml
68+
helper=hack/e2e-apps/talos-image-cache.sh
69+
label=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.endpointSelector.matchLabels["k8s:cdi.kubevirt.io"]' "$manifest")
70+
ns=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.endpointSelector.matchLabels["k8s:io.kubernetes.pod.namespace"]' "$manifest")
71+
[ "$label" = "importer" ]
72+
[ "$ns" = "tenant-test" ]
73+
# The probe pod must carry the same label the policy selects, else it faces a
74+
# different egress than a real importer and the guarantee breaks.
75+
grep -q "cdi.kubevirt.io=importer" "$helper" || { echo "probe pod label drifted from the egress selector" >&2; exit 1; }
76+
grep -q "TALOS_IMAGE_CACHE_PROBE_NS:-tenant-test" "$helper" || { echo "probe namespace drifted from the egress selector" >&2; exit 1; }
77+
}
78+
79+
@test "egress rule targets the mirror pod's own identity" {
80+
manifest=hack/e2e-talos-image-cache.yaml
81+
target=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.egress[0].toEndpoints[0].matchLabels["k8s:app.kubernetes.io/name"]' "$manifest")
82+
target_ns=$(yq 'select(.kind == "CiliumClusterwideNetworkPolicy") | .spec.egress[0].toEndpoints[0].matchLabels["k8s:io.kubernetes.pod.namespace"]' "$manifest")
83+
mirror=$(yq 'select(.kind == "Deployment") | .spec.template.metadata.labels["app.kubernetes.io/name"]' "$manifest")
84+
mirror_ns=$(yq 'select(.kind == "Deployment") | .metadata.namespace' "$manifest")
85+
[ "$target" = "talos-image-cache" ]
86+
[ "$target_ns" = "kube-system" ]
87+
# The allow's destination must equal the mirror Deployment's own pod label and
88+
# namespace, otherwise the hole is punched toward nothing: moving the mirror
89+
# would leave these tests green while importers could no longer reach it.
90+
[ "$target" = "$mirror" ]
91+
[ "$target_ns" = "$mirror_ns" ]
92+
}
93+
94+
@test "strict 206 gate selects the mirror on a byte-range success" {
95+
. hack/e2e-apps/talos-image-cache.sh
96+
if ! _talos_image_cache_probe_succeeded "code=206"; then
97+
echo "expected a 206 probe result to select the mirror" >&2
98+
exit 1
99+
fi
100+
}
101+
102+
@test "strict 206 gate finds the marker inside surrounding probe output" {
103+
. hack/e2e-apps/talos-image-cache.sh
104+
out=$(printf 'pod/talos-image-cache-probe created\ncode=206\npod "talos-image-cache-probe" deleted\n')
105+
if ! _talos_image_cache_probe_succeeded "$out"; then
106+
echo "expected 206 to be found in multi-line attach output" >&2
107+
exit 1
108+
fi
109+
}
110+
111+
@test "strict 206 gate falls back on range-less, unreachable, empty and unrelated output" {
112+
. hack/e2e-apps/talos-image-cache.sh
113+
# A plain 200 means the mirror answered without range support; 000 means curl
114+
# never connected. Both fall back to the public factory, as does output that
115+
# carried no result marker at all (an attach stream lost with no logs to recover).
116+
for bad in "code=200" "code=000" "code=404" "" "error: pods not found"; do
117+
if _talos_image_cache_probe_succeeded "$bad"; then
118+
echo "expected non-206 output to fall back to the public factory: [$bad]" >&2
119+
exit 1
120+
fi
121+
done
122+
}
123+
124+
@test "probe overrides builder emits valid JSON with the image, command and restricted PSA" {
125+
. hack/e2e-apps/talos-image-cache.sh
126+
img=example.invalid/img:tag
127+
cmd='curl -s -o /dev/null -w code=%{http_code} -r 0-0 http://talos-image-cache.kube-system.svc/x.raw.xz'
128+
json=$(_talos_image_cache_probe_overrides "$img" "$cmd")
129+
# Invalid JSON would make every probe fail and silently drop CI back to the
130+
# public factory, so parse it rather than pattern-match it.
131+
printf '%s' "$json" | yq -p=json -o=json '.' >/dev/null
132+
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].name')" = "c" ]
133+
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].image')" = "$img" ]
134+
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].command[2]')" = "$cmd" ]
135+
[ "$(printf '%s' "$json" | yq -p=json '.spec.securityContext.runAsNonRoot')" = "true" ]
136+
[ "$(printf '%s' "$json" | yq -p=json '.spec.containers[0].securityContext.allowPrivilegeEscalation')" = "false" ]
137+
}

0 commit comments

Comments
 (0)