Skip to content

Commit 47e7d22

Browse files
test(e2e): serve Talos worker OS image from an in-sandbox cache
The kubernetes-* e2e tests fail intermittently and run 18-22 min each because every tenant worker VM imports its Talos OS disk directly from the public factory.talos.dev: that endpoint has no byte-range support (forcing CDI into a full copy-to-scratch) and its egress stream-resets / stalls mid-transfer from the CI runner, so a worker DataVolume import hangs past the chart's 12-minute node-join deadline and the test fails with zero tenant nodes joined (observed on run 28924455578, and on the unrelated branch in run 28929095889 - i.e. not specific to any one PR). Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml) during install: an initContainer fetches the image once with an aggressive curl retry loop, then busybox httpd serves it locally with range support. Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the mirror being Available and falls back to the public factory otherwise, so the mirror can only help, never make CI worse. Deployed early so the seed overlaps the Cozystack install; readiness is resolved once and cached across bats files. Uses the overridable chart value added in the previous commit. Refs #3231 Co-Authored-By: Claude Opus 4.8 <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 3eb182d commit 47e7d22

6 files changed

Lines changed: 227 additions & 0 deletions

‎hack/e2e-apps/kubernetes-oidc-customconfig.bats‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@
2121
# inline at the top of each @test.
2222
TEST_NAME="oidc-byo"
2323

24+
# Point worker DataVolume imports at the in-sandbox Talos image cache when it is
25+
# up (falls back to the public factory otherwise) — see run-kubernetes.sh.
26+
. hack/e2e-apps/talos-image-cache.sh
27+
2428
cleanup_kr() {
2529
kubectl -n tenant-test delete kubernetes.apps.cozystack.io "${TEST_NAME}" \
2630
--ignore-not-found --wait=false 2>/dev/null || true
@@ -32,13 +36,16 @@ cozy_cleanup() { cleanup_kr; }
3236

3337
@test "Kubernetes CR accepts spec.oidc.mode=CustomConfig with inline config" {
3438
cleanup_kr
39+
local talos_block
40+
talos_block=$(talos_image_factory_spec_block)
3541
kubectl apply -f - <<EOF
3642
apiVersion: apps.cozystack.io/v1alpha1
3743
kind: Kubernetes
3844
metadata:
3945
name: ${TEST_NAME}
4046
namespace: tenant-test
4147
spec:
48+
${talos_block}
4249
addons:
4350
ingressNginx:
4451
enabled: false

‎hack/e2e-apps/kubernetes-oidc-system.bats‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@
2929
# inline at the top of each @test.
3030
TEST_NAME="oidc-system"
3131

32+
# Point worker DataVolume imports at the in-sandbox Talos image cache when it is
33+
# up (falls back to the public factory otherwise) — see run-kubernetes.sh.
34+
. hack/e2e-apps/talos-image-cache.sh
35+
3236
cleanup_kr() {
3337
kubectl -n tenant-test delete kubernetes.apps.cozystack.io "${TEST_NAME}" \
3438
--ignore-not-found --wait=false 2>/dev/null || true
@@ -40,13 +44,16 @@ cozy_cleanup() { cleanup_kr; }
4044

4145
@test "Kubernetes CR accepts spec.oidc.mode=System with users[]" {
4246
cleanup_kr
47+
local talos_block
48+
talos_block=$(talos_image_factory_spec_block)
4349
kubectl apply -f - <<EOF
4450
apiVersion: apps.cozystack.io/v1alpha1
4551
kind: Kubernetes
4652
metadata:
4753
name: ${TEST_NAME}
4854
namespace: tenant-test
4955
spec:
56+
${talos_block}
5057
addons:
5158
ingressNginx:
5259
enabled: false

‎hack/e2e-apps/run-kubernetes.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
. hack/e2e-apps/remediation-guard.sh
2+
. hack/e2e-apps/talos-image-cache.sh
23

34
# kubectl_wait_retry: wraps `kubectl wait` with retries against transient
45
# management-cluster apiserver/etcd errors.
@@ -244,13 +245,20 @@ YAML
244245
)
245246
fi
246247

248+
# Point worker DataVolume imports at the in-sandbox Talos image cache when it
249+
# is up (falls back to the public factory otherwise). Emitted right under spec:
250+
# as `talos: { imageFactoryURL: ... }`, or an empty line when the default applies.
251+
local talos_block
252+
talos_block=$(talos_image_factory_spec_block)
253+
247254
kubectl apply -f - <<EOF
248255
apiVersion: apps.cozystack.io/v1alpha1
249256
kind: Kubernetes
250257
metadata:
251258
name: "${test_name}"
252259
namespace: tenant-test
253260
spec:
261+
${talos_block}
254262
addons:
255263
certManager:
256264
enabled: false

‎hack/e2e-apps/talos-image-cache.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Shared helper: choose the Talos Image Factory base URL for tenant Kubernetes
2+
# e2e CRs.
3+
#
4+
# Prefer the in-sandbox caching mirror (hack/e2e-talos-image-cache.yaml, deployed
5+
# by hack/e2e-install-cozystack.bats) when it is Available, otherwise fall back to
6+
# the chart default (the public factory.talos.dev) by emitting nothing. The
7+
# mirror rides out the public factory's flaky/range-less egress that otherwise
8+
# stalls worker DataVolume imports past the 12-minute node-join deadline — see
9+
# cozystack/cozystack#3231. Falling back to the default means the mirror can only
10+
# help, never make CI worse.
11+
12+
TALOS_IMAGE_CACHE_SVC_URL="${TALOS_IMAGE_CACHE_SVC_URL:-http://talos-image-cache.kube-system.svc}"
13+
_TALOS_IMAGE_FACTORY_DECISION_FILE="${_TALOS_IMAGE_FACTORY_DECISION_FILE:-/tmp/e2e-talos-image-factory-url}"
14+
15+
# resolve_talos_image_factory_url: print the imageFactoryURL to use, or an empty
16+
# string to signal "use the chart default". The decision is resolved once (with a
17+
# bounded wait for the mirror to finish seeding) and cached in a /tmp file that
18+
# persists across bats files (they all exec in the same sandbox container), so
19+
# only the first tenant Kubernetes test pays the readiness wait.
20+
resolve_talos_image_factory_url() {
21+
if [ -f "$_TALOS_IMAGE_FACTORY_DECISION_FILE" ]; then
22+
cat "$_TALOS_IMAGE_FACTORY_DECISION_FILE"
23+
return 0
24+
fi
25+
local url=""
26+
if kubectl -n kube-system get deploy talos-image-cache >/dev/null 2>&1; then
27+
# Available only after the seed initContainer has fully fetched the image, so
28+
# this is the signal that the mirror is warm and safe to point tenants at.
29+
if kubectl -n kube-system rollout status deploy/talos-image-cache --timeout=12m >/dev/null 2>&1; then
30+
url="$TALOS_IMAGE_CACHE_SVC_URL"
31+
echo "talos-image-cache mirror ready — tenant workers import from ${url}" >&2
32+
else
33+
echo "WARNING: talos-image-cache mirror not Available in time — tenant workers fall back to public factory.talos.dev" >&2
34+
fi
35+
else
36+
echo "talos-image-cache mirror not deployed — tenant workers use public factory.talos.dev" >&2
37+
fi
38+
printf '%s' "$url" > "$_TALOS_IMAGE_FACTORY_DECISION_FILE"
39+
printf '%s' "$url"
40+
}
41+
42+
# talos_image_factory_spec_block: emit a two-line YAML block
43+
# talos:
44+
# imageFactoryURL: <url>
45+
# indented for insertion directly under a tenant Kubernetes CR `spec:`, or
46+
# nothing when the chart default should apply. Ends with a trailing newline when
47+
# non-empty so it can prefix the next `spec` key in a heredoc.
48+
talos_image_factory_spec_block() {
49+
local url
50+
url=$(resolve_talos_image_factory_url)
51+
if [ -n "$url" ]; then
52+
printf ' talos:\n imageFactoryURL: %s\n' "$url"
53+
fi
54+
}

‎hack/e2e-install-cozystack.bats‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,36 @@
3333
fi
3434
}
3535

36+
@test "Deploy Talos image factory cache (best-effort e2e mirror)" {
37+
# Tenant Kubernetes worker VMs boot from a Talos raw disk image that CDI
38+
# streams over HTTP from the public Talos Image Factory. That endpoint has no
39+
# byte-range support and intermittently stream-resets/stalls mid-transfer from
40+
# the CI runner, so a worker DataVolume import can hang past the chart's
41+
# 12-minute node-join deadline and fail (or merely make) the kubernetes-* tests
42+
# (see cozystack/cozystack#3231). This Deployment fetches the image ONCE with a
43+
# hard curl retry loop and serves it locally; tenant CRs then point
44+
# spec.talos.imageFactoryURL at its Service. Deployed here (before the long
45+
# install) so the seed download overlaps the install churn — readiness is gated
46+
# at point-of-use in hack/e2e-apps/run-kubernetes.sh, which falls back to the
47+
# public factory if the mirror never becomes Available, so this can only help.
48+
# Best-effort: never fail the suite on the band-aid. Remove once tenant workers
49+
# no longer bulk-pull the OS image from the public internet in CI.
50+
local sid ver
51+
sid=$(yq '.talos.schematicID' packages/apps/kubernetes/values.yaml 2>/dev/null)
52+
ver=$(yq '.talos.version' packages/apps/kubernetes/values.yaml 2>/dev/null)
53+
if [ -z "$sid" ] || [ "$sid" = "null" ] || [ -z "$ver" ] || [ "$ver" = "null" ]; then
54+
echo "WARNING: could not read talos.schematicID/version from values.yaml — skipping mirror (fallback to public factory)"
55+
return 0
56+
fi
57+
sed -e "s|__SCHEMATIC_ID__|${sid}|g" -e "s|__TALOS_VERSION__|${ver}|g" hack/e2e-talos-image-cache.yaml \
58+
| kubectl apply -f - || echo "WARNING: failed to apply talos-image-cache (fallback to public factory)"
59+
if kubectl -n kube-system get deploy talos-image-cache >/dev/null 2>&1; then
60+
echo "talos-image-cache Deployment created (seeding ${sid}/${ver} in background)"
61+
else
62+
echo "WARNING: talos-image-cache Deployment NOT created — tenant workers will use public factory.talos.dev"
63+
fi
64+
}
65+
3666
@test "Required installer chart exists" {
3767
if [ ! -f packages/core/installer/Chart.yaml ]; then
3868
echo "Missing: packages/core/installer/Chart.yaml" >&2

‎hack/e2e-talos-image-cache.yaml‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# In-sandbox caching mirror for the Talos Image Factory worker OS disk image.
2+
#
3+
# Why this exists: tenant Kubernetes worker VMs boot from a Talos raw disk image
4+
# that CDI streams over HTTP from the PUBLIC Talos Image Factory
5+
# (https://factory.talos.dev/image/<schematicID>/<version>/openstack-amd64.raw.xz).
6+
# That endpoint does not support byte-range requests (so CDI falls back to a full
7+
# sequential copy-to-scratch) and its egress is intermittently flaky from the CI
8+
# runner — a mid-transfer HTTP/2 stream reset or a stalled download makes the
9+
# worker DataVolume import never finish, the worker VM never boots, no tenant
10+
# node joins within the chart's 12-minute node-join deadline, and the
11+
# `kubernetes-*` e2e tests fail (and, even when they pass, take 18-22 minutes
12+
# each dominated by these downloads). See cozystack/cozystack#3231.
13+
#
14+
# This Deployment fetches the image ONCE at e2e-install time with an aggressive
15+
# curl retry loop (decoupling the one flaky external fetch from the many
16+
# per-worker imports), then serves it locally over plain HTTP with range support.
17+
# Tenant `Kubernetes` CRs point `spec.talos.imageFactoryURL` at this Service
18+
# (see hack/e2e-apps/run-kubernetes.sh) so every worker imports from the warm
19+
# local copy instead of the public internet. If this Deployment never becomes
20+
# Available the e2e harness falls back to the public factory (status quo), so the
21+
# mirror can only help, never make CI worse.
22+
#
23+
# The __SCHEMATIC_ID__ / __TALOS_VERSION__ placeholders are substituted at apply
24+
# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes/values.yaml.
25+
#
26+
# CI band-aid, not a product artifact. The durable fix is the overridable
27+
# `talos.imageFactoryURL` chart value this manifest exercises; a self-hosted
28+
# Talos Image Factory or an operator's own mirror is the production path.
29+
apiVersion: v1
30+
kind: Service
31+
metadata:
32+
name: talos-image-cache
33+
namespace: kube-system
34+
labels:
35+
app.kubernetes.io/name: talos-image-cache
36+
spec:
37+
selector:
38+
app.kubernetes.io/name: talos-image-cache
39+
ports:
40+
- name: http
41+
port: 80
42+
targetPort: http
43+
---
44+
apiVersion: apps/v1
45+
kind: Deployment
46+
metadata:
47+
name: talos-image-cache
48+
namespace: kube-system
49+
labels:
50+
app.kubernetes.io/name: talos-image-cache
51+
spec:
52+
replicas: 1
53+
selector:
54+
matchLabels:
55+
app.kubernetes.io/name: talos-image-cache
56+
strategy:
57+
type: Recreate
58+
template:
59+
metadata:
60+
labels:
61+
app.kubernetes.io/name: talos-image-cache
62+
spec:
63+
# The seed fetch and static serving use ordinary pod networking — the same
64+
# egress path the CDI importer pods already use to reach factory.talos.dev.
65+
initContainers:
66+
# Fetch the compressed raw disk image ONCE, retrying hard against the
67+
# flaky upstream. The Deployment only becomes Available after this
68+
# succeeds, so "Available" is the e2e harness's signal that the mirror is
69+
# warm and safe to point tenants at.
70+
- name: seed
71+
# Reuse the in-tree, digest-pinned image already shipped by the
72+
# installer and pre-pulled for the cilium-leak-healer watchdog, so this
73+
# pod does not add a fresh external image pull of its own.
74+
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
75+
command: [/bin/sh, -ec]
76+
args:
77+
- |
78+
dest="/data/image/__SCHEMATIC_ID__/__TALOS_VERSION__/openstack-amd64.raw.xz"
79+
url="https://factory.talos.dev/image/__SCHEMATIC_ID__/__TALOS_VERSION__/openstack-amd64.raw.xz"
80+
mkdir -p "$(dirname "$dest")"
81+
echo "seeding $dest from $url"
82+
# --retry-all-errors + a long retry budget rides out the mid-transfer
83+
# stream resets / stalls that break the CDI importer's single attempt.
84+
curl --fail --location --show-error \
85+
--retry 30 --retry-all-errors --retry-delay 15 \
86+
--connect-timeout 30 \
87+
--output "$dest" \
88+
"$url"
89+
ls -la "$dest"
90+
volumeMounts:
91+
- name: data
92+
mountPath: /data
93+
containers:
94+
- name: serve
95+
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
96+
# busybox httpd serves static files with HTTP range support, so a warm
97+
# mirror also lets CDI use its faster range path instead of the
98+
# copy-to-scratch fallback the range-less public factory forces.
99+
command: [/bin/sh, -ec]
100+
args:
101+
- exec busybox httpd -f -v -p 8080 -h /data
102+
ports:
103+
- name: http
104+
containerPort: 8080
105+
readinessProbe:
106+
tcpSocket:
107+
port: http
108+
initialDelaySeconds: 2
109+
periodSeconds: 5
110+
resources:
111+
requests:
112+
cpu: 20m
113+
memory: 32Mi
114+
limits:
115+
memory: 128Mi
116+
volumeMounts:
117+
- name: data
118+
mountPath: /data
119+
volumes:
120+
- name: data
121+
emptyDir: {}

0 commit comments

Comments
 (0)