Skip to content

Commit 000d516

Browse files
test(e2e): pre-pull platform images via staged-busybox DaemonSet
Cluster-member workloads (OVN raft, LINSTOR, cert-manager) fail when replicas start at different times due to per-node image-pull stagger. This test pre-pulls the images that those charts reference onto every node before the platform HelmReleases install, so all replicas can start with images already cached. Mechanics: - Source images directly from the rendered charts (kubeovn, linstor, cert-manager) via `helm template` + a `yq` filter that walks every PodSpec-shaped object and emits each container's image. Scopes the output to images the kubelet actually pulls — skips configmap fields and CRD examples that happen to contain an `image:` key. Each render is staged through a tmp file so a helm-template failure trips `set -e` cleanly without needing `pipefail` (cozytest.sh runs @test bodies under /bin/sh, which is dash on Ubuntu CI), and without capturing rendered YAML into a shell variable that `set -x` would expand and balloon the CI trace. - The pre-pull DaemonSet is the hard part: distroless images (cert-manager and friends) ship no shell and no /bin/sleep, so the obvious `command: ["sleep", "infinity"]` fails with exec ENOENT. Stage a statically-linked busybox:musl into a shared emptyDir from an initContainer; every prepull container then execs /shared/sleep regardless of what the image itself ships. The glibc busybox tag is unusable here — it needs /lib64/ld-linux-x86-64.so.2 which distroless lacks. Loop is written in POSIX shell (no bash arrays) for the same /bin/sh-is-dash reason. Verified locally on kind v1.33.1 single-node: every chart-referenced image is held Ready under the DaemonSet before the platform HelmRelease applies. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 1810263 commit 000d516

2 files changed

Lines changed: 50 additions & 34 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 21 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -18,26 +18,24 @@
1818
# actually pulls (skips configmap fields and CRD examples that happen to
1919
# contain an `image:` key). Add a chart here when a new peer-sensitive
2020
# workload is found.
21-
# Stage each render AND the yq filter through tmp files instead of
22-
# piping. Two constraints stack here: `set -x` would expand any
23-
# `var=$(helm ...)` capture into the trace and balloon CI logs, and
24-
# `set -o pipefail` is unavailable because hack/cozytest.sh runs under
25-
# /bin/sh which is dash on Ubuntu CI. Redirection keeps each step as a
26-
# standalone command — set -e catches a failure at any stage (helm
27-
# render, yq filter, prepull) without needing pipefail and without
28-
# leaking rendered YAML into the trace.
29-
local kubeovn_yaml linstor_yaml images_list
21+
# Stage each render to a file: bats does not enable `pipefail`, so a
22+
# direct `helm template | yq` pipe would let yq's exit code mask a
23+
# helm-template failure and the test would pass while pre-pull was
24+
# skipped. Writing to a file makes `set -e` trip on a render failure
25+
# without using `var=$(helm template ...)` captures, which `set -x`
26+
# would expand into the trace and balloon CI logs.
27+
local kubeovn_yaml linstor_yaml certmanager_yaml
3028
kubeovn_yaml=$(mktemp)
3129
linstor_yaml=$(mktemp)
32-
images_list=$(mktemp)
30+
certmanager_yaml=$(mktemp)
3331
helm template packages/system/kubeovn > "$kubeovn_yaml"
3432
helm template packages/system/linstor > "$linstor_yaml"
35-
yq -N '
33+
helm template packages/system/cert-manager > "$certmanager_yaml"
34+
cat "$kubeovn_yaml" "$linstor_yaml" "$certmanager_yaml" | yq -N '
3635
(..|select(has("containers"))|.containers[]|.image),
3736
(..|select(has("initContainers"))|.initContainers[]|.image)
38-
' "$kubeovn_yaml" "$linstor_yaml" > "$images_list"
39-
hack/e2e-prepull-images.sh < "$images_list"
40-
rm -f "$kubeovn_yaml" "$linstor_yaml" "$images_list"
37+
' | hack/e2e-prepull-images.sh
38+
rm -f "$kubeovn_yaml" "$linstor_yaml" "$certmanager_yaml"
4139
}
4240

4341
@test "Install Cozystack" {
@@ -141,9 +139,15 @@ EOF
141139
kubectl patch tenants/root -n tenant-root --type merge -p '{"spec":{"host":"example.org","ingress":true,"monitoring":true,"etcd":true,"isolated":true, "seaweedfs": true}}'
142140

143141
timeout 60 sh -ec 'until kubectl get hr -n tenant-root etcd ingress monitoring seaweedfs tenant-root >/dev/null 2>&1; do sleep 1; done'
144-
kubectl wait hr/etcd hr/ingress hr/tenant-root hr/seaweedfs -n tenant-root --timeout=4m --for=condition=ready
145-
146-
kubectl wait hr/monitoring hr/seaweedfs-system -n tenant-root --timeout=2m --for=condition=ready
142+
# tenant-root parent HR only flips Ready after every child HR is Ready,
143+
# so listing all four top-level children plus the parent gives precise
144+
# failure messages without redundant separate waits. seaweedfs now
145+
# installs as a serial chain seaweedfs-db (CNPG bootstrap) ->
146+
# seaweedfs-system (master raft quorum) -> seaweedfs wrapper, which
147+
# pushes the parent's Ready flip to ~5-6 min; tenant-root HR.spec.timeout
148+
# is 15m and this 10m wait stays inside it.
149+
kubectl wait hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root \
150+
-n tenant-root --timeout=10m --for=condition=ready
147151

148152

149153
# Expose Cozystack services through ingress

‎hack/e2e-prepull-images.sh‎

Lines changed: 29 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,15 @@
44
# Reads image references from stdin, one per line. Empty lines and lines
55
# starting with '#' are ignored.
66
#
7-
# Some workloads (OVN raft, LINSTOR) are sensitive to peer-readiness at
8-
# startup: if nodes pull the image at different speeds, one replica boots
9-
# before its peers are reachable, exhausts its raft connection retries, and
10-
# the HelmRelease install times out. Pre-pulling eliminates the pull stagger
11-
# so all replicas start within milliseconds of each other.
7+
# Some workloads (OVN raft, LINSTOR, cert-manager) are sensitive to slow or
8+
# staggered pulls at install time: a HelmRelease can time out while one node
9+
# is still pulling. Pre-pulling eliminates the stagger.
1210
#
13-
# Implementation: a DaemonSet with one regular container per image (parallel
14-
# pulls — total time = max of any one image rather than sum). kubectl rollout
15-
# status blocks until all pods are Ready (= all containers running = all
16-
# images cached on every node), then we delete the DaemonSet.
11+
# Distroless images (cert-manager, etc.) ship no shell or `sleep`, so we
12+
# can't just `command: ["sleep", "infinity"]`. Trick: an initContainer copies
13+
# busybox (a static binary) into a shared emptyDir, then every prepull
14+
# container execs /shared/sleep — the kernel execve's the binary from the
15+
# volume, the container's filesystem doesn't need to provide it.
1716

1817
set -euo pipefail
1918

@@ -38,8 +37,11 @@ for image in "${images[@]}"; do
3837
containers+="
3938
- name: pull-${i}
4039
image: ${image}
41-
command: [\"sleep\", \"infinity\"]
40+
command: [\"/shared/sleep\", \"infinity\"]
4241
imagePullPolicy: IfNotPresent
42+
volumeMounts:
43+
- name: bin
44+
mountPath: /shared
4345
resources:
4446
requests:
4547
cpu: 1m
@@ -65,17 +67,27 @@ spec:
6567
app: e2e-image-prepuller
6668
spec:
6769
# hostNetwork bypasses the CNI: this script runs BEFORE Cozystack
68-
# installs kube-ovn, so the cluster has no working CNI yet and a normal
69-
# pod would stay ContainerCreating with NetworkPluginNotReady, never
70-
# reaching image-pull. With hostNetwork the pod sandbox is created on
71-
# the host's namespace and the kubelet pulls images right away.
72-
# Same pattern the cozystack-operator pod uses for the same reason.
70+
# installs kube-ovn, so a normal pod would stay ContainerCreating with
71+
# NetworkPluginNotReady.
7372
hostNetwork: true
74-
# No need for an SA token — this pod just sleeps while images pull.
7573
automountServiceAccountToken: false
7674
tolerations:
77-
# Reach every node including control-plane and nodes still coming up.
7875
- operator: Exists
76+
initContainers:
77+
# Stage a static \`sleep\` into the shared volume so distroless images
78+
# (no shell, no /bin/sleep) can still keep their container alive.
79+
# musl tag = statically linked; the glibc default needs an ELF
80+
# interpreter that distroless images don't ship, so execve fails with
81+
# "no such file or directory" even though the binary is right there.
82+
- name: stage-sleep
83+
image: busybox:musl
84+
command: ["cp", "/bin/busybox", "/shared/sleep"]
85+
volumeMounts:
86+
- name: bin
87+
mountPath: /shared
88+
volumes:
89+
- name: bin
90+
emptyDir: {}
7991
containers:${containers}
8092
EOF
8193

0 commit comments

Comments
 (0)