Skip to content

Commit 9df5950

Browse files
test(tests): pre-pull timing-sensitive platform images before install
Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server) fail when replicas start at materially different times due to image-pull stagger across nodes. Add a DaemonSet-based pre-pull step that runs before `helm install`, ensuring every node has the images cached so replicas start within milliseconds of each other. The script accepts image refs on stdin and creates one container per image so pulls run in parallel — total time = max(any single image) rather than sum. The bats test sources image refs directly from the rendered platform charts via `yq`, walking only PodSpec-shaped objects, so version bumps stay in sync without a hardcoded list. Signed-off-by: Myasnikov Daniil <[email protected]> Assisted-By: Claude <[email protected]>
1 parent 5786afe commit 9df5950

2 files changed

Lines changed: 109 additions & 0 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,30 @@
77
fi
88
}
99

10+
@test "Pre-pull platform images" {
11+
# Cluster-member workloads (OVN raft, LINSTOR) fail if replicas start at
12+
# different times due to image-pull stagger across nodes. Pre-pull these
13+
# images to every node so all replicas start with images already cached.
14+
#
15+
# Source images directly from the rendered charts so version bumps stay in
16+
# sync automatically. yq walks every PodSpec-shaped object and emits the
17+
# images of each container — this scopes the result to images the kubelet
18+
# actually pulls (skips configmap fields and CRD examples that happen to
19+
# contain an `image:` key). Add a chart here when a new peer-sensitive
20+
# workload is found.
21+
# Capture each render into a variable first: bats does not enable
22+
# `pipefail`, so a `helm template` failure inside a brace-grouped pipe
23+
# would be silently masked by yq's exit code, the script would receive
24+
# empty stdin, and the test would pass while pre-pull was skipped.
25+
# Assigning to a variable lets `set -e` trigger on a render failure.
26+
kubeovn_manifests=$(helm template packages/system/kubeovn)
27+
linstor_manifests=$(helm template packages/system/linstor)
28+
printf '%s\n%s\n' "$kubeovn_manifests" "$linstor_manifests" | yq -N '
29+
(..|select(has("containers"))|.containers[]|.image),
30+
(..|select(has("initContainers"))|.initContainers[]|.image)
31+
' | hack/e2e-prepull-images.sh
32+
}
33+
1034
@test "Install Cozystack" {
1135
# Install cozy-installer chart (operator installs CRDs on startup via --install-crds)
1236
helm upgrade installer packages/core/installer \

‎hack/e2e-prepull-images.sh‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
#!/usr/bin/env bash
2+
# Pre-pull images to all cluster nodes.
3+
#
4+
# Reads image references from stdin, one per line. Empty lines and lines
5+
# starting with '#' are ignored.
6+
#
7+
# Some workloads (OVN raft, LINSTOR) are sensitive to peer-readiness at
8+
# startup: if nodes pull the image at different speeds, one replica boots
9+
# before its peers are reachable, exhausts its raft connection retries, and
10+
# the HelmRelease install times out. Pre-pulling eliminates the pull stagger
11+
# so all replicas start within milliseconds of each other.
12+
#
13+
# Implementation: a DaemonSet with one regular container per image (parallel
14+
# pulls — total time = max of any one image rather than sum). kubectl rollout
15+
# status blocks until all pods are Ready (= all containers running = all
16+
# images cached on every node), then we delete the DaemonSet.
17+
18+
set -euo pipefail
19+
20+
cleanup() {
21+
kubectl delete daemonset e2e-image-prepuller -n kube-system --ignore-not-found
22+
}
23+
trap cleanup EXIT
24+
25+
mapfile -t images < <(grep -Ev '^[[:space:]]*(#|$)' | sort -u)
26+
27+
if [[ ${#images[@]} -eq 0 ]]; then
28+
echo "e2e-prepull-images: no images on stdin, nothing to do" >&2
29+
exit 0
30+
fi
31+
32+
echo "Pre-pulling ${#images[@]} image(s):"
33+
printf ' %s\n' "${images[@]}"
34+
35+
containers=""
36+
i=0
37+
for image in "${images[@]}"; do
38+
containers+="
39+
- name: pull-${i}
40+
image: ${image}
41+
command: [\"sleep\", \"infinity\"]
42+
imagePullPolicy: IfNotPresent
43+
resources:
44+
requests:
45+
cpu: 1m
46+
memory: 1Mi"
47+
i=$((i + 1))
48+
done
49+
50+
kubectl apply -f - <<EOF
51+
apiVersion: apps/v1
52+
kind: DaemonSet
53+
metadata:
54+
name: e2e-image-prepuller
55+
namespace: kube-system
56+
labels:
57+
app: e2e-image-prepuller
58+
spec:
59+
selector:
60+
matchLabels:
61+
app: e2e-image-prepuller
62+
template:
63+
metadata:
64+
labels:
65+
app: e2e-image-prepuller
66+
spec:
67+
# hostNetwork bypasses the CNI: this script runs BEFORE Cozystack
68+
# installs kube-ovn, so the cluster has no working CNI yet and a normal
69+
# pod would stay ContainerCreating with NetworkPluginNotReady, never
70+
# reaching image-pull. With hostNetwork the pod sandbox is created on
71+
# the host's namespace and the kubelet pulls images right away.
72+
# Same pattern the cozystack-operator pod uses for the same reason.
73+
hostNetwork: true
74+
# No need for an SA token — this pod just sleeps while images pull.
75+
automountServiceAccountToken: false
76+
tolerations:
77+
# Reach every node including control-plane and nodes still coming up.
78+
- operator: Exists
79+
containers:${containers}
80+
EOF
81+
82+
echo "Waiting for e2e-image-prepuller DaemonSet to become ready..."
83+
kubectl rollout status daemonset/e2e-image-prepuller -n kube-system --timeout=10m
84+
85+
echo "Image pre-pull complete."

0 commit comments

Comments
 (0)