Skip to content

Commit 0d53f13

Browse files
e2e: pre-pull timing-sensitive platform images before install
Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server) fail when replicas start at materially different times due to image-pull stagger across nodes. Add a DaemonSet-based pre-pull step that runs before `helm install`, ensuring every node has the images cached so replicas start within milliseconds of each other. The script accepts image refs on stdin and creates one container per image so pulls run in parallel — total time = max(any single image) rather than sum. The bats test sources image refs directly from the rendered platform charts via `yq`, walking only PodSpec-shaped objects, so version bumps stay in sync without a hardcoded list. Signed-off-by: Myasnikov Daniil <[email protected]> Assisted-By: Claude <[email protected]>
1 parent 5786afe commit 0d53f13

2 files changed

Lines changed: 101 additions & 0 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,26 @@
77
fi
88
}
99

10+
@test "Pre-pull platform images" {
11+
# Cluster-member workloads (OVN raft, LINSTOR) fail if replicas start at
12+
# different times due to image-pull stagger across nodes. Pre-pull these
13+
# images to every node so all replicas start with images already cached.
14+
#
15+
# Source images directly from the rendered charts so version bumps stay in
16+
# sync automatically. yq walks every PodSpec-shaped object and emits the
17+
# images of each container — this scopes the result to images the kubelet
18+
# actually pulls (skips configmap fields and CRD examples that happen to
19+
# contain an `image:` key). Add a chart here when a new peer-sensitive
20+
# workload is found.
21+
{
22+
helm template packages/system/kubeovn
23+
helm template packages/system/linstor
24+
} | yq -N '
25+
(..|select(has("containers"))|.containers[]|.image),
26+
(..|select(has("initContainers"))|.initContainers[]|.image)
27+
' | hack/e2e-prepull-images.sh
28+
}
29+
1030
@test "Install Cozystack" {
1131
# Install cozy-installer chart (operator installs CRDs on startup via --install-crds)
1232
helm upgrade installer packages/core/installer \

‎hack/e2e-prepull-images.sh‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
#!/usr/bin/env bash
2+
# Pre-pull images to all cluster nodes.
3+
#
4+
# Reads image references from stdin, one per line. Empty lines and lines
5+
# starting with '#' are ignored.
6+
#
7+
# Some workloads (OVN raft, LINSTOR) are sensitive to peer-readiness at
8+
# startup: if nodes pull the image at different speeds, one replica boots
9+
# before its peers are reachable, exhausts its raft connection retries, and
10+
# the HelmRelease install times out. Pre-pulling eliminates the pull stagger
11+
# so all replicas start within milliseconds of each other.
12+
#
13+
# Implementation: a DaemonSet with one regular container per image (parallel
14+
# pulls — total time = max of any one image rather than sum). kubectl rollout
15+
# status blocks until all pods are Ready (= all containers running = all
16+
# images cached on every node), then we delete the DaemonSet.
17+
18+
set -euo pipefail
19+
20+
mapfile -t images < <(grep -Ev '^[[:space:]]*(#|$)' | sort -u)
21+
22+
if [[ ${#images[@]} -eq 0 ]]; then
23+
echo "e2e-prepull-images: no images on stdin, nothing to do" >&2
24+
exit 0
25+
fi
26+
27+
echo "Pre-pulling ${#images[@]} image(s):"
28+
printf ' %s\n' "${images[@]}"
29+
30+
containers=""
31+
i=0
32+
for image in "${images[@]}"; do
33+
containers+="
34+
- name: pull-${i}
35+
image: ${image}
36+
command: [\"sleep\", \"infinity\"]
37+
imagePullPolicy: IfNotPresent
38+
resources:
39+
requests:
40+
cpu: 1m
41+
memory: 1Mi"
42+
i=$((i + 1))
43+
done
44+
45+
kubectl apply -f - <<EOF
46+
apiVersion: apps/v1
47+
kind: DaemonSet
48+
metadata:
49+
name: e2e-image-prepuller
50+
namespace: kube-system
51+
labels:
52+
app: e2e-image-prepuller
53+
spec:
54+
selector:
55+
matchLabels:
56+
app: e2e-image-prepuller
57+
template:
58+
metadata:
59+
labels:
60+
app: e2e-image-prepuller
61+
spec:
62+
# hostNetwork bypasses the CNI: this script runs BEFORE Cozystack
63+
# installs kube-ovn, so the cluster has no working CNI yet and a normal
64+
# pod would stay ContainerCreating with NetworkPluginNotReady, never
65+
# reaching image-pull. With hostNetwork the pod sandbox is created on
66+
# the host's namespace and the kubelet pulls images right away.
67+
# Same pattern the cozystack-operator pod uses for the same reason.
68+
hostNetwork: true
69+
# No need for an SA token — this pod just sleeps while images pull.
70+
automountServiceAccountToken: false
71+
tolerations:
72+
# Reach every node including control-plane and nodes still coming up.
73+
- operator: Exists
74+
containers:${containers}
75+
EOF
76+
77+
echo "Waiting for e2e-image-prepuller DaemonSet to become ready..."
78+
kubectl rollout status daemonset/e2e-image-prepuller -n kube-system --timeout=10m
79+
80+
kubectl delete daemonset e2e-image-prepuller -n kube-system --ignore-not-found
81+
echo "Image pre-pull complete."

0 commit comments

Comments
 (0)