Skip to content

Commit ccd8a28

Browse files
authored
fix(e2e): budget each linstor wait separately in post-install-prep (#3715)
## What this PR does Both waits in `hack/e2e-post-install-prep.sh` ran off one 15-minute deadline anchored at the script's start, so the head of the chain ate the window and the tail got whatever was left. Two runs on 2026-08-09 failed that way. The linstor HelmRelease went Ready 452s in, and the Deployment needed a further ~520s because its DB migration lost the apiserver and retried four times. Neither figure passes 900s alone. Their sum does, and the install reported the Deployment as broken. Each wait now gets its own budget, started when the previous link finished. Same shape as the node wait and the MetalLB CRD wait further down the same file. What it buys is that every link gets its whole allowance regardless of what the links ahead of it spent. The cost is on both sides and I measured it. Worst case goes from one 15m window to 900 + 900 + 300 + 300 = 40m, against the 180m the e2e job carries. A Deployment that never converges is now reported at 1355s instead of 900s, from the 452s above plus the loop's 5s poll. A HelmRelease that never goes Ready is still reported at 900s, which I checked by running the new suite against the pre-change script. The timeout now dumps the pods and recent events of `cozy-linstor`. The same "Deployment not Available" has come from a crash-looping migration init container and from a pod waiting on a client TLS Secret cert-manager had not issued yet, and telling those apart used to need the run's bundle. Events come from `kubectl events`, which sorts by last-seen and carries no `--sort-by` to fail on a key some item lacks. Both reads keep their stderr and are bounded, with the client budget smaller than the outer one so kubectl gets to say why before the kill lands. `hack/post-install-prep.bats` runs the script end to end against a stub PATH with a virtual clock, so a 15-minute budget elapses in a few hundred forks and the timings are exact. Four of its six tests fail against the pre-change script. The other two guard the bound that stays. ### Known gaps in the new tests Dropping the `2>&1` from either diagnostic read leaves all six tests green, because the harness catches the script's own stderr into the same file the assertion reads. The property holds in production anyway, since the caller merges both streams into one log, but nothing catches the removal. The test for "one failing read does not suppress the other" needs the pods read to run first; swap the two reads and it passes vacuously. Its comment says so at the point where someone would make that change. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the map against the diff. Two files, both under `hack/`. The `ccp` trigger is about moving or renaming things under `hack/` or changing what a make target does, and its anchors `hack/package.mk` and `hack/common-envs.mk` are untouched. `talm` and `ansible-cozystack` key off node prerequisites in `hack/e2e-prepare-cluster.bats`, a different file. ### Related Same family, found while working here and filed instead of folded in. #3707 and #3708 for unbounded reads elsewhere in this script, #3712 for the `--sort-by=.lastTimestamp` form in fourteen other places, #3713 for the wait loop dropping kubectl's stderr for its whole budget, #3714 for the MetalLB CRD wait failing without a message. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved LINSTOR readiness checks by giving each prerequisite its own 15-minute wait period. * Added clearer timeout diagnostics, including relevant pod and event information. * Ensured subsequent readiness checks are skipped when an earlier HelmRelease check fails. * **Tests** * Added end-to-end coverage for timeout handling, sequential failures, diagnostic errors, and readiness behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 6ee56e3 + 86674c7 commit ccd8a28

2 files changed

Lines changed: 319 additions & 12 deletions

File tree

‎hack/e2e-post-install-prep.sh‎

Lines changed: 64 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,28 +13,80 @@
1313
# (timeout 60 / timeout 300) were anchored to this script's start, so they
1414
# raced that reconcile latency; when one lost (linstor HR appeared at ~+70s
1515
# against a 60s budget) `set -e` aborted the whole script and the install
16-
# failed. Instead, drive every wait off one shared deadline -- the same 15m
17-
# window the installer's `kubectl wait hr --all` uses -- and tolerate
18-
# not-yet-created objects without aborting, while still failing hard if a
19-
# resource never becomes ready inside the budget.
16+
# failed. So every wait below tolerates a not-yet-created object without
17+
# aborting, and each gets its own budget, started when the link before it
18+
# completed rather than when this script did -- the same shape the LINSTOR
19+
# node and MetalLB CRD waits further down already have.
20+
#
21+
# One budget shared by the whole chain instead fails the install whenever the
22+
# chain merely runs long, because the head of it eats the window and the tail
23+
# inherits the remainder. Both halves were observed inside a single 15m window
24+
# on one run: the linstor HR went Ready 452s in, and the Deployment needed a
25+
# further ~520s because the controller's DB migration lost its connection to
26+
# the apiserver and retried four times before succeeding. Neither figure is
27+
# anomalous and neither exceeds the budget on its own; only their sum did, and
28+
# the install failed naming the Deployment as though it were broken.
29+
#
30+
# What a per-link budget buys is not speed but a guarantee: every link gets its
31+
# whole allowance no matter what the links ahead of it spent, so the tail of the
32+
# chain can no longer be failed by the head merely being slow. Detecting a link
33+
# that is genuinely stuck does not get faster, and for the second link it gets
34+
# slower, because its budget starts when the first one finished: a Deployment
35+
# that never converges is reported later by however long the HelmRelease took.
36+
# A first link that never becomes Ready is reported at the same moment as
37+
# before, since its budget still starts with the script.
38+
#
39+
# The cost is the ceiling: two waits at LINK_BUDGET each put the worst case at
40+
# twice that figure rather than once, ahead of the 300s node wait and the 300s
41+
# MetalLB wait below, and still well inside the timeout the job carries.
2042
set -eu
2143

22-
DEADLINE=$(( $(date +%s) + 900 ))
44+
# Per-link budget in seconds, applied by wait_for_linstor to each link separately.
45+
LINK_BUDGET=900
2346

24-
# wait_for <description> <kubectl-wait-args...>
25-
# Polls `kubectl wait` until it succeeds or the shared deadline elapses.
47+
# wait_for_linstor <description> <kubectl-wait-args...>
48+
# Polls `kubectl wait` until it succeeds or this link's budget elapses. The
49+
# name says linstor because the timeout diagnostics below read cozy-linstor
50+
# unconditionally: the wait itself is generic over its kubectl arguments, but
51+
# what it dumps on the way out is not, and a caller elsewhere would get the
52+
# wrong namespace's pods presented as evidence.
2653
# kubectl wait exits non-zero immediately when the object does not exist yet,
2754
# so the loop tolerates "not created yet" without the set -e cliff that a bare
2855
# `kubectl wait` would trigger on a NotFound. The per-attempt timeout shrinks
2956
# to the budget remaining, so the final attempt can consume the rest of it.
30-
wait_for() {
57+
wait_for_linstor() {
3158
desc=$1
3259
shift
3360
echo "[post-install-prep] waiting for ${desc}"
61+
deadline=$(( $(date +%s) + LINK_BUDGET ))
3462
while :; do
35-
remaining=$(( DEADLINE - $(date +%s) ))
63+
remaining=$(( deadline - $(date +%s) ))
3664
if [ "$remaining" -le 0 ]; then
37-
echo "[post-install-prep] timed out waiting for ${desc}" >&2
65+
echo "[post-install-prep] timed out after ${LINK_BUDGET}s waiting for ${desc}" >&2
66+
# The description names the object waited on, not the link of the chain
67+
# that held it up, and those differ: the same "Deployment not Available"
68+
# has been reached with the controller pod crash-looping its migration
69+
# init container and with that pod unschedulable because cert-manager had
70+
# not issued its client TLS Secret yet. The pod list separates the two,
71+
# and the namespace events carry the reason a volume or an image failed.
72+
# `kubectl events` rather than `kubectl get events`: it orders by when an
73+
# event was last seen, so a condition that has been repeating for minutes
74+
# -- which is what a stuck mount looks like -- stays inside the window
75+
# instead of being pushed out of it by newer one-off events. It also has
76+
# no --sort-by, so it cannot fail the whole read the way a sort key does
77+
# when it is absent from any single item; .lastTimestamp is unset on an
78+
# Event written through events.k8s.io/v1, and that failure prints nothing
79+
# at all. Both reads keep their stderr, because a diagnostic that fails
80+
# silently is indistinguishable from a namespace with nothing to report
81+
# -- the very distinction being drawn here. Both are bounded, since the
82+
# caller is blocked in `wait` and a wedged apiserver would otherwise hold
83+
# the install open until the job's own timeout; the client budget is
84+
# strictly the smaller of the two, so kubectl gets to name the reason
85+
# before the outer kill takes it away.
86+
timeout -k 5 30 kubectl get pods -n cozy-linstor -o wide \
87+
--request-timeout=10s 2>&1 | tail -n 30 >&2
88+
timeout -k 5 30 kubectl events -n cozy-linstor \
89+
--request-timeout=10s 2>&1 | tail -n 30 >&2
3890
return 1
3991
fi
4092
if kubectl wait "$@" --timeout="${remaining}s" 2>/dev/null; then
@@ -57,9 +109,9 @@ controller_reachable() {
57109
-o jsonpath='{.subsets[*].addresses[*].ip}' 2>/dev/null)" ]
58110
}
59111

60-
wait_for "linstor HelmRelease to be Ready" \
112+
wait_for_linstor "linstor HelmRelease to be Ready" \
61113
helmrelease/linstor -n cozy-linstor --for=condition=Ready
62-
wait_for "linstor-controller Deployment to be Available" \
114+
wait_for_linstor "linstor-controller Deployment to be Available" \
63115
deployment/linstor-controller -n cozy-linstor --for=condition=available
64116

65117
# Wait for 3 satellites to register Online, but gate every probe on a ready

‎hack/post-install-prep.bats‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for the wait budgets in hack/e2e-post-install-prep.sh.
4+
#
5+
# The script drives a serial reconcile chain -- cozystack-operator -> platform
6+
# HR -> linstor HR -> piraeus-operator -> cert-manager issues the controller
7+
# TLS -> linstor-controller Deployment -- and every link of it is waited on in
8+
# order. What these tests pin is how the waits are budgeted: each link gets its
9+
# own window measured from the moment the link before it completed, so a long
10+
# but healthy head cannot starve the tail, while a link that never converges
11+
# still fails inside one window rather than running forever.
12+
#
13+
# Strategy: the script is run end to end against a stub PATH carrying a virtual
14+
# clock. `date` reads a counter file, `sleep` advances it, and `kubectl` decides
15+
# each wait from that same counter -- so a 15-minute budget elapses in a few
16+
# hundred forks and the timings are exact rather than wall-clock approximate.
17+
# The stub also records every kubectl invocation, which is how the timeout
18+
# diagnostics are asserted. Mock IPs use the RFC 5737 documentation range.
19+
#
20+
# The convergence times fed in are the ones measured on the two runs that
21+
# motivated this: the linstor HelmRelease went Ready 452s after the script
22+
# started, and the Deployment became Available a further ~520s later.
23+
#
24+
# Title syntax constraints (inherited from cozytest.sh's awk parser):
25+
# - Titles delimited by ASCII double quotes; embedded quotes truncate.
26+
# - Only [A-Za-z0-9] from the title survives into the function name, so keep
27+
# titles distinctive in their alphanumeric run.
28+
# - A line that is exactly `}` is rewritten, so stub bodies below use `case`
29+
# rather than nested functions.
30+
#
31+
# Run with: hack/cozytest.sh hack/post-install-prep.bats
32+
# -----------------------------------------------------------------------------
33+
34+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
35+
SCRIPT="$HACK_DIR/e2e-post-install-prep.sh"
36+
37+
# prep_sandbox <dir> -- lay down the stub PATH and the virtual clock.
38+
#
39+
# STUB_HR_READY_AT / STUB_DEPLOY_READY_AT are virtual-clock seconds at which
40+
# the corresponding `kubectl wait` starts succeeding, or the string `never`.
41+
# Every other kubectl call succeeds, so the script's post-LINSTOR tail (storage
42+
# pools, StorageClasses, MetalLB) runs through without a cluster.
43+
prep_sandbox() {
44+
d=$1
45+
mkdir -p "$d/bin"
46+
echo 0 > "$d/clock"
47+
: > "$d/calls"
48+
49+
cat > "$d/bin/date" <<'STUB'
50+
#!/bin/sh
51+
cat "$STUB_CLOCK"
52+
STUB
53+
54+
# Advancing the clock in `sleep` is what makes a 900s budget cost ~180 forks
55+
# instead of 15 minutes. The ceiling is a runaway guard: if the script under
56+
# test loses its deadline check entirely, the wait loop would otherwise spin
57+
# forever, and a hung unit test is worse than a failing one. It has to stay
58+
# low enough to trip in seconds -- a guard that itself takes minutes to fire
59+
# is the hang it was meant to prevent -- and above every budget under test.
60+
cat > "$d/bin/sleep" <<'STUB'
61+
#!/bin/sh
62+
now=$(cat "$STUB_CLOCK")
63+
now=$(( now + ${1%%.*} ))
64+
echo "$now" > "$STUB_CLOCK"
65+
[ "$now" -lt 5000 ]
66+
STUB
67+
68+
# Records the parsed bound and then runs the real command, so a wrapped call
69+
# still reaches the kubectl stub and the existing call assertions hold. The
70+
# duration is parsed here rather than in the test because `timeout [-k G] N`
71+
# puts two numbers on the line and only the second one is the wall clock.
72+
cat > "$d/bin/timeout" <<'STUB'
73+
#!/bin/sh
74+
grace=""
75+
dur=""
76+
while [ $# -gt 0 ]; do
77+
case $1 in
78+
-k) grace=$2; shift 2 ;;
79+
[0-9]*) dur=$1; shift; break ;;
80+
*) break ;;
81+
esac
82+
done
83+
printf 'timeout dur=%s grace=%s -- %s\n' "$dur" "$grace" "$*" >> "$STUB_CALLS"
84+
exec "$@"
85+
STUB
86+
87+
cat > "$d/bin/kubectl" <<'STUB'
88+
#!/bin/sh
89+
now=$(cat "$STUB_CLOCK")
90+
printf '%s\n' "$*" >> "$STUB_CALLS"
91+
case "$*" in
92+
'wait helmrelease/linstor '*)
93+
[ "$STUB_HR_READY_AT" != never ] && [ "$now" -ge "$STUB_HR_READY_AT" ] ;;
94+
'wait deployment/linstor-controller '*)
95+
[ "$STUB_DEPLOY_READY_AT" != never ] && [ "$now" -ge "$STUB_DEPLOY_READY_AT" ] ;;
96+
'get endpoints '*) echo 192.0.2.11 ;;
97+
'get pods '*)
98+
if [ -n "${STUB_PODS_FAIL:-}" ]; then
99+
echo "Error from server (Forbidden): pods is forbidden" >&2
100+
exit 1
101+
fi
102+
echo "STUB-POD linstor-controller 0/1 Init:CrashLoopBackOff" ;;
103+
'events '*|'get events '*)
104+
# Model the failure the real sorter has: an Event written through
105+
# events.k8s.io/v1 reaches the core API with .lastTimestamp unset, and
106+
# kubectl fails the whole read rather than skipping that item. A
107+
# diagnostic that dies this way prints nothing, which reads exactly like
108+
# a namespace with no events -- so the assertion below is on the content
109+
# arriving, not on the call being made.
110+
case "$*" in
111+
*--sort-by=.lastTimestamp*)
112+
echo "error: couldn't find any field with path {.lastTimestamp}" >&2
113+
exit 1 ;;
114+
esac
115+
# Any other reason the read can fail -- RBAC, a wedged apiserver, a
116+
# renamed namespace. Kept separate from the sort-key case above so the
117+
# cause and the consequence are pinned by different tests.
118+
if [ -n "${STUB_EVENTS_FAIL:-}" ]; then
119+
echo "Error from server (Forbidden): events is forbidden" >&2
120+
exit 1
121+
fi
122+
echo "STUB-EVENT MountVolume.SetUp failed for volume client-tls" ;;
123+
*'linstor node list') printf 'Online\nOnline\nOnline\n' ;;
124+
*) exit 0 ;;
125+
esac
126+
STUB
127+
128+
chmod +x "$d/bin/date" "$d/bin/sleep" "$d/bin/kubectl" "$d/bin/timeout"
129+
}
130+
131+
# run_prep <dir> <hr-ready-at> <deploy-ready-at> [events-fail] [pods-fail]
132+
# Runs the script under the stubs, capturing stdout, stderr and the exit status
133+
# without tripping set -e. The last two are non-empty to make the corresponding
134+
# diagnostic read fail; pass an empty string for the fourth to reach the fifth.
135+
run_prep() {
136+
d=$1
137+
STUB_CLOCK="$d/clock" STUB_CALLS="$d/calls" \
138+
STUB_HR_READY_AT=$2 STUB_DEPLOY_READY_AT=$3 \
139+
STUB_EVENTS_FAIL="${4:-}" STUB_PODS_FAIL="${5:-}" \
140+
PATH="$d/bin:$PATH" \
141+
"$SCRIPT" > "$d/out" 2> "$d/err" && echo 0 > "$d/rc" || echo $? > "$d/rc"
142+
}
143+
144+
@test "the Deployment wait keeps its full budget after a slow HelmRelease wait" {
145+
tmp=$(mktemp -d)
146+
prep_sandbox "$tmp"
147+
148+
run_prep "$tmp" 452 972
149+
150+
cat "$tmp/err" >&2
151+
[ "$(cat "$tmp/rc")" -eq 0 ]
152+
grep -q '\[post-install-prep\] done' "$tmp/out"
153+
# The Deployment converged past the point where a single chain-wide 15m
154+
# window anchored at script start would already have expired.
155+
[ "$(cat "$tmp/clock")" -ge 972 ]
156+
rm -rf "$tmp"
157+
}
158+
159+
@test "a Deployment that never becomes Available fails inside one budget" {
160+
tmp=$(mktemp -d)
161+
prep_sandbox "$tmp"
162+
163+
run_prep "$tmp" 0 never
164+
165+
[ "$(cat "$tmp/rc")" -ne 0 ]
166+
# One budget, not two and not unbounded: the wait started at 0 and gave up
167+
# at 900. The upper bound allows a single 5s poll interval beyond that and
168+
# nothing more, so a second budget would fail it.
169+
final=$(cat "$tmp/clock")
170+
[ "$final" -ge 900 ]
171+
[ "$final" -le 905 ]
172+
rm -rf "$tmp"
173+
}
174+
175+
@test "the Deployment timeout reports the linstor pods and events" {
176+
tmp=$(mktemp -d)
177+
prep_sandbox "$tmp"
178+
179+
run_prep "$tmp" 0 never
180+
181+
[ "$(cat "$tmp/rc")" -ne 0 ]
182+
grep -q 'timed out' "$tmp/err"
183+
grep -q 'linstor-controller Deployment to be Available' "$tmp/err"
184+
# Which link was actually blocking -- a crash-looping init container versus a
185+
# Secret cert-manager has not issued yet -- is only visible in the pods and
186+
# the namespace events, so the timeout has to read both.
187+
grep -q '^get pods -n cozy-linstor' "$tmp/calls"
188+
grep -q '^events -n cozy-linstor' "$tmp/calls"
189+
# And their content has to reach the log, not merely be asked for: a read
190+
# that fails prints nothing, which is the same shape as a namespace with
191+
# nothing wrong in it.
192+
grep -q 'STUB-POD' "$tmp/err"
193+
grep -q 'STUB-EVENT' "$tmp/err"
194+
# Both reads carry a wall-clock bound, and the client budget inside each is
195+
# strictly smaller than it. Equal values would put the outer kill first, so
196+
# kubectl would never get to say why it could not read -- which is the whole
197+
# point of reading at all on this path.
198+
for what in 'get pods' 'events'; do
199+
line=$(grep "^timeout dur=.* kubectl $what -n cozy-linstor" "$tmp/calls" | head -1)
200+
[ -n "$line" ]
201+
outer=${line#timeout dur=}
202+
outer=${outer%% *}
203+
inner=$(printf '%s' "$line" | sed -n 's/.*--request-timeout=\([0-9]*\)s.*/\1/p')
204+
[ -n "$outer" ]
205+
[ -n "$inner" ]
206+
[ "$inner" -lt "$outer" ]
207+
done
208+
rm -rf "$tmp"
209+
}
210+
211+
@test "an events read that fails names its reason instead of going quiet" {
212+
tmp=$(mktemp -d)
213+
prep_sandbox "$tmp"
214+
215+
run_prep "$tmp" 0 never fail-events
216+
217+
[ "$(cat "$tmp/rc")" -ne 0 ]
218+
# Silence here would be read as "the namespace had nothing to say", which is
219+
# the opposite of what happened. The reason has to survive to the log.
220+
grep -q 'Forbidden' "$tmp/err"
221+
rm -rf "$tmp"
222+
}
223+
224+
@test "a leg that fails does not take the other leg down with it" {
225+
tmp=$(mktemp -d)
226+
prep_sandbox "$tmp"
227+
228+
# The pods read runs first, so failing it is the only way to ask whether the
229+
# events read still happens. Failing the events read instead proves nothing:
230+
# the pods output has already been emitted by then either way.
231+
run_prep "$tmp" 0 never '' fail-pods
232+
233+
[ "$(cat "$tmp/rc")" -ne 0 ]
234+
grep -q 'Forbidden' "$tmp/err"
235+
grep -q 'STUB-EVENT' "$tmp/err"
236+
rm -rf "$tmp"
237+
}
238+
239+
@test "a HelmRelease that never goes Ready fails before the Deployment is waited on" {
240+
tmp=$(mktemp -d)
241+
prep_sandbox "$tmp"
242+
243+
run_prep "$tmp" never 0
244+
245+
[ "$(cat "$tmp/rc")" -ne 0 ]
246+
grep -q 'linstor HelmRelease to be Ready' "$tmp/err"
247+
final=$(cat "$tmp/clock")
248+
[ "$final" -ge 900 ]
249+
[ "$final" -le 905 ]
250+
if grep -q '^wait deployment/linstor-controller' "$tmp/calls"; then
251+
echo "the Deployment was waited on after the HelmRelease wait failed" >&2
252+
return 1
253+
fi
254+
rm -rf "$tmp"
255+
}

0 commit comments

Comments
 (0)