Skip to content

Commit 86674c7

Browse files
committed
fix(e2e): budget each linstor wait separately in post-install-prep
The post-install prep script waits out a serial reconcile chain -- the linstor HelmRelease, then the linstor-controller Deployment -- against a single 15-minute deadline anchored at the script's start. The head of the chain therefore eats the window and the tail inherits whatever remains, so an install fails whenever the chain merely runs long rather than never converging. Measured on one such failure: the HelmRelease went Ready 452s in, and the Deployment needed a further ~520s because the controller's DB migration lost its apiserver connection and retried four times before succeeding. Neither figure exceeds 900s on its own; only their sum did, and the install reported the Deployment as though it were broken. Give each wait its own budget, started when the link before it completed -- the shape the LINSTOR node and MetalLB CRD waits further down already have. What that buys is not speed but a guarantee: every link gets its whole allowance regardless of what the links ahead of it spent, so the tail can no longer be failed by the head being slow. It is a trade, and the other half is worth stating. The script's worst case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job carries, where before the two waits shared one 15m window. And a link that is genuinely stuck is reported later rather than sooner: a Deployment that never converges surfaces at 1355s where the shared window surfaced it at 900s, on that run's timings. A HelmRelease that never becomes Ready surfaces at the same moment as before, since its budget still starts with the script. A timeout now also lists the linstor pods and the namespace events. The description names the object waited on, not the link that held it up, and the same "Deployment not Available" has been reached both with the controller pod crash-looping its migration init container and with that pod blocked on a client TLS Secret cert-manager had not issued yet. Telling those apart previously required the run's bundle. The events come from `kubectl events` rather than `kubectl get events`. It orders by when an event was last seen, so a condition repeating for minutes -- what a stuck mount looks like -- stays inside the window instead of being pushed out by newer one-off events, and it carries no --sort-by, so it cannot fail the whole read the way a sort key does when one item lacks it. Both reads keep their stderr and carry a bound. A read that fails silently is indistinguishable from a namespace with nothing to report, which is the distinction being drawn; and the caller is blocked in `wait`, so an unbounded read against a wedged apiserver would hold the install open until the job's own timeout. The client budget is strictly the smaller of the two so kubectl gets to name the reason before the outer kill takes it away. The wait helper is named for linstor because those diagnostics read one namespace unconditionally: the wait is generic over its kubectl arguments, what it dumps on the way out is not. The new unit test runs the script end to end against a stub PATH carrying a virtual clock, so a 15-minute budget elapses in a few hundred forks and the timings are exact rather than wall-clock approximate. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 677d80a commit 86674c7

2 files changed

Lines changed: 319 additions & 12 deletions

File tree

‎hack/e2e-post-install-prep.sh‎

Lines changed: 64 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,28 +13,80 @@
1313
# (timeout 60 / timeout 300) were anchored to this script's start, so they
1414
# raced that reconcile latency; when one lost (linstor HR appeared at ~+70s
1515
# against a 60s budget) `set -e` aborted the whole script and the install
16-
# failed. Instead, drive every wait off one shared deadline -- the same 15m
17-
# window the installer's `kubectl wait hr --all` uses -- and tolerate
18-
# not-yet-created objects without aborting, while still failing hard if a
19-
# resource never becomes ready inside the budget.
16+
# failed. So every wait below tolerates a not-yet-created object without
17+
# aborting, and each gets its own budget, started when the link before it
18+
# completed rather than when this script did -- the same shape the LINSTOR
19+
# node and MetalLB CRD waits further down already have.
20+
#
21+
# One budget shared by the whole chain instead fails the install whenever the
22+
# chain merely runs long, because the head of it eats the window and the tail
23+
# inherits the remainder. Both halves were observed inside a single 15m window
24+
# on one run: the linstor HR went Ready 452s in, and the Deployment needed a
25+
# further ~520s because the controller's DB migration lost its connection to
26+
# the apiserver and retried four times before succeeding. Neither figure is
27+
# anomalous and neither exceeds the budget on its own; only their sum did, and
28+
# the install failed naming the Deployment as though it were broken.
29+
#
30+
# What a per-link budget buys is not speed but a guarantee: every link gets its
31+
# whole allowance no matter what the links ahead of it spent, so the tail of the
32+
# chain can no longer be failed by the head merely being slow. Detecting a link
33+
# that is genuinely stuck does not get faster, and for the second link it gets
34+
# slower, because its budget starts when the first one finished: a Deployment
35+
# that never converges is reported later by however long the HelmRelease took.
36+
# A first link that never becomes Ready is reported at the same moment as
37+
# before, since its budget still starts with the script.
38+
#
39+
# The cost is the ceiling: two waits at LINK_BUDGET each put the worst case at
40+
# twice that figure rather than once, ahead of the 300s node wait and the 300s
41+
# MetalLB wait below, and still well inside the timeout the job carries.
2042
set -eu
2143

22-
DEADLINE=$(( $(date +%s) + 900 ))
44+
# Per-link budget in seconds, applied by wait_for_linstor to each link separately.
45+
LINK_BUDGET=900
2346

24-
# wait_for <description> <kubectl-wait-args...>
25-
# Polls `kubectl wait` until it succeeds or the shared deadline elapses.
47+
# wait_for_linstor <description> <kubectl-wait-args...>
48+
# Polls `kubectl wait` until it succeeds or this link's budget elapses. The
49+
# name says linstor because the timeout diagnostics below read cozy-linstor
50+
# unconditionally: the wait itself is generic over its kubectl arguments, but
51+
# what it dumps on the way out is not, and a caller elsewhere would get the
52+
# wrong namespace's pods presented as evidence.
2653
# kubectl wait exits non-zero immediately when the object does not exist yet,
2754
# so the loop tolerates "not created yet" without the set -e cliff that a bare
2855
# `kubectl wait` would trigger on a NotFound. The per-attempt timeout shrinks
2956
# to the budget remaining, so the final attempt can consume the rest of it.
30-
wait_for() {
57+
wait_for_linstor() {
3158
desc=$1
3259
shift
3360
echo "[post-install-prep] waiting for ${desc}"
61+
deadline=$(( $(date +%s) + LINK_BUDGET ))
3462
while :; do
35-
remaining=$(( DEADLINE - $(date +%s) ))
63+
remaining=$(( deadline - $(date +%s) ))
3664
if [ "$remaining" -le 0 ]; then
37-
echo "[post-install-prep] timed out waiting for ${desc}" >&2
65+
echo "[post-install-prep] timed out after ${LINK_BUDGET}s waiting for ${desc}" >&2
66+
# The description names the object waited on, not the link of the chain
67+
# that held it up, and those differ: the same "Deployment not Available"
68+
# has been reached with the controller pod crash-looping its migration
69+
# init container and with that pod unschedulable because cert-manager had
70+
# not issued its client TLS Secret yet. The pod list separates the two,
71+
# and the namespace events carry the reason a volume or an image failed.
72+
# `kubectl events` rather than `kubectl get events`: it orders by when an
73+
# event was last seen, so a condition that has been repeating for minutes
74+
# -- which is what a stuck mount looks like -- stays inside the window
75+
# instead of being pushed out of it by newer one-off events. It also has
76+
# no --sort-by, so it cannot fail the whole read the way a sort key does
77+
# when it is absent from any single item; .lastTimestamp is unset on an
78+
# Event written through events.k8s.io/v1, and that failure prints nothing
79+
# at all. Both reads keep their stderr, because a diagnostic that fails
80+
# silently is indistinguishable from a namespace with nothing to report
81+
# -- the very distinction being drawn here. Both are bounded, since the
82+
# caller is blocked in `wait` and a wedged apiserver would otherwise hold
83+
# the install open until the job's own timeout; the client budget is
84+
# strictly the smaller of the two, so kubectl gets to name the reason
85+
# before the outer kill takes it away.
86+
timeout -k 5 30 kubectl get pods -n cozy-linstor -o wide \
87+
--request-timeout=10s 2>&1 | tail -n 30 >&2
88+
timeout -k 5 30 kubectl events -n cozy-linstor \
89+
--request-timeout=10s 2>&1 | tail -n 30 >&2
3890
return 1
3991
fi
4092
if kubectl wait "$@" --timeout="${remaining}s" 2>/dev/null; then
@@ -57,9 +109,9 @@ controller_reachable() {
57109
-o jsonpath='{.subsets[*].addresses[*].ip}' 2>/dev/null)" ]
58110
}
59111

60-
wait_for "linstor HelmRelease to be Ready" \
112+
wait_for_linstor "linstor HelmRelease to be Ready" \
61113
helmrelease/linstor -n cozy-linstor --for=condition=Ready
62-
wait_for "linstor-controller Deployment to be Available" \
114+
wait_for_linstor "linstor-controller Deployment to be Available" \
63115
deployment/linstor-controller -n cozy-linstor --for=condition=available
64116

65117
# Wait for 3 satellites to register Online, but gate every probe on a ready

‎hack/post-install-prep.bats‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for the wait budgets in hack/e2e-post-install-prep.sh.
4+
#
5+
# The script drives a serial reconcile chain -- cozystack-operator -> platform
6+
# HR -> linstor HR -> piraeus-operator -> cert-manager issues the controller
7+
# TLS -> linstor-controller Deployment -- and every link of it is waited on in
8+
# order. What these tests pin is how the waits are budgeted: each link gets its
9+
# own window measured from the moment the link before it completed, so a long
10+
# but healthy head cannot starve the tail, while a link that never converges
11+
# still fails inside one window rather than running forever.
12+
#
13+
# Strategy: the script is run end to end against a stub PATH carrying a virtual
14+
# clock. `date` reads a counter file, `sleep` advances it, and `kubectl` decides
15+
# each wait from that same counter -- so a 15-minute budget elapses in a few
16+
# hundred forks and the timings are exact rather than wall-clock approximate.
17+
# The stub also records every kubectl invocation, which is how the timeout
18+
# diagnostics are asserted. Mock IPs use the RFC 5737 documentation range.
19+
#
20+
# The convergence times fed in are the ones measured on the two runs that
21+
# motivated this: the linstor HelmRelease went Ready 452s after the script
22+
# started, and the Deployment became Available a further ~520s later.
23+
#
24+
# Title syntax constraints (inherited from cozytest.sh's awk parser):
25+
# - Titles delimited by ASCII double quotes; embedded quotes truncate.
26+
# - Only [A-Za-z0-9] from the title survives into the function name, so keep
27+
# titles distinctive in their alphanumeric run.
28+
# - A line that is exactly `}` is rewritten, so stub bodies below use `case`
29+
# rather than nested functions.
30+
#
31+
# Run with: hack/cozytest.sh hack/post-install-prep.bats
32+
# -----------------------------------------------------------------------------
33+
34+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
35+
SCRIPT="$HACK_DIR/e2e-post-install-prep.sh"
36+
37+
# prep_sandbox <dir> -- lay down the stub PATH and the virtual clock.
38+
#
39+
# STUB_HR_READY_AT / STUB_DEPLOY_READY_AT are virtual-clock seconds at which
40+
# the corresponding `kubectl wait` starts succeeding, or the string `never`.
41+
# Every other kubectl call succeeds, so the script's post-LINSTOR tail (storage
42+
# pools, StorageClasses, MetalLB) runs through without a cluster.
43+
prep_sandbox() {
44+
d=$1
45+
mkdir -p "$d/bin"
46+
echo 0 > "$d/clock"
47+
: > "$d/calls"
48+
49+
cat > "$d/bin/date" <<'STUB'
50+
#!/bin/sh
51+
cat "$STUB_CLOCK"
52+
STUB
53+
54+
# Advancing the clock in `sleep` is what makes a 900s budget cost ~180 forks
55+
# instead of 15 minutes. The ceiling is a runaway guard: if the script under
56+
# test loses its deadline check entirely, the wait loop would otherwise spin
57+
# forever, and a hung unit test is worse than a failing one. It has to stay
58+
# low enough to trip in seconds -- a guard that itself takes minutes to fire
59+
# is the hang it was meant to prevent -- and above every budget under test.
60+
cat > "$d/bin/sleep" <<'STUB'
61+
#!/bin/sh
62+
now=$(cat "$STUB_CLOCK")
63+
now=$(( now + ${1%%.*} ))
64+
echo "$now" > "$STUB_CLOCK"
65+
[ "$now" -lt 5000 ]
66+
STUB
67+
68+
# Records the parsed bound and then runs the real command, so a wrapped call
69+
# still reaches the kubectl stub and the existing call assertions hold. The
70+
# duration is parsed here rather than in the test because `timeout [-k G] N`
71+
# puts two numbers on the line and only the second one is the wall clock.
72+
cat > "$d/bin/timeout" <<'STUB'
73+
#!/bin/sh
74+
grace=""
75+
dur=""
76+
while [ $# -gt 0 ]; do
77+
case $1 in
78+
-k) grace=$2; shift 2 ;;
79+
[0-9]*) dur=$1; shift; break ;;
80+
*) break ;;
81+
esac
82+
done
83+
printf 'timeout dur=%s grace=%s -- %s\n' "$dur" "$grace" "$*" >> "$STUB_CALLS"
84+
exec "$@"
85+
STUB
86+
87+
cat > "$d/bin/kubectl" <<'STUB'
88+
#!/bin/sh
89+
now=$(cat "$STUB_CLOCK")
90+
printf '%s\n' "$*" >> "$STUB_CALLS"
91+
case "$*" in
92+
'wait helmrelease/linstor '*)
93+
[ "$STUB_HR_READY_AT" != never ] && [ "$now" -ge "$STUB_HR_READY_AT" ] ;;
94+
'wait deployment/linstor-controller '*)
95+
[ "$STUB_DEPLOY_READY_AT" != never ] && [ "$now" -ge "$STUB_DEPLOY_READY_AT" ] ;;
96+
'get endpoints '*) echo 192.0.2.11 ;;
97+
'get pods '*)
98+
if [ -n "${STUB_PODS_FAIL:-}" ]; then
99+
echo "Error from server (Forbidden): pods is forbidden" >&2
100+
exit 1
101+
fi
102+
echo "STUB-POD linstor-controller 0/1 Init:CrashLoopBackOff" ;;
103+
'events '*|'get events '*)
104+
# Model the failure the real sorter has: an Event written through
105+
# events.k8s.io/v1 reaches the core API with .lastTimestamp unset, and
106+
# kubectl fails the whole read rather than skipping that item. A
107+
# diagnostic that dies this way prints nothing, which reads exactly like
108+
# a namespace with no events -- so the assertion below is on the content
109+
# arriving, not on the call being made.
110+
case "$*" in
111+
*--sort-by=.lastTimestamp*)
112+
echo "error: couldn't find any field with path {.lastTimestamp}" >&2
113+
exit 1 ;;
114+
esac
115+
# Any other reason the read can fail -- RBAC, a wedged apiserver, a
116+
# renamed namespace. Kept separate from the sort-key case above so the
117+
# cause and the consequence are pinned by different tests.
118+
if [ -n "${STUB_EVENTS_FAIL:-}" ]; then
119+
echo "Error from server (Forbidden): events is forbidden" >&2
120+
exit 1
121+
fi
122+
echo "STUB-EVENT MountVolume.SetUp failed for volume client-tls" ;;
123+
*'linstor node list') printf 'Online\nOnline\nOnline\n' ;;
124+
*) exit 0 ;;
125+
esac
126+
STUB
127+
128+
chmod +x "$d/bin/date" "$d/bin/sleep" "$d/bin/kubectl" "$d/bin/timeout"
129+
}
130+
131+
# run_prep <dir> <hr-ready-at> <deploy-ready-at> [events-fail] [pods-fail]
132+
# Runs the script under the stubs, capturing stdout, stderr and the exit status
133+
# without tripping set -e. The last two are non-empty to make the corresponding
134+
# diagnostic read fail; pass an empty string for the fourth to reach the fifth.
135+
run_prep() {
136+
d=$1
137+
STUB_CLOCK="$d/clock" STUB_CALLS="$d/calls" \
138+
STUB_HR_READY_AT=$2 STUB_DEPLOY_READY_AT=$3 \
139+
STUB_EVENTS_FAIL="${4:-}" STUB_PODS_FAIL="${5:-}" \
140+
PATH="$d/bin:$PATH" \
141+
"$SCRIPT" > "$d/out" 2> "$d/err" && echo 0 > "$d/rc" || echo $? > "$d/rc"
142+
}
143+
144+
@test "the Deployment wait keeps its full budget after a slow HelmRelease wait" {
145+
tmp=$(mktemp -d)
146+
prep_sandbox "$tmp"
147+
148+
run_prep "$tmp" 452 972
149+
150+
cat "$tmp/err" >&2
151+
[ "$(cat "$tmp/rc")" -eq 0 ]
152+
grep -q '\[post-install-prep\] done' "$tmp/out"
153+
# The Deployment converged past the point where a single chain-wide 15m
154+
# window anchored at script start would already have expired.
155+
[ "$(cat "$tmp/clock")" -ge 972 ]
156+
rm -rf "$tmp"
157+
}
158+
159+
@test "a Deployment that never becomes Available fails inside one budget" {
160+
tmp=$(mktemp -d)
161+
prep_sandbox "$tmp"
162+
163+
run_prep "$tmp" 0 never
164+
165+
[ "$(cat "$tmp/rc")" -ne 0 ]
166+
# One budget, not two and not unbounded: the wait started at 0 and gave up
167+
# at 900. The upper bound allows a single 5s poll interval beyond that and
168+
# nothing more, so a second budget would fail it.
169+
final=$(cat "$tmp/clock")
170+
[ "$final" -ge 900 ]
171+
[ "$final" -le 905 ]
172+
rm -rf "$tmp"
173+
}
174+
175+
@test "the Deployment timeout reports the linstor pods and events" {
176+
tmp=$(mktemp -d)
177+
prep_sandbox "$tmp"
178+
179+
run_prep "$tmp" 0 never
180+
181+
[ "$(cat "$tmp/rc")" -ne 0 ]
182+
grep -q 'timed out' "$tmp/err"
183+
grep -q 'linstor-controller Deployment to be Available' "$tmp/err"
184+
# Which link was actually blocking -- a crash-looping init container versus a
185+
# Secret cert-manager has not issued yet -- is only visible in the pods and
186+
# the namespace events, so the timeout has to read both.
187+
grep -q '^get pods -n cozy-linstor' "$tmp/calls"
188+
grep -q '^events -n cozy-linstor' "$tmp/calls"
189+
# And their content has to reach the log, not merely be asked for: a read
190+
# that fails prints nothing, which is the same shape as a namespace with
191+
# nothing wrong in it.
192+
grep -q 'STUB-POD' "$tmp/err"
193+
grep -q 'STUB-EVENT' "$tmp/err"
194+
# Both reads carry a wall-clock bound, and the client budget inside each is
195+
# strictly smaller than it. Equal values would put the outer kill first, so
196+
# kubectl would never get to say why it could not read -- which is the whole
197+
# point of reading at all on this path.
198+
for what in 'get pods' 'events'; do
199+
line=$(grep "^timeout dur=.* kubectl $what -n cozy-linstor" "$tmp/calls" | head -1)
200+
[ -n "$line" ]
201+
outer=${line#timeout dur=}
202+
outer=${outer%% *}
203+
inner=$(printf '%s' "$line" | sed -n 's/.*--request-timeout=\([0-9]*\)s.*/\1/p')
204+
[ -n "$outer" ]
205+
[ -n "$inner" ]
206+
[ "$inner" -lt "$outer" ]
207+
done
208+
rm -rf "$tmp"
209+
}
210+
211+
@test "an events read that fails names its reason instead of going quiet" {
212+
tmp=$(mktemp -d)
213+
prep_sandbox "$tmp"
214+
215+
run_prep "$tmp" 0 never fail-events
216+
217+
[ "$(cat "$tmp/rc")" -ne 0 ]
218+
# Silence here would be read as "the namespace had nothing to say", which is
219+
# the opposite of what happened. The reason has to survive to the log.
220+
grep -q 'Forbidden' "$tmp/err"
221+
rm -rf "$tmp"
222+
}
223+
224+
@test "a leg that fails does not take the other leg down with it" {
225+
tmp=$(mktemp -d)
226+
prep_sandbox "$tmp"
227+
228+
# The pods read runs first, so failing it is the only way to ask whether the
229+
# events read still happens. Failing the events read instead proves nothing:
230+
# the pods output has already been emitted by then either way.
231+
run_prep "$tmp" 0 never '' fail-pods
232+
233+
[ "$(cat "$tmp/rc")" -ne 0 ]
234+
grep -q 'Forbidden' "$tmp/err"
235+
grep -q 'STUB-EVENT' "$tmp/err"
236+
rm -rf "$tmp"
237+
}
238+
239+
@test "a HelmRelease that never goes Ready fails before the Deployment is waited on" {
240+
tmp=$(mktemp -d)
241+
prep_sandbox "$tmp"
242+
243+
run_prep "$tmp" never 0
244+
245+
[ "$(cat "$tmp/rc")" -ne 0 ]
246+
grep -q 'linstor HelmRelease to be Ready' "$tmp/err"
247+
final=$(cat "$tmp/clock")
248+
[ "$final" -ge 900 ]
249+
[ "$final" -le 905 ]
250+
if grep -q '^wait deployment/linstor-controller' "$tmp/calls"; then
251+
echo "the Deployment was waited on after the HelmRelease wait failed" >&2
252+
return 1
253+
fi
254+
rm -rf "$tmp"
255+
}

0 commit comments

Comments
 (0)