Skip to content

Commit c696154

Browse files
committed
fix(tests): report a data-plane capture cut short by its backstop
Both callers of the data-plane collector wrap it in a wall-clock backstop and threw the result away with `|| true`, while the previous-instance logs leg sitting beside each of them records the exit status and prints "INCOMPLETE (exit N); kept what landed in ...". When a backstop killed the collector, nothing said so, and a truncated dataplane/ directory read exactly like a complete one that found little. The collector itself now names every read of its own it could not finish, which left these two call sites as the only remaining way for that leg to die without a word. The Chainsaw one is the tighter of the pair -- 300s against 600s, because it shares an op envelope with the snapshot leg -- so it is both likelier to fire and more expensive to leave silent. In each place the status is read, never propagated: a diagnostic still cannot change a test's outcome. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 9b11db1 commit c696154

4 files changed

Lines changed: 129 additions & 6 deletions

File tree

‎hack/capture-dataplane.bats‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1331,3 +1331,42 @@ STUB
13311331
# A success still wins: the address answered, whatever else could not be tried.
13321332
[ "$(printf 'unknown\nok\n' | lb_capture_decision)" = "skip" ]
13331333
}
1334+
1335+
@test "the Chainsaw caller also reports a capture its backstop cut short" {
1336+
# This collector has two callers, and the honesty contract has to hold at both
1337+
# or the reader learns to distrust the one that keeps it. hack/cozytest.sh's
1338+
# side is covered behaviourally in cozytest-capture-gate.bats; this runs the
1339+
# Chainsaw catch itself, extracted from the config it lives in, so neither
1340+
# side can quietly stop reporting. The Chainsaw backstop is the shorter of the
1341+
# two -- 300s against 600s, because it shares an op envelope with the snapshot
1342+
# leg -- so it is the likelier of the pair to fire.
1343+
#
1344+
# The backstop is simulated with a `timeout` on PATH that exits 124, for the
1345+
# same reason the cozytest.sh-side test does it: the call site sees a non-zero
1346+
# status and whatever landed, whichever way the kill arrived.
1347+
# Named here rather than left to fail downstream: without it a missing yq
1348+
# surfaces as the extraction guard below going red, which reads as the catch
1349+
# op having been renamed. Same form the other yq-using suites use.
1350+
command -v yq >/dev/null || { echo "yq (mikefarah v4+) is required to read the Chainsaw catch script" >&2; exit 1; }
1351+
d=$(mktemp -d)
1352+
mkdir -p "$d/bin" "$d/hack/e2e-chainsaw/suite"
1353+
printf '#!/bin/sh\nexit 124\n' >"$d/bin/timeout"
1354+
printf '#!/bin/sh\nexit 0\n' >"$d/bin/crust-gather"
1355+
# The catch resolves this collector as ../../e2e-capture-dataplane.sh from the
1356+
# failing suite's own directory, and gates the leg on it being executable.
1357+
printf '#!/bin/sh\nexit 0\n' >"$d/hack/e2e-capture-dataplane.sh"
1358+
chmod +x "$d/bin/timeout" "$d/bin/crust-gather" "$d/hack/e2e-capture-dataplane.sh"
1359+
yq -r '.spec.error.catch[] | select(.description == "crust-gather host snapshot on failure") | .script.content' \
1360+
"$HACK_DIR/e2e-chainsaw/.chainsaw.yaml" >"$d/catch.sh"
1361+
# The extraction found the right op: a renamed description would otherwise
1362+
# leave an empty script that satisfies nothing and reports nothing.
1363+
grep -q 'e2e-capture-dataplane.sh' "$d/catch.sh"
1364+
( cd "$d/hack/e2e-chainsaw/suite" \
1365+
&& PATH="$d/bin:$PATH" COZY_REPORT_DIR="$d/report" TEST_NAME=fixture sh "$d/catch.sh" ) \
1366+
>"$d/out" 2>&1 || true
1367+
# Positive control: the leg was reached, so the line below is about what it
1368+
# said and not about the catch having stopped earlier.
1369+
grep -q 'capturing host->pod data-plane' "$d/out"
1370+
grep -q 'data-plane capture INCOMPLETE (exit 124)' "$d/out"
1371+
rm -rf "$d"
1372+
}

‎hack/cozytest-capture-gate.bats‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,10 +112,68 @@ run_fixture() {
112112
# Positive control for the negative assertion in the last test: same fixture,
113113
# same stubs, e2e filename.
114114
grep -qF "$DATAPLANE_MARKER" "$dir/out"
115+
# The capture ran to its own end here -- the stub kubectl answers instantly --
116+
# so the incompleteness line the next test pins must NOT appear. Without this
117+
# the same line would satisfy that test whether it tracked the leg's status or
118+
# was printed unconditionally.
119+
if grep -qF 'data-plane capture INCOMPLETE' "$dir/out"; then
120+
echo "a capture that finished was reported as cut short:"
121+
cat "$dir/out"
122+
exit 1
123+
fi
115124
assert_legs_armed "$dir/report" e2e-gate-fixture
116125
rm -rf "$dir"
117126
}
118127

128+
@test "a data-plane capture cut short by its backstop says so" {
129+
# The collector names every read of its own that could not finish, which
130+
# leaves the outer backstop here as the one way its leg can still die without
131+
# a word -- and a truncated dataplane/ then reads exactly like a complete one
132+
# that found little. The sibling previous-logs leg has reported this since it
133+
# was added; this is the same discipline at the third leg.
134+
#
135+
# The backstop is simulated with a `timeout` on PATH that exits 124 rather
136+
# than by making the collector overrun a real 600s bound. The call site
137+
# cannot tell the difference: it sees a non-zero status from `timeout` and
138+
# whatever had been written before the kill, which is the whole input to the
139+
# behaviour under test.
140+
dir=$(make_fixture_dir e2e-backstop-fixture.bats)
141+
printf '#!/bin/sh\nexit 124\n' >"$dir/bin/timeout"
142+
chmod +x "$dir/bin/timeout"
143+
run_fixture "$dir" e2e-backstop-fixture.bats
144+
# Positive control: the leg was armed and reached, so the assertion below is
145+
# about what the leg said and not about the gate having skipped it.
146+
grep -qF "$DATAPLANE_MARKER" "$dir/out"
147+
grep -qF 'data-plane capture INCOMPLETE (exit 124)' "$dir/out"
148+
rm -rf "$dir"
149+
}
150+
151+
@test "a data-plane collector that is not there is not reported as a capture" {
152+
# This leg is gated on kubectl alone, while the previous-logs leg beside it
153+
# requires its collector to be executable and so does the Chainsaw caller.
154+
# The asymmetry cost nothing while the status was discarded; now that a
155+
# non-zero status prints "kept what landed in <dir>", a tree without the
156+
# collector claims a partial capture in a directory nothing ever created.
157+
dir=$(make_fixture_dir e2e-nocollector-fixture.bats)
158+
# The runner resolves both collectors relative to its own path, so copying it
159+
# somewhere without its siblings is what makes them absent -- nothing in the
160+
# worktree is touched.
161+
mkdir -p "$dir/hack"
162+
cp "$RUNNER" "$dir/hack/cozytest.sh"
163+
PATH="$dir/bin:$PATH" COZY_REPORT_DIR="$dir/report" \
164+
"$dir/hack/cozytest.sh" "$dir/e2e-nocollector-fixture.bats" >"$dir/out" 2>&1 || true
165+
# Positive controls: the fixture failed, so the handler ran, and the gate
166+
# armed -- otherwise the absence below would be satisfied by nothing running.
167+
grep -qF 'Test failed' "$dir/out"
168+
[ -f "$dir/report/snapshots/e2e-nocollector-fixture/crust-gather.log" ]
169+
if grep -qF 'data-plane capture INCOMPLETE' "$dir/out"; then
170+
echo "a collector that was never there was reported as a capture cut short:"
171+
cat "$dir/out"
172+
exit 1
173+
fi
174+
rm -rf "$dir"
175+
}
176+
119177
@test "a live cluster suite under an e2e directory keeps its captures" {
120178
# hack/e2e-apps/ holds suites whose own filenames carry no prefix. Matching
121179
# the basename alone disarms them, which takes the captures away from a suite

‎hack/cozytest.sh‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -209,12 +209,27 @@ _cozy_on_exit() {
209209
# delegates host->local-pod routing to kube-ovn/ovn0) can be root-caused
210210
# from the uploaded artifact. crust-gather captures object state but not the
211211
# node's L3 forwarding state. This NEVER affects the test outcome: every
212-
# capture is time-boxed and `|| true`, and the whole run is wrapped in a
213-
# wall-clock backstop so it cannot stall the job. It no-ops when there are no
214-
# affected pods or when kubectl/the tooling is absent.
215-
if [ "$_cozy_cluster_captures" -eq 1 ] && command -v kubectl >/dev/null 2>&1; then
212+
# capture inside is time-boxed and `|| true`, the whole run is wrapped in a
213+
# wall-clock backstop so it cannot stall the job, and the backstop's own
214+
# status is read only to print a line. It no-ops when there are no affected
215+
# pods or when kubectl/the tooling is absent.
216+
# The `-x` check matches the previous-logs leg above and the Chainsaw
217+
# caller, and it stopped being cosmetic once the status below is printed: a
218+
# tree without the collector would otherwise report "INCOMPLETE (exit 127);
219+
# kept what landed" about a directory nothing ever created.
220+
if [ "$_cozy_cluster_captures" -eq 1 ] && command -v kubectl >/dev/null 2>&1 && [ -x "$(dirname "$0")/e2e-capture-dataplane.sh" ]; then
216221
echo "» capturing host->pod data-plane for NotReady pods -> $_snap/dataplane"
217-
timeout -k 30 600 "$(dirname "$0")/e2e-capture-dataplane.sh" "$_snap/dataplane" 2>&1 || true
222+
_dp_rc=0
223+
timeout -k 30 600 "$(dirname "$0")/e2e-capture-dataplane.sh" "$_snap/dataplane" 2>&1 || _dp_rc=$?
224+
# Read, not propagated: the status still cannot change the job's outcome,
225+
# it only gets a line. The collector names every read of its own that it
226+
# could not finish, which leaves this backstop as the one remaining way
227+
# for that leg to die silently -- and a truncated dataplane/ is otherwise
228+
# indistinguishable from a complete one that found little. Same discipline
229+
# as the previous-logs leg above, whose comment carries the arithmetic.
230+
if [ "$_dp_rc" -ne 0 ]; then
231+
echo "» data-plane capture INCOMPLETE (exit $_dp_rc); kept what landed in $_snap/dataplane"
232+
fi
218233
fi
219234
fi
220235
if command -v cozy_cleanup >/dev/null 2>&1; then

‎hack/e2e-chainsaw/.chainsaw.yaml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,18 @@ spec:
188188
# op, so it keeps the wider budget.
189189
if [ -x ../../e2e-capture-dataplane.sh ]; then
190190
echo "» capturing host->pod data-plane for NotReady pods -> ${snap}/dataplane"
191-
timeout -k 30 300 ../../e2e-capture-dataplane.sh "${snap}/dataplane" 2>&1 || true
191+
timeout -k 30 300 ../../e2e-capture-dataplane.sh "${snap}/dataplane" 2>&1
192+
dp_rc=$?
193+
# Read, not propagated: the `exit 0` below still stands. The
194+
# collector names every read of its own it could not finish, so this
195+
# backstop is the last way its leg can die without a word, and a
196+
# truncated dataplane/ then reads exactly like a complete one that
197+
# found little. The 300s here cuts sooner than the 600s the
198+
# cozytest.sh call site allows, so this is the likelier of the two
199+
# to fire, and the previous-logs leg above already reports its own.
200+
if [ "${dp_rc}" -ne 0 ]; then
201+
echo "» data-plane capture INCOMPLETE (exit ${dp_rc}); kept what landed in ${snap}/dataplane"
202+
fi
192203
fi
193204
# A non-zero collector must not fail the catch and bury the test's own
194205
# error, which is the thing worth reading.

0 commit comments

Comments
 (0)