Skip to content

Commit 7be8ff4

Browse files
committed
test(e2e): capture previous container logs and stop killing the snapshot
Root-causing a crash-loop failure was blocked twice over by the diagnostics collected on failure. `kubectl logs` shows only the current container instance, so for a crash-looping pod the suites' `podLogs` collector captured the Nth restart -- a fresh replay of the same startup path -- while the evidence that explained the failure sat in the first, interrupted instance. Chainsaw's podLogs collector has no `previous` option, so add e2e-capture-previous-logs.sh and drive it from the global catch and from the BATS failure hook. It discovers containers with restartCount > 0 at runtime (init containers included, where an interrupted bootstrap lives), orders the app suites' namespace first, and dumps each to the job log and the snapshot tree. Containers that never restarted are skipped, so a healthy run emits one line rather than a page of "previous terminated container not found". Every way that capture can come up short is named rather than inferred, because kubectl exits 1 for a garbage-collected container, an RBAC denial and an unreachable node alike: the container cap, a pod list that never returned (which is not the same as nothing having restarted), a read that timed out versus one kubectl refused and the reason it gave, and a previous instance that produced no output. A partial capture is kept and marked truncated rather than deleted -- it is the only copy of a log already gone from the cluster. The crust-gather snapshot was killed by the Chainsaw op containing it. A script op is bounded by timeouts.exec (5m), and the catch's own wall-clock wrappers summed to 960s, so the script was SIGKILLed partway through -- taking out the process group mid-write and skipping the data-plane capture entirely. Give each catch op an explicit timeout above the budgets it contains, so an inner timeout fires instead and leaves a partial, still-servable snapshot behind. The data-plane leg drops from 600s to 300s at that call site so both legs fit one envelope; the BATS hook keeps the wider budget, having no containing op. Set crust-gather --duration explicitly as well. It is the collection phase's own budget, it defaults to 60s, and on elapse it abandons the collection and skips its finish step, leaving a truncated tree with no indication anything is missing. The outer wall-clock has to exceed it because the API discovery that runs first is not covered by that budget at all -- with an unhealthy aggregated APIService, the state a failed run is usually in, discovery alone can consume the whole timeout before a single object is written. Send each collector's output to a log beside its snapshot instead of /dev/null, with a complete/truncated/failed verdict, so that question is answerable from the artifact. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 28a59da commit 7be8ff4

7 files changed

Lines changed: 649 additions & 14 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,9 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
8989
- On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The `chainsaw test` run is one step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite.
9090
- Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom.
9191
- `make collect-report` → `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact.
92+
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 exposes only `container`/`tail`/`selector` and has no `previous` option) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the first, interrupted instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the BATS tests) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster.
93+
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to be killed — it reports `SCRIPT | ERROR | signal: killed` and yields nothing. That kill is strictly worse than an inner `timeout` firing: SIGKILL takes out the process group, so the collector dies mid-write **and** every later step in the script is skipped, whereas an inner `timeout` exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
94+
- **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact.
9295
- **Per-failed-test crust-gather snapshots** are the richest diagnostic — a `crust-gather serve`-able archive of the cluster *at the moment of failure*, before cleanup. The global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` captures the **host** cluster into `_out/cozyreport/snapshots/<test>/host` on any failure (the Chainsaw analog of the `hack/cozytest.sh` EXIT-trap snapshot used by the BATS tests); the kubernetes suites additionally snapshot their **tenant** cluster from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` while the tenant API LB is still routable. `hack/cozyreport.sh` folds `snapshots/` into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state.
9396

9497
### 9. Keep the test environment deterministic

‎hack/capture-previous-logs.bats‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for the pure selection helpers in
4+
# hack/e2e-capture-previous-logs.sh -- the logic that decides WHICH restarted
5+
# containers get their previous instance dumped, and in what order:
6+
#
7+
# - prevlog_filter_restarted -- keep only containers with restartCount > 0;
8+
# - prevlog_prioritize -- put the failing test's namespace first;
9+
# - prevlog_cap -- bound the dump so a wedged cluster cannot
10+
# explode the catch;
11+
# - prevlog_logfile_name -- per-container artifact filename.
12+
#
13+
# The `kubectl logs --previous` capture itself is not unit-testable (it needs a
14+
# live cluster with a crash-looping pod); these tests pin the derivations that
15+
# decide what is worth asking for at all. The restart filter is the one that
16+
# matters most: without it every healthy container would be asked for a
17+
# previous instance that does not exist, turning the catch into a page of noise.
18+
#
19+
# Strategy: the script is sourced once with E2E_CAPTURE_PREVLOGS_LIB set, which
20+
# the script's sourcing guard honours by defining the helpers and returning
21+
# before it touches $1 or runs any capture -- so no cluster is required and the
22+
# capture body never executes. Each @test then calls the helpers directly and
23+
# asserts with `[ ... ]`, matching this repo's plain-shell bats convention (no
24+
# `run` helper).
25+
#
26+
# Title syntax constraints (inherited from cozytest.sh's awk parser):
27+
# - Titles delimited by ASCII double quotes; embedded quotes truncate.
28+
# - Only [A-Za-z0-9] from the title survives into the function name, so keep
29+
# titles distinctive in their alphanumeric run.
30+
#
31+
# Run with: hack/cozytest.sh hack/capture-previous-logs.bats
32+
# (or `bats hack/capture-previous-logs.bats` if the bats binary is
33+
# installed; cozytest.sh is the CI path.)
34+
# -----------------------------------------------------------------------------
35+
36+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
37+
SCRIPT="$HACK_DIR/e2e-capture-previous-logs.sh"
38+
39+
# Load the pure helpers. The guard returns before the capture body, so this is
40+
# side-effect-free and needs no cluster.
41+
E2E_CAPTURE_PREVLOGS_LIB=1
42+
# shellcheck source=/dev/null
43+
. "$SCRIPT"
44+
45+
@test "filter restarted keeps only containers whose restart count is above zero" {
46+
rows="$(printf '%s\n' \
47+
'tenant-test|mariadb-test-0|mariadb|container|3' \
48+
'tenant-test|mariadb-test-1|mariadb|container|0' \
49+
'cozy-system|healthy|app|container|0' \
50+
'tenant-test|mariadb-test-0|init-datadir|init|2')"
51+
52+
out="$(printf '%s\n' "$rows" | prevlog_filter_restarted)"
53+
54+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 2 ]
55+
printf '%s\n' "$out" | grep -q '^tenant-test|mariadb-test-0|mariadb|container|3$'
56+
printf '%s\n' "$out" | grep -q '^tenant-test|mariadb-test-0|init-datadir|init|2$'
57+
# `! cmd` is vacuous under cozytest's `set -e` (errexit is suppressed for a
58+
# `!`-negated pipeline), so a filter regression that let these rows through
59+
# would not fail the test. Assert the absence via `if cmd; then ...; false`.
60+
if printf '%s\n' "$out" | grep -q 'mariadb-test-1'; then echo "FAIL: must drop the zero-restart replica"; false; fi
61+
if printf '%s\n' "$out" | grep -q 'cozy-system'; then echo "FAIL: must drop the zero-restart system pod"; false; fi
62+
}
63+
64+
@test "filter restarted drops rows with a malformed or empty restart count" {
65+
# A non-numeric count must be dropped rather than compared. The row would be
66+
# skipped either way (`[ "<none>" -gt 0 ]` exits 2, which the filter's
67+
# `|| continue` catches), so what this pins is the contract -- malformed rows
68+
# never reach the capture -- not a crash.
69+
rows="$(printf '%s\n' \
70+
'tenant-test|pod-a|app|container|' \
71+
'tenant-test|pod-b|app|container|<none>' \
72+
'tenant-test|pod-c|app|container|-1' \
73+
'tenant-test|pod-d|app|container|1')"
74+
75+
out="$(printf '%s\n' "$rows" | prevlog_filter_restarted)"
76+
77+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 1 ]
78+
printf '%s\n' "$out" | grep -q '^tenant-test|pod-d|'
79+
}
80+
81+
@test "filter restarted drops rows missing a namespace pod or container field" {
82+
rows="$(printf '%s\n' \
83+
'|pod-a|app|container|4' \
84+
'tenant-test||app|container|4' \
85+
'tenant-test|pod-c||container|4' \
86+
'tenant-test|pod-d|app|container|4')"
87+
88+
out="$(printf '%s\n' "$rows" | prevlog_filter_restarted)"
89+
90+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 1 ]
91+
printf '%s\n' "$out" | grep -q '^tenant-test|pod-d|'
92+
}
93+
94+
@test "prioritize puts the failing test namespace ahead of everything else" {
95+
rows="$(printf '%s\n' \
96+
'cozy-linstor|satellite-x|linstor|container|5' \
97+
'tenant-test|mariadb-test-0|mariadb|container|3' \
98+
'cozy-cilium|agent-y|cilium|container|2' \
99+
'tenant-test|mariadb-test-1|mariadb|container|1')"
100+
101+
out="$(printf '%s\n' "$rows" | prevlog_prioritize tenant-test)"
102+
103+
# All four survive -- prioritisation reorders, it never drops.
104+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 4 ]
105+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f2)" = "mariadb-test-0" ]
106+
[ "$(printf '%s\n' "$out" | sed -n '2p' | cut -d'|' -f2)" = "mariadb-test-1" ]
107+
# Non-matching rows keep their relative input order behind the matches.
108+
[ "$(printf '%s\n' "$out" | sed -n '3p' | cut -d'|' -f2)" = "satellite-x" ]
109+
[ "$(printf '%s\n' "$out" | sed -n '4p' | cut -d'|' -f2)" = "agent-y" ]
110+
}
111+
112+
@test "prioritize with an empty namespace passes every row through unchanged" {
113+
rows="$(printf '%s\n' \
114+
'cozy-linstor|satellite-x|linstor|container|5' \
115+
'tenant-test|mariadb-test-0|mariadb|container|3')"
116+
117+
out="$(printf '%s\n' "$rows" | prevlog_prioritize '')"
118+
119+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 2 ]
120+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f2)" = "satellite-x" ]
121+
}
122+
123+
@test "prioritize matches the namespace field only and not a pod name substring" {
124+
# The match must anchor on the LEADING namespace field. The first row is the
125+
# fixture that makes this test bite: a pod named exactly like the namespace,
126+
# in a different namespace, so the row contains `tenant-test|` but does not
127+
# start with it. Drop the `^` from prevlog_prioritize and that row is promoted
128+
# ahead of the genuine namespace match below, flipping the first two
129+
# assertions. (A pod merely named `tenant-test-runner` does NOT distinguish
130+
# the two -- it lacks the trailing `|` and so matches neither form -- which is
131+
# why it is kept only as the third row rather than carrying the test.)
132+
# The second row is the other half of the anchor: a SIBLING NAMESPACE sharing
133+
# the prefix, which nested tenants make routine. It pins the trailing `|` --
134+
# drop that from the pattern and `tenant-test-child` is treated as the failing
135+
# test's own namespace and promoted, which is how the cap would get spent on
136+
# an unrelated tenant's restarts.
137+
rows="$(printf '%s\n' \
138+
'cozy-system|tenant-test|app|container|1' \
139+
'tenant-test-child|neighbour|app|container|1' \
140+
'cozy-system|tenant-test-runner|app|container|1' \
141+
'tenant-test|real|app|container|1')"
142+
143+
out="$(printf '%s\n' "$rows" | prevlog_prioritize tenant-test)"
144+
145+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 4 ]
146+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f1)" = "tenant-test" ]
147+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f2)" = "real" ]
148+
# Everything else keeps input order behind the single genuine match.
149+
[ "$(printf '%s\n' "$out" | sed -n '2p' | cut -d'|' -f2)" = "tenant-test" ]
150+
[ "$(printf '%s\n' "$out" | sed -n '3p' | cut -d'|' -f2)" = "neighbour" ]
151+
[ "$(printf '%s\n' "$out" | sed -n '4p' | cut -d'|' -f2)" = "tenant-test-runner" ]
152+
}
153+
154+
@test "cap keeps at most the requested number of rows" {
155+
rows="$(printf '%s\n' \
156+
'ns|p1|c|container|1' 'ns|p2|c|container|1' 'ns|p3|c|container|1' \
157+
'ns|p4|c|container|1' 'ns|p5|c|container|1')"
158+
159+
out="$(printf '%s\n' "$rows" | prevlog_cap 3)"
160+
161+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 3 ]
162+
printf '%s\n' "$out" | grep -q '^ns|p1|'
163+
if printf '%s\n' "$out" | grep -q '^ns|p4|'; then echo "FAIL: cap must drop rows past the limit"; false; fi
164+
}
165+
166+
@test "cap leaves a shorter input untouched" {
167+
rows="$(printf '%s\n' 'ns|p1|c|container|1' 'ns|p2|c|container|1')"
168+
169+
out="$(printf '%s\n' "$rows" | prevlog_cap 12)"
170+
171+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 2 ]
172+
}
173+
174+
@test "cap falls back to the default limit when given a malformed value" {
175+
# COZY_PREVLOG_MAX is operator-supplied, so a typo must degrade to the default
176+
# rather than make `head -n` fail and drop the whole capture.
177+
rows="$(i=1; while [ "$i" -le 20 ]; do printf 'ns|p%s|c|container|1\n' "$i"; i=$((i + 1)); done)"
178+
[ "$(printf '%s\n' "$rows" | grep -c .)" -eq 20 ]
179+
180+
out="$(printf '%s\n' "$rows" | prevlog_cap 'twelve')"
181+
182+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 12 ]
183+
}
184+
185+
@test "logfile name joins namespace pod and container with underscores" {
186+
name="$(prevlog_logfile_name tenant-test mariadb-test-0 mariadb)"
187+
188+
[ "$name" = "tenant-test_mariadb-test-0_mariadb.log" ]
189+
}

‎hack/cozytest.sh‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,32 @@ _cozy_on_exit() {
9898
if [ "$_rc" -ne 0 ]; then
9999
_snap="$COZY_REPORT_DIR/snapshots/$(basename "$TEST_FILE" .bats)"
100100
mkdir -p "$_snap" 2>/dev/null || true
101+
# Previous-instance container logs for anything that crash-looped. `kubectl
102+
# logs` shows only the current instance, so for a crash-looping pod the
103+
# decisive evidence — the first, interrupted run — is reachable only via
104+
# `--previous` and is lost once the kubelet garbage-collects that container.
105+
# FIRST, ahead of both captures below: it is the cheapest leg and the only
106+
# one whose subject is perishable. The kubelet retains one dead container
107+
# per pod by default, so each further restart of a still-looping pod drops
108+
# the instance we are here to read — and the snapshot leg alone can hold the
109+
# trap for 390s. The other two captures read state that keeps.
110+
if command -v kubectl >/dev/null 2>&1 && [ -x "$(dirname "$0")/e2e-capture-previous-logs.sh" ]; then
111+
_prev_rc=0
112+
timeout -k 30 300 "$(dirname "$0")/e2e-capture-previous-logs.sh" \
113+
"$_snap/previous-logs" "${COZY_TEST_NAMESPACE:-tenant-test}" 2>&1 || _prev_rc=$?
114+
# A backstop that fires must say so, matching the crust-gather leg below.
115+
# 300s sits above the ~270s worst case inside (a ~30s pod list plus up to
116+
# 12 × 20s log reads), so it should not fire; if it does, a silently cut
117+
# capture would read as a complete one listing fewer crash-loops than
118+
# actually occurred. This handler's whole worst case is ~1350s (330 + 390
119+
# + 630, each leg including its 30s kill grace) against a job capped at
120+
# 180 minutes with no per-step timeout —
121+
# deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml:
122+
# a bounded, honest wait beats a kill that collects nothing.
123+
if [ "$_prev_rc" -ne 0 ]; then
124+
echo "» previous-instance capture INCOMPLETE (exit $_prev_rc); kept what landed in $_snap/previous-logs"
125+
fi
126+
fi
101127
if command -v crust-gather >/dev/null 2>&1; then
102128
echo "» capturing crust-gather snapshot of failed $(basename "$TEST_FILE") -> $_snap"
103129
# Bound with a timeout: crust-gather collect has hung indefinitely on a
@@ -106,7 +132,26 @@ _cozy_on_exit() {
106132
# ample for a host snapshot; a partial capture (timeout exits 124, swallowed
107133
# by `|| true`) still beats a multi-hour hang. -k 30 hard-kills if a blocked
108134
# collect ignores the SIGTERM.
109-
timeout -k 30 300 crust-gather collect --exclude-kind Secret -f "$_snap/host" >/dev/null 2>&1 || true
135+
# --duration is crust-gather's own budget for the collection phase and
136+
# defaults to 60s, which silently truncates a snapshot of a cluster this
137+
# size: on elapse it abandons the collection and skips its finish step,
138+
# leaving a partial tree with no indication that the rest is missing. Set
139+
# it explicitly. The outer wall-clock must exceed it because crust-gather
140+
# runs API discovery BEFORE that timer starts and that phase is unbounded
141+
# — with an unhealthy aggregated APIService (the state a failed run is
142+
# usually in) discovery alone can eat the whole budget before a single
143+
# object is written.
144+
# Output goes to a log inside the snapshot instead of /dev/null so
145+
# "is this snapshot complete or truncated?" is answerable from the
146+
# uploaded artifact rather than guessed, matching the Chainsaw catch.
147+
_cg_rc=0
148+
timeout -k 30 360 crust-gather collect --duration 180s \
149+
--exclude-kind Secret -f "$_snap/host" >"$_snap/crust-gather.log" 2>&1 || _cg_rc=$?
150+
case "$_cg_rc" in
151+
0) echo "» crust-gather host snapshot complete" ;;
152+
124 | 137) echo "» crust-gather host snapshot TRUNCATED (wall-clock $_cg_rc); partial state kept, see $_snap/crust-gather.log" ;;
153+
*) echo "» crust-gather host snapshot FAILED (exit $_cg_rc); see $_snap/crust-gather.log" ;;
154+
esac
110155
fi
111156
# Diagnostic-only: capture the host->pod CNI data-plane state for any
112157
# NotReady pod so the recurrent host->local-pod "connection refused"

0 commit comments

Comments
 (0)