Skip to content

Commit 55195de

Browse files
test(e2e): capture previous container logs and stop killing the snapshot (#3346)
## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents d5806eb + f3a0f2d commit 55195de

7 files changed

Lines changed: 656 additions & 14 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,9 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
8989
- On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The `chainsaw test` run is one step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite.
9090
- Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom.
9191
- `make collect-report` → `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact.
92+
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 exposes only `container`/`tail`/`selector` and has no `previous` option) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the BATS tests) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster.
93+
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to be killed — it reports `SCRIPT | ERROR | signal: killed` and yields nothing. That kill is strictly worse than an inner `timeout` firing: SIGKILL takes out the process group, so the collector dies mid-write **and** every later step in the script is skipped, whereas an inner `timeout` exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
94+
- **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact.
9295
- **Per-failed-test crust-gather snapshots** are the richest diagnostic — a `crust-gather serve`-able archive of the cluster *at the moment of failure*, before cleanup. The global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` captures the **host** cluster into `_out/cozyreport/snapshots/<test>/host` on any failure (the Chainsaw analog of the `hack/cozytest.sh` EXIT-trap snapshot used by the BATS tests); the kubernetes suites additionally snapshot their **tenant** cluster from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` while the tenant API LB is still routable. `hack/cozyreport.sh` folds `snapshots/` into the artifact. Do not remove the global catch — without it the Chainsaw suite uploads a report with no per-test cluster state.
9396

9497
### 9. Keep the test environment deterministic

‎hack/capture-previous-logs.bats‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for the pure selection helpers in
4+
# hack/e2e-capture-previous-logs.sh -- the logic that decides WHICH restarted
5+
# containers get their previous instance dumped, and in what order:
6+
#
7+
# - prevlog_filter_restarted -- keep only containers with restartCount > 0;
8+
# - prevlog_prioritize -- put the failing test's namespace first;
9+
# - prevlog_cap -- bound the dump so a wedged cluster cannot
10+
# explode the catch;
11+
# - prevlog_logfile_name -- per-container artifact filename.
12+
#
13+
# The `kubectl logs --previous` capture itself is not unit-testable (it needs a
14+
# live cluster with a crash-looping pod); these tests pin the derivations that
15+
# decide what is worth asking for at all. The restart filter is the one that
16+
# matters most: without it every healthy container would be asked for a
17+
# previous instance that does not exist, turning the catch into a page of noise.
18+
#
19+
# Strategy: the script is sourced once with E2E_CAPTURE_PREVLOGS_LIB set, which
20+
# the script's sourcing guard honours by defining the helpers and returning
21+
# before it touches $1 or runs any capture -- so no cluster is required and the
22+
# capture body never executes. Each @test then calls the helpers directly and
23+
# asserts with `[ ... ]`, matching this repo's plain-shell bats convention (no
24+
# `run` helper).
25+
#
26+
# Title syntax constraints (inherited from cozytest.sh's awk parser):
27+
# - Titles delimited by ASCII double quotes; embedded quotes truncate.
28+
# - Only [A-Za-z0-9] from the title survives into the function name, so keep
29+
# titles distinctive in their alphanumeric run.
30+
#
31+
# Run with: hack/cozytest.sh hack/capture-previous-logs.bats
32+
# (or `bats hack/capture-previous-logs.bats` if the bats binary is
33+
# installed; cozytest.sh is the CI path.)
34+
# -----------------------------------------------------------------------------
35+
36+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
37+
SCRIPT="$HACK_DIR/e2e-capture-previous-logs.sh"
38+
39+
# Load the pure helpers. The guard returns before the capture body, so this is
40+
# side-effect-free and needs no cluster.
41+
E2E_CAPTURE_PREVLOGS_LIB=1
42+
# shellcheck source=/dev/null
43+
. "$SCRIPT"
44+
45+
@test "filter restarted keeps only containers whose restart count is above zero" {
46+
rows="$(printf '%s\n' \
47+
'tenant-test|mariadb-test-0|mariadb|container|3' \
48+
'tenant-test|mariadb-test-1|mariadb|container|0' \
49+
'cozy-system|healthy|app|container|0' \
50+
'tenant-test|mariadb-test-0|init-datadir|init|2')"
51+
52+
out="$(printf '%s\n' "$rows" | prevlog_filter_restarted)"
53+
54+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 2 ]
55+
printf '%s\n' "$out" | grep -q '^tenant-test|mariadb-test-0|mariadb|container|3$'
56+
printf '%s\n' "$out" | grep -q '^tenant-test|mariadb-test-0|init-datadir|init|2$'
57+
# `! cmd` is vacuous under cozytest's `set -e` (errexit is suppressed for a
58+
# `!`-negated pipeline), so a filter regression that let these rows through
59+
# would not fail the test. Assert the absence via `if cmd; then ...; false`.
60+
if printf '%s\n' "$out" | grep -q 'mariadb-test-1'; then echo "FAIL: must drop the zero-restart replica"; false; fi
61+
if printf '%s\n' "$out" | grep -q 'cozy-system'; then echo "FAIL: must drop the zero-restart system pod"; false; fi
62+
}
63+
64+
@test "filter restarted drops rows with a malformed or empty restart count" {
65+
# A non-numeric count must be dropped rather than compared. The row would be
66+
# skipped either way (`[ "<none>" -gt 0 ]` exits 2, which the filter's
67+
# `|| continue` catches), so what this pins is the contract -- malformed rows
68+
# never reach the capture -- not a crash.
69+
rows="$(printf '%s\n' \
70+
'tenant-test|pod-a|app|container|' \
71+
'tenant-test|pod-b|app|container|<none>' \
72+
'tenant-test|pod-c|app|container|-1' \
73+
'tenant-test|pod-d|app|container|1')"
74+
75+
out="$(printf '%s\n' "$rows" | prevlog_filter_restarted)"
76+
77+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 1 ]
78+
printf '%s\n' "$out" | grep -q '^tenant-test|pod-d|'
79+
}
80+
81+
@test "filter restarted drops rows missing a namespace pod or container field" {
82+
rows="$(printf '%s\n' \
83+
'|pod-a|app|container|4' \
84+
'tenant-test||app|container|4' \
85+
'tenant-test|pod-c||container|4' \
86+
'tenant-test|pod-d|app|container|4')"
87+
88+
out="$(printf '%s\n' "$rows" | prevlog_filter_restarted)"
89+
90+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 1 ]
91+
printf '%s\n' "$out" | grep -q '^tenant-test|pod-d|'
92+
}
93+
94+
@test "prioritize puts the failing test namespace ahead of everything else" {
95+
rows="$(printf '%s\n' \
96+
'cozy-linstor|satellite-x|linstor|container|5' \
97+
'tenant-test|mariadb-test-0|mariadb|container|3' \
98+
'cozy-cilium|agent-y|cilium|container|2' \
99+
'tenant-test|mariadb-test-1|mariadb|container|1')"
100+
101+
out="$(printf '%s\n' "$rows" | prevlog_prioritize tenant-test)"
102+
103+
# All four survive -- prioritisation reorders, it never drops.
104+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 4 ]
105+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f2)" = "mariadb-test-0" ]
106+
[ "$(printf '%s\n' "$out" | sed -n '2p' | cut -d'|' -f2)" = "mariadb-test-1" ]
107+
# Non-matching rows keep their relative input order behind the matches.
108+
[ "$(printf '%s\n' "$out" | sed -n '3p' | cut -d'|' -f2)" = "satellite-x" ]
109+
[ "$(printf '%s\n' "$out" | sed -n '4p' | cut -d'|' -f2)" = "agent-y" ]
110+
}
111+
112+
@test "prioritize with an empty namespace passes every row through unchanged" {
113+
rows="$(printf '%s\n' \
114+
'cozy-linstor|satellite-x|linstor|container|5' \
115+
'tenant-test|mariadb-test-0|mariadb|container|3')"
116+
117+
out="$(printf '%s\n' "$rows" | prevlog_prioritize '')"
118+
119+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 2 ]
120+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f2)" = "satellite-x" ]
121+
}
122+
123+
@test "prioritize matches the namespace field only and not a pod name substring" {
124+
# The match must anchor on the LEADING namespace field. The first row is the
125+
# fixture that makes this test bite: a pod named exactly like the namespace,
126+
# in a different namespace, so the row contains `tenant-test|` but does not
127+
# start with it. Drop the `^` from prevlog_prioritize and that row is promoted
128+
# ahead of the genuine namespace match below, flipping the first two
129+
# assertions. (A pod merely named `tenant-test-runner` does NOT distinguish
130+
# the two -- it lacks the trailing `|` and so matches neither form -- which is
131+
# why it is kept only as the third row rather than carrying the test.)
132+
# The second row is the other half of the anchor: a SIBLING NAMESPACE sharing
133+
# the prefix, which nested tenants make routine. It pins the trailing `|` --
134+
# drop that from the pattern and `tenant-test-child` is treated as the failing
135+
# test's own namespace and promoted, which is how the cap would get spent on
136+
# an unrelated tenant's restarts.
137+
rows="$(printf '%s\n' \
138+
'cozy-system|tenant-test|app|container|1' \
139+
'tenant-test-child|neighbour|app|container|1' \
140+
'cozy-system|tenant-test-runner|app|container|1' \
141+
'tenant-test|real|app|container|1')"
142+
143+
out="$(printf '%s\n' "$rows" | prevlog_prioritize tenant-test)"
144+
145+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 4 ]
146+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f1)" = "tenant-test" ]
147+
[ "$(printf '%s\n' "$out" | sed -n '1p' | cut -d'|' -f2)" = "real" ]
148+
# Everything else keeps input order behind the single genuine match.
149+
[ "$(printf '%s\n' "$out" | sed -n '2p' | cut -d'|' -f2)" = "tenant-test" ]
150+
[ "$(printf '%s\n' "$out" | sed -n '3p' | cut -d'|' -f2)" = "neighbour" ]
151+
[ "$(printf '%s\n' "$out" | sed -n '4p' | cut -d'|' -f2)" = "tenant-test-runner" ]
152+
}
153+
154+
@test "cap keeps at most the requested number of rows" {
155+
rows="$(printf '%s\n' \
156+
'ns|p1|c|container|1' 'ns|p2|c|container|1' 'ns|p3|c|container|1' \
157+
'ns|p4|c|container|1' 'ns|p5|c|container|1')"
158+
159+
out="$(printf '%s\n' "$rows" | prevlog_cap 3)"
160+
161+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 3 ]
162+
printf '%s\n' "$out" | grep -q '^ns|p1|'
163+
if printf '%s\n' "$out" | grep -q '^ns|p4|'; then echo "FAIL: cap must drop rows past the limit"; false; fi
164+
}
165+
166+
@test "cap leaves a shorter input untouched" {
167+
rows="$(printf '%s\n' 'ns|p1|c|container|1' 'ns|p2|c|container|1')"
168+
169+
out="$(printf '%s\n' "$rows" | prevlog_cap 12)"
170+
171+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 2 ]
172+
}
173+
174+
@test "cap falls back to the default limit when given a malformed value" {
175+
# COZY_PREVLOG_MAX is operator-supplied, so a typo must degrade to the default
176+
# rather than make `head -n` fail and drop the whole capture.
177+
rows="$(i=1; while [ "$i" -le 20 ]; do printf 'ns|p%s|c|container|1\n' "$i"; i=$((i + 1)); done)"
178+
[ "$(printf '%s\n' "$rows" | grep -c .)" -eq 20 ]
179+
180+
out="$(printf '%s\n' "$rows" | prevlog_cap 'twelve')"
181+
182+
[ "$(printf '%s\n' "$out" | grep -c .)" -eq 12 ]
183+
}
184+
185+
@test "logfile name joins namespace pod and container with underscores" {
186+
name="$(prevlog_logfile_name tenant-test mariadb-test-0 mariadb)"
187+
188+
[ "$name" = "tenant-test_mariadb-test-0_mariadb.log" ]
189+
}

‎hack/cozytest.sh‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,32 @@ _cozy_on_exit() {
9898
if [ "$_rc" -ne 0 ]; then
9999
_snap="$COZY_REPORT_DIR/snapshots/$(basename "$TEST_FILE" .bats)"
100100
mkdir -p "$_snap" 2>/dev/null || true
101+
# Previous-instance container logs for anything that crash-looped. `kubectl
102+
# logs` shows only the current instance, so for a crash-looping pod the
103+
# decisive evidence — the immediately preceding run — is reachable only via
104+
# `--previous` and is lost once the kubelet garbage-collects that container.
105+
# FIRST, ahead of both captures below: it is the cheapest leg and the only
106+
# one whose subject is perishable. The kubelet retains one dead container
107+
# per pod by default, so each further restart of a still-looping pod drops
108+
# the instance we are here to read — and the snapshot leg alone can hold the
109+
# trap for 390s. The other two captures read state that keeps.
110+
if command -v kubectl >/dev/null 2>&1 && [ -x "$(dirname "$0")/e2e-capture-previous-logs.sh" ]; then
111+
_prev_rc=0
112+
timeout -k 30 300 "$(dirname "$0")/e2e-capture-previous-logs.sh" \
113+
"$_snap/previous-logs" "${COZY_TEST_NAMESPACE:-tenant-test}" 2>&1 || _prev_rc=$?
114+
# A backstop that fires must say so, matching the crust-gather leg below.
115+
# 300s sits above the ~270s worst case inside (a ~30s pod list plus up to
116+
# 12 × 20s log reads), so it should not fire; if it does, a silently cut
117+
# capture would read as a complete one listing fewer crash-loops than
118+
# actually occurred. This handler's whole worst case is ~1350s (330 + 390
119+
# + 630, each leg including its 30s kill grace) against a job capped at
120+
# 180 minutes with no per-step timeout —
121+
# deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml:
122+
# a bounded, honest wait beats a kill that collects nothing.
123+
if [ "$_prev_rc" -ne 0 ]; then
124+
echo "» previous-instance capture INCOMPLETE (exit $_prev_rc); kept what landed in $_snap/previous-logs"
125+
fi
126+
fi
101127
if command -v crust-gather >/dev/null 2>&1; then
102128
echo "» capturing crust-gather snapshot of failed $(basename "$TEST_FILE") -> $_snap"
103129
# Bound with a timeout: crust-gather collect has hung indefinitely on a
@@ -106,7 +132,26 @@ _cozy_on_exit() {
106132
# ample for a host snapshot; a partial capture (timeout exits 124, swallowed
107133
# by `|| true`) still beats a multi-hour hang. -k 30 hard-kills if a blocked
108134
# collect ignores the SIGTERM.
109-
timeout -k 30 300 crust-gather collect --exclude-kind Secret -f "$_snap/host" >/dev/null 2>&1 || true
135+
# --duration is crust-gather's own budget for the collection phase and
136+
# defaults to 60s, which silently truncates a snapshot of a cluster this
137+
# size: on elapse it abandons the collection and skips its finish step,
138+
# leaving a partial tree with no indication that the rest is missing. Set
139+
# it explicitly. The outer wall-clock must exceed it because crust-gather
140+
# runs API discovery BEFORE that timer starts and that phase is unbounded
141+
# — with an unhealthy aggregated APIService (the state a failed run is
142+
# usually in) discovery alone can eat the whole budget before a single
143+
# object is written.
144+
# Output goes to a log inside the snapshot instead of /dev/null so
145+
# "is this snapshot complete or truncated?" is answerable from the
146+
# uploaded artifact rather than guessed, matching the Chainsaw catch.
147+
_cg_rc=0
148+
timeout -k 30 360 crust-gather collect --duration 180s \
149+
--exclude-kind Secret -f "$_snap/host" >"$_snap/crust-gather.log" 2>&1 || _cg_rc=$?
150+
case "$_cg_rc" in
151+
0) echo "» crust-gather host snapshot complete" ;;
152+
124 | 137) echo "» crust-gather host snapshot TRUNCATED (wall-clock $_cg_rc); partial state kept, see $_snap/crust-gather.log" ;;
153+
*) echo "» crust-gather host snapshot FAILED (exit $_cg_rc); see $_snap/crust-gather.log" ;;
154+
esac
110155
fi
111156
# Diagnostic-only: capture the host->pod CNI data-plane state for any
112157
# NotReady pod so the recurrent host->local-pod "connection refused"

0 commit comments

Comments
 (0)