Skip to content

Commit bfaac1e

Browse files
myasnikovdaniilclaude
myasnikovdaniil
and
claude
committed
test(e2e): enrich host crust-gather snapshot with helm + collect logs
cozytest.sh's parent-shell EXIT trap snapshots the HOST cluster on a failed test (nested tenant clusters are captured separately, by run-kubernetes.sh's in-subshell trap, while their port-forward is still alive). That host snapshot had three gaps: the collect's output went to /dev/null so a failed collect left no clue why; Helm release storage (kind Secret) was dropped along with every other Secret, losing the revision history that explains a HelmRelease stuck mid-install; and the node host-log scanner burned the collect's --duration deadline on debug pods PodSecurity can never admit, forcing a non-zero exit even though the API tree was already on disk. - factor the collect into a _cozy_collect helper that captures stdout/stderr/rc to a sibling .collect.log and prints a per-target status, so a failed collect leaves evidence in the artifact instead of vanishing; bound it with `timeout -k 30 5m` so a degraded cluster can't stall the job - classify a non-zero collect that still wrote a resource tree as PARTIAL rather than FAILED, so a usable snapshot isn't discarded over the label (capture $? as the first statement in the else — after `fi` it would be the if-compound's 0, not the collect's exit code) - add a dedicated include-only collect (_cozy_collect_helm) for Helm release secrets (sh.helm.release.*), so the artifact carries each release's revision history and status - pass --disable-additional-logs so the host-log scanner doesn't burn the deadline on debug pods that can never be admitted; container/pod logs come from a separate path and are unaffected, and Talos host logs aren't reachable that way anyway crust-gather puts all CLI filters in one FilterList evaluated with .all(), so `--include-kind Secret --include-name <re>` keeps a Secret only when both match — verified from source. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 4404c5c commit bfaac1e

1 file changed

Lines changed: 54 additions & 7 deletions

File tree

‎hack/cozytest.sh‎

Lines changed: 54 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -93,20 +93,67 @@ TMP_SH=$(mktemp) || { echo "Failed to create temp file" >&2; exit 1; }
9393
# forward — so a test that creates tenant clusters (run-kubernetes.sh) captures
9494
# them from its OWN in-subshell EXIT trap, while the forward is still alive.
9595
COZY_REPORT_DIR="${COZY_REPORT_DIR:-_out/cozyreport}"
96+
97+
# Run one crust-gather collect, capturing its full output + exit code into a
98+
# sibling .collect.log. A failed collect then leaves a breadcrumb inside the
99+
# uploaded artifact instead of vanishing into /dev/null (the previous behaviour
100+
# silently dropped snapshots with no clue why). Bounded by `timeout -k 30 5m`
101+
# so a wedged/degraded cluster can't stall the whole job (-k hard-kills a
102+
# collect that ignores the SIGTERM). The caller supplies the filter flags; the
103+
# helper only owns -f, the timeout and logging.
104+
#
105+
# crust-gather exits non-zero when its --duration window closes with a scanner
106+
# still in flight, even though the API tree and pod logs are already on disk —
107+
# so a non-zero rc that still produced resources is reported as PARTIAL, not
108+
# FAILED. A 100+MB usable snapshot must not be mistaken for an empty one and
109+
# discarded over a scary label. $1=label $2=dest-dir [crust-gather collect args…].
110+
_cozy_collect() {
111+
_label=$1; _dest=$2; shift 2
112+
if timeout -k 30 5m crust-gather collect -f "$_dest" "$@" >"$_dest.collect.log" 2>&1; then
113+
echo " ✓ crust-gather $_label -> $_dest"
114+
else
115+
# First statement in the else, so $? is still the failed collect's status
116+
# (after `fi` it would be the if-compound's 0). Treat a non-zero rc that
117+
# nonetheless wrote a resource tree as PARTIAL rather than FAILED.
118+
_crc=$?
119+
if [ -d "$_dest/cluster" ] || find "$_dest/namespaces" -name '*.yaml' -print -quit 2>/dev/null | grep -q .; then
120+
echo " ⚠ crust-gather $_label PARTIAL (rc=$_crc) — $(tail -n1 "$_dest.collect.log"); see $_dest.collect.log"
121+
else
122+
echo " ✗ crust-gather $_label FAILED (rc=$_crc) — see $_dest.collect.log"
123+
fi
124+
fi
125+
}
126+
127+
# Helm stores each release as a kind=Secret object (sh.helm.release.v1.<rel>.v<n>),
128+
# so the main dump below drops it along with every other Secret. This dedicated
129+
# include-only collect captures JUST those secrets — verified AND-composition:
130+
# crust-gather puts all CLI filters in one FilterList evaluated with .all(), so
131+
# `--include-kind Secret --include-name <re>` keeps a Secret only when BOTH match.
132+
# It carries each release's revision history + status, which is what explains a
133+
# HelmRelease wedged mid-install. $1=label $2=dest-dir [extra args e.g. -k kc].
134+
_cozy_collect_helm() {
135+
_label=$1; _dest=$2; shift 2
136+
_cozy_collect "$_label" "$_dest" \
137+
--include-kind Secret --include-name 'sh\.helm\.release\..*' "$@"
138+
}
139+
96140
_cozy_on_exit() {
97141
_rc=$?
98142
if [ "$_rc" -ne 0 ]; then
99143
_snap="$COZY_REPORT_DIR/snapshots/$(basename "$TEST_FILE" .bats)"
100144
mkdir -p "$_snap" 2>/dev/null || true
101145
if command -v crust-gather >/dev/null 2>&1; then
102146
echo "» capturing crust-gather snapshot of failed $(basename "$TEST_FILE") -> $_snap"
103-
# Bound with a timeout: crust-gather collect has hung indefinitely on a
104-
# contended/degraded cluster (e.g. streaming logs from a crashlooping pod),
105-
# wedging the whole test step for hours until the job-level cancel. 5 min is
106-
# ample for a host snapshot; a partial capture (timeout exits 124, swallowed
107-
# by `|| true`) still beats a multi-hour hang. -k 30 hard-kills if a blocked
108-
# collect ignores the SIGTERM.
109-
timeout -k 30 300 crust-gather collect --exclude-kind Secret -f "$_snap/host" >/dev/null 2>&1 || true
147+
# --disable-additional-logs: skip the node host-log scanner. It spawns a
148+
# privileged busybox debug pod per node which PodSecurity rejects (403) in
149+
# cozystack's baseline/restricted namespaces, then retries until --duration
150+
# elapses — burning the deadline and forcing a non-zero exit while the
151+
# already-collected API tree + container logs sit on disk. Talos host logs
152+
# aren't reachable that way anyway, and aren't needed for app-level failures.
153+
_cozy_collect host "$_snap/host" --exclude-kind Secret --disable-additional-logs
154+
# Helm release storage is excluded by --exclude-kind Secret above; grab it
155+
# separately so a wedged HelmRelease's revision history is in the artifact.
156+
_cozy_collect_helm host-helm "$_snap/host-helm"
110157
fi
111158
# Diagnostic-only: capture the host->pod CNI data-plane state for any
112159
# NotReady pod so the recurrent host->local-pod "connection refused"

0 commit comments

Comments
 (0)