Skip to content

Commit a234a61

Browse files
test(e2e): make crust-gather failure snapshots reliable and complete
The per-test crust-gather snapshot silently dropped the tenant-cluster dump: the collect's output went to /dev/null, the tenant kubeconfig's localhost port-forward is usually dead by the time the EXIT trap runs, and Helm release storage (kind Secret) was excluded along with every other Secret. The dump that mattered most for a tenant HelmRelease stuck mid-install was therefore missing, with no clue why. - capture each collect's stdout/stderr/rc to a sibling .collect.log and print a per-target status, so a failed collect leaves evidence in the artifact instead of vanishing - probe the tenant API and re-establish the port-forward from the kubeconfig (name + server port) before collecting, so the tenant dump is actually reachable - bound every collect with `timeout 5m` so a wedged forward can't stall the job - add a dedicated include-only collect for Helm release secrets (sh.helm.release.*) for host and each tenant, so the artifact carries each release's revision history and status crust-gather v0.15.2 puts all CLI filters in one FilterList evaluated with .all(), so `--include-kind Secret --include-name <re>` captures only secrets matching both — verified from source. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 770ce36 commit a234a61

1 file changed

Lines changed: 61 additions & 2 deletions

File tree

‎hack/cozytest.sh‎

Lines changed: 61 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,16 +88,75 @@ TMP_SH=$(mktemp) || { echo "Failed to create temp file" >&2; exit 1; }
8888
# cozy_cleanup is a plain shell function a .bats file may define — there are no
8989
# bats setup/teardown directives here, this runner only knows @test + bash.
9090
COZY_REPORT_DIR="${COZY_REPORT_DIR:-_out/cozyreport}"
91+
# Host-cluster namespace the e2e tenant Kubernetes control planes provision into.
92+
# Only the kubernetes-*.bats files emit tenantkubeconfig-* files and they all use
93+
# this namespace; it's the service location used to re-forward to a tenant below.
94+
COZY_TENANT_NS="${COZY_TENANT_NS:-tenant-test}"
95+
96+
# Run one crust-gather collect, capturing its full output + exit code into a
97+
# sibling .collect.log. A failed collect then leaves a breadcrumb inside the
98+
# uploaded artifact instead of vanishing into /dev/null (the previous behaviour
99+
# silently dropped tenant snapshots with no clue why). Bounded by `timeout` so a
100+
# wedged port-forward can't stall the whole job. The caller supplies the filter
101+
# flags (and -k for tenants); the helper only owns -f, the timeout and logging.
102+
# $1=label $2=dest-dir [crust-gather collect args…].
103+
_cozy_collect() {
104+
_label=$1; _dest=$2; shift 2
105+
if timeout 5m crust-gather collect -f "$_dest" "$@" >"$_dest.collect.log" 2>&1; then
106+
echo " ✓ crust-gather $_label -> $_dest"
107+
else
108+
echo " ✗ crust-gather $_label FAILED (rc=$?) — see $_dest.collect.log"
109+
fi
110+
}
111+
112+
# Helm stores each release as a kind=Secret object (sh.helm.release.v1.<rel>.v<n>),
113+
# so the main dumps below drop it along with every other Secret. This dedicated
114+
# include-only collect captures JUST those secrets — verified AND-composition:
115+
# crust-gather puts all CLI filters in one FilterList evaluated with .all(), so
116+
# `--include-kind Secret --include-name <re>` keeps a Secret only when BOTH match.
117+
# It carries each release's revision history + status, which is what explains a
118+
# HelmRelease wedged mid-install. $1=label $2=dest-dir [extra args e.g. -k kc].
119+
_cozy_collect_helm() {
120+
_label=$1; _dest=$2; shift 2
121+
_cozy_collect "$_label" "$_dest" \
122+
--include-kind Secret --include-name 'sh\.helm\.release\..*' "$@"
123+
}
124+
91125
_cozy_on_exit() {
92126
_rc=$?
93127
if [ "$_rc" -ne 0 ] && command -v crust-gather >/dev/null 2>&1; then
94128
_snap="$COZY_REPORT_DIR/snapshots/$(basename "$TEST_FILE" .bats)"
95129
mkdir -p "$_snap" 2>/dev/null || true
96130
echo "» capturing crust-gather snapshot of failed $(basename "$TEST_FILE") -> $_snap"
97-
crust-gather collect --exclude-kind Secret -f "$_snap/host" >/dev/null 2>&1 || true
131+
_cozy_collect host "$_snap/host" --exclude-kind Secret
132+
_cozy_collect_helm host-helm "$_snap/host-helm"
98133
for _kc in tenantkubeconfig-*; do
99134
[ -f "$_kc" ] || continue
100-
crust-gather collect -k "$_kc" --exclude-kind Secret -f "$_snap/$_kc" >/dev/null 2>&1 || true
135+
# The tenant kubeconfig points at https://localhost:<port>, served by a
136+
# `kubectl port-forward` the test backgrounded many minutes earlier. That
137+
# forward is frequently dead by the time this trap runs, so crust-gather
138+
# can't reach the tenant API and the most-wanted dump goes missing. Probe
139+
# /readyz; if unreachable, re-establish the forward from the kubeconfig
140+
# (file name carries the cluster name, server URL carries the local port).
141+
_name=${_kc#tenantkubeconfig-}
142+
_srv=$(kubectl --kubeconfig "$_kc" config view -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null || true)
143+
_port=${_srv##*:}
144+
_pf_pid=
145+
if ! kubectl --kubeconfig "$_kc" --request-timeout=8s get --raw='/readyz' >/dev/null 2>&1; then
146+
case "$_port" in
147+
''|*[!0-9]*) echo " ! $_kc unreachable, no numeric local port to re-forward (server=$_srv)" ;;
148+
*)
149+
echo " ! $_kc unreachable; re-forwarding service/kubernetes-$_name -n $COZY_TENANT_NS $_port:6443"
150+
pkill -f "port-forward.*${_port}:" 2>/dev/null || true
151+
kubectl port-forward "service/kubernetes-${_name}" -n "$COZY_TENANT_NS" "${_port}:6443" >/dev/null 2>&1 &
152+
_pf_pid=$!
153+
timeout 20 sh -ec "until kubectl --kubeconfig '$_kc' --request-timeout=5s get --raw='/readyz' >/dev/null 2>&1; do sleep 1; done" || true
154+
;;
155+
esac
156+
fi
157+
_cozy_collect "$_kc" "$_snap/$_kc" -k "$_kc" --exclude-kind Secret
158+
_cozy_collect_helm "$_kc-helm" "$_snap/$_kc-helm" -k "$_kc"
159+
[ -n "$_pf_pid" ] && kill "$_pf_pid" 2>/dev/null || true
101160
done
102161
fi
103162
if command -v cozy_cleanup >/dev/null 2>&1; then

0 commit comments

Comments
 (0)