Skip to content

Commit 8cb1023

Browse files
committed
fix(e2e): fail on a failed release of cilium and csi as on the parent
The history guard covered only the parent Kubernetes HelmRelease. A failed action on one of the addon releases it installs left no mark the suite inspected, so a run surfaced only the downstream symptom, typically the node-join deadline, with nothing naming the release behind it. The tenant CNI and CSI are the sharp cases, since they carry a cluster-wide precondition. The guard now reads every addon release of the parent, selected from the cluster by name prefix so a release added later is covered too. A "failed" or "uninstalled" Snapshot fails the run on a release whose install strategy is RetryOnFailure: the parent, which the aggregated apiserver stamps that way, and cilium and csi. The same footprint on the other addons, which pair the default strategy with retries: -1, is printed as a note and leaves the run green. The footprint is not split into a teardown and a retried failure. In helm-controller v1.5.0, the version the platform installs, and unchanged in v1.5.1, a release with an active retry strategy never remediates: atomic_release.go returns an upgrade for a failed release before it reaches the remediation switch (lines 465-475), and the two other paths that uninstall, a changed release target and deletion, both call ClearHistory afterwards. On a RetryOnFailure release an "uninstalled" Snapshot is therefore unreachable, and the "failed" one is the only signal. A retry that later recovered still fails the run, by design, with one blind spot the statuses cannot close: a failed install that the very next attempt recovers is marked superseded by Helm, because there is no deployed release to supersede instead, and its history reads like an ordinary upgrade. The guard runs after the assertions, while every deadline a failed addon breaks ends the script earlier, so the same classification is also printed without a verdict from the node-join collector and from the exit handler. That handler is now armed at the tenant CR apply, so the management-cluster waits before the tenant answers are covered too. Every read is bounded and never folded into "no history", and the parent must read back RetryOnFailure: an empty value there means the jsonpath stopped resolving, and any other value means the apiserver stamp is gone, either of which would turn its footprint into a note. The history read takes its jsonpath from HELMRELEASE_HISTORY_JSONPATH, and the upstream-type pin accepts the bounded wrapper the read goes through. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent bf03c84 commit 8cb1023

10 files changed

Lines changed: 2328 additions & 67 deletions

‎hack/e2e-chainsaw/_lib/remediation-guard.sh‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,28 @@
1313
# tracks the Helm release state: deployed, superseded, failed, uninstalled,
1414
# and so on. A remediation cycle leaves the footprint behind: a snapshot
1515
# with status "uninstalled" (from install/upgrade remediation) or "failed"
16-
# (Helm release failure that remediation then uninstalled). Those stay in
17-
# history even after a subsequent successful reinstall.
16+
# (a Helm action that failed). Those stay in history after a subsequent
17+
# successful reinstall or retry, with one exception: a failed install that
18+
# the next upgrade recovers is marked superseded by Helm, so its Snapshot
19+
# no longer reads "failed" and nothing here can see it.
20+
#
21+
# "Bounded" is literal. The controller truncates history on every in-sync
22+
# reconcile, and again before it retries or remediates a failed release:
23+
# on a release whose strategy is a retry it keeps the newest
24+
# five Snapshots, and on any other it keeps those down to the previous
25+
# deployed or superseded one, falling back to that same five when history
26+
# holds no such Snapshot to cut at. A footprint therefore outlives the
27+
# reconcile that follows it, not an arbitrary number of them. Reading
28+
# history right after the actions that wrote it, which is what an e2e run
29+
# does, sees it; reading it after the release has moved on may not.
1830
#
1931
# helmrelease_has_remediation_cycle takes a newline-delimited list of
2032
# snapshot statuses (whatever the caller extracted via kubectl -o jsonpath
2133
# or equivalent) and returns 0 (detected) when any entry is "failed" or
2234
# "uninstalled", 1 otherwise. Empty input is treated as "no history yet,
23-
# no cycle observed".
35+
# no cycle observed". Whether a detected footprint fails a run is the
36+
# caller's decision, because it depends on the release's own strategy:
37+
# see cozy_guard_helmrelease in run-kubernetes.sh.
2438

2539
# The jsonpath a caller hands to kubectl to read those statuses, one per
2640
# line. It lives here rather than inline at the call site so that it is a
@@ -35,6 +49,7 @@
3549
HELMRELEASE_HISTORY_JSONPATH='{range .status.history[*]}{.status}{"\n"}{end}'
3650

3751
helmrelease_has_remediation_cycle() {
52+
local statuses
3853
statuses="$1"
3954
if [ -z "${statuses}" ]; then
4055
return 1

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 576 additions & 47 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ spec:
4242
# worker network counters
4343
# worker block IO counters
4444
# serial-console family (~6m25s at worst, ~4m05s at the pool's minimum)
45+
# HelmRelease remediation footprint (a 90s budget of its own on starting reads, not a ceiling; below the console because the console alone covers a worker that never reached apid, while these releases stay readable after the run)
4546
# worker CPU usage and throttling counters, two samples
4647
# sandbox node CPU time, two samples
4748
# worker per-thread CPU time inside virt-launcher, two samples

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ spec:
2929
# worker network counters
3030
# worker block IO counters
3131
# serial-console family (~6m25s at worst, ~4m05s at the pool's minimum)
32+
# HelmRelease remediation footprint (a 90s budget of its own on starting reads, not a ceiling; below the console because the console alone covers a worker that never reached apid, while these releases stay readable after the run)
3233
# worker CPU usage and throttling counters, two samples
3334
# sandbox node CPU time, two samples
3435
# worker per-thread CPU time inside virt-launcher, two samples

‎hack/remediation-guard.bats‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,3 +80,16 @@
8080
exit 1
8181
fi
8282
}
83+
84+
@test "the predicate does not overwrite a caller variable named statuses" {
85+
# This library is sourced rather than executed, and the helper assigns
86+
# "statuses" internally, so without a local the argument lands in the
87+
# caller's scope.
88+
. hack/e2e-chainsaw/_lib/remediation-guard.sh
89+
statuses=sentinel
90+
helmrelease_has_remediation_cycle "failed"
91+
if [ "${statuses}" != sentinel ]; then
92+
echo "helmrelease_has_remediation_cycle replaced the caller's statuses with: ${statuses}" >&2
93+
exit 1
94+
fi
95+
}

‎hack/run-kubernetes-node-join_test.bats‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -939,8 +939,11 @@ read_cost_inputs() {
939939
# The grace is in this list because `timeout -k abc 20` exits 125 BEFORE running the
940940
# command, so a non-numeric grace makes every read in the block report exit 125 and
941941
# collects nothing -- the same total loss as the other three, by a different route.
942+
# The report budget is in this list because the classification runs inside this
943+
# block: rejected, it falls back to its default like the four above, and the
944+
# collector keeps its walk.
942945
for knob in COZY_DIAG_PHASE_BUDGET COZY_DIAG_READ_TIMEOUT COZY_DIAG_MAX_IMPORTERS \
943-
COZY_DIAG_READ_GRACE; do
946+
COZY_DIAG_READ_GRACE COZY_GUARD_REPORT_BUDGET; do
944947
# Only values a `:-` default cannot absorb. An empty value set here is
945948
# indistinguishable from unset, so it takes the default silently and correctly;
946949
# the empty-string hazard lives on the post-source path and is checked there.
@@ -1417,9 +1420,18 @@ read_cost_inputs() {
14171420
echo "found no gated collector at all, so this guard checked nothing" >&2
14181421
return 1
14191422
fi
1423+
# Same block filter as the spend-order test below, and for the same reason:
1424+
# the exit handler calls one of these collectors from outside any phase, and
1425+
# a call that spends no phase budget cannot spend it ahead of the console.
1426+
block=$(grep -n '^cozy_report_node_join_failure()' "$lib" | cut -d: -f1)
1427+
if [ -z "$block" ]; then
1428+
echo "expected to find the node-join diagnostics block in $lib" >&2
1429+
return 1
1430+
fi
14201431
for entry in $gated; do
14211432
line=${entry%%:*}
14221433
fn=${entry#*:}
1434+
[ "$line" -gt "$block" ] || continue
14231435
[ "$line" -lt "$console" ] || continue
14241436
case "$fn" in
14251437
# Not behind the phase gate at all: it runs ahead of the headline so the
@@ -1636,6 +1648,7 @@ ${carrier}
16361648
# step. A function that is neither still fails below, which is what makes
16371649
# this a list of exemptions rather than a list of everything.
16381650
cozy_diag_read | _talos_image_cache_bounded_read) continue ;;
1651+
_cozy_guard_kubectl) continue ;;
16391652
_cozy_cadvisor_node_stream | _cozy_virt_launcher_listing) continue ;;
16401653
# The canary carries its guard in the body that runs one arm, the way the
16411654
# cAdvisor captures carry theirs in the stream they share, so the sentence
@@ -1690,9 +1703,30 @@ ${carrier}
16901703
echo "found no gated collector at all, so this test checked nothing" >&2
16911704
exit 1
16921705
fi
1706+
# Only the calls inside the diagnostics block. The same collector may also be
1707+
# called from elsewhere in the file -- the exit handler classifies the tenant
1708+
# HelmReleases on every other failing path -- and such a call belongs to no
1709+
# phase and has no place in this order. Matched on position rather than
1710+
# exempted by name, so a second call added INSIDE the block still has to
1711+
# appear in the list.
1712+
block=$(grep -n '^cozy_report_node_join_failure()' "$lib" | cut -d: -f1)
1713+
if [ -z "$block" ]; then
1714+
echo "expected to find the node-join diagnostics block in $lib" >&2
1715+
exit 1
1716+
fi
1717+
# Bounded at both ends: without the closing brace every collector call below
1718+
# the block, run_kubernetes_test's included, would be read as inside it.
1719+
block_end=$(awk -v b="$block" 'NR > b && /^}/ { print NR; exit }' "$lib")
1720+
if [ -z "$block_end" ]; then
1721+
echo "expected to find the end of the node-join diagnostics block in $lib" >&2
1722+
exit 1
1723+
fi
16931724
expected=
16941725
for entry in $gated; do
1726+
line=${entry%%:*}
16951727
fn=${entry#*:}
1728+
[ "$line" -gt "$block" ] || continue
1729+
[ "$line" -lt "$block_end" ] || continue
16961730
case "$fn" in
16971731
cozy_capture_tenant_worker_cpu_throttle) phrase='worker CPU usage and throttling counters' ;;
16981732
cozy_capture_sandbox_node_cpu_time) phrase='sandbox node CPU time' ;;
@@ -1701,6 +1735,7 @@ ${carrier}
17011735
cozy_capture_tenant_worker_block_io) phrase='worker block IO counters' ;;
17021736
cozy_capture_tenant_serial_console) phrase='serial-console family' ;;
17031737
cozy_capture_tenant_talos) phrase='guest Talos capture' ;;
1738+
cozy_report_helmrelease_remediation) phrase='HelmRelease remediation footprint' ;;
17041739
talos_image_cache_diagnose) phrase='talos-image-cache diagnosis' ;;
17051740
# Called with the same suffix but not behind the phase gate: it runs ahead
17061741
# of the headline so the console experiment's own failure is named before

0 commit comments

Comments
 (0)