Skip to content

Commit 98f7d8c

Browse files
committed
fix(e2e): collect the console before the HelmRelease classification
The classification ran ahead of the guest serial console in the node-join block, on the rule that cheap reads precede collectors costing minutes. That rule is wrong here, and the ordering follows the evidence instead. The console is the only artefact covering a worker that never reached apid, which is the dominant shape of this failure and one nothing else in the block describes. Its measured worst case is most of the phase budget, so anything ahead of it competes with it on exactly the slow runs it exists for. The classification reads objects on the management cluster that outlive the run, so a walk the budget declines can still be answered afterwards. Irreplaceable evidence first, whichever is cheaper. Its section letter moves to (e): the block already had a (d). Both suites' spend-order comments list it in its new position, and the two guards over that order now match calls by position inside the block rather than by name. The same collector is called from the exit handler, which belongs to no phase, and by-name matching either mistook that call for one in the block or would have had to exempt the name outright, which would also have exempted a future call genuinely placed ahead of the console. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent b6ae56d commit 98f7d8c

5 files changed

Lines changed: 53 additions & 18 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,10 @@ A HelmRelease that hit its wait timeout, uninstalled, and reinstalled is a silen
6666
- Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile, which is what makes it readable at all after the release is Ready. It does not survive an unbounded number of them: the controller truncates history on every in-sync reconcile, retaining the newest five Snapshots on a release whose strategy is a retry and, on any other, those down to the previous `deployed` or `superseded` Snapshot — falling back to the same newest-five cap when there is no such Snapshot to cut at. Only the truncation itself is checkable from here: it lives in `helm-controller/api`, which this repository pins in `go.mod`. Which of the two a release gets is decided in the controller, and that module is not a dependency at all, so that half rests on upstream source this tree does not carry. A single e2e run applies the release once and never upgrades it afterwards, so the footprint is still present when the guard reads it — but a guard placed after further release actions, or one asserting on history days later, is reading a window and needs to say so. Use the shared helpers in `hack/e2e-chainsaw/_lib/remediation-guard.sh` from a `script` step.
6767
- **Fail on a teardown the release is configured not to perform; report everything else.** `helmrelease_has_teardown` reports an `uninstalled` Snapshot, which is proof the release was removed and put back. Whether the release could have performed that removal on itself is two fields, and `cozy_guard_helmrelease` reads both: `RetryOnFailure` keeps the manifests applied and retries in place, so it never uninstalls, and the default strategy uninstalls only between the retry attempts `.spec.install.remediation.retries` allows, so at a budget of `0` — which is also what an absent `remediation` block means, since the field is `omitempty` on an int — it performs none either. An `uninstalled` Snapshot on a release in either shape has no explanation inside it and fails the run. The tenant `cilium` and `csi` set `RetryOnFailure`, as does every Application HelmRelease, which the aggregated apiserver stamps that way (`pkg/registry/apps/application/rest.go`); the other eighteen addon releases the chart declares pair the default strategy with `retries: -1`, where the same Snapshot is that strategy doing its documented job and is printed instead. The only other configuration that writes the status is an upgrade remediation set to `strategy: uninstall`, which nothing under `packages/` uses.
6868
- **The strategy is read through a jsonpath, so something has to notice when it stops resolving.** `kubectl -o jsonpath` prints nothing and exits 0 for a path that no longer matches, and empty takes the lenient branch — a rename of `spec.install.strategy` in a future helm-controller would report every teardown as an install remediation, on a green run, with the unit tests still passing because they feed the value from a fixture rather than through the jsonpath. An empty value is not by itself evidence of that, since most addons legitimately set no strategy, so the check lives on the one release whose strategy the cluster is known to set: the parent, where an empty value fails the run. That is enough, because the same rename empties the field on every release at once.
69-
- **"Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on.** Three releases can fail a run on a *teardown*: the parent, and the tenant `cilium` and `csi`, which are the only ones this chart configures never to remove themselves to recover. On every other addon it renders, a teardown and a `failed` Snapshot alike are notes. That is the teardown rule and not the whole gate — the two conditions that red a run on any release, addons included, are a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read, and a read that failed, which is never folded into "no history". Stated as three rather than as a fraction because the denominator is not fixed: most addon templates are gated on their own `addons.<name>.enabled`, so how many releases a run has at all depends on what it enabled. That follows from the rule above rather than working around it, but it is worth stating, since the breadth of the reading and the breadth of the gate are not the same number.
69+
- **"Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on.** Three releases can fail a run on a *teardown*: the parent, and the tenant `cilium` and `csi`, which are the only ones this chart configures never to remove themselves to recover. On every other addon it renders, a teardown and a `failed` Snapshot alike are notes. That is the teardown rule and not the whole gate. Five conditions red a run and only the first is that rule: a teardown with no explanation inside the release, which reaches the parent, `cilium` and `csi`; a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read; a read that failed, which is never folded into "no history"; an empty `.spec.install.strategy.name` on the parent, the one release whose strategy the cluster is known to set; and an addon prefix that selects no HelmRelease at all, which is about the selection rather than about any one release, and which the passing path fails on rather than passing silently over nothing. Stated as three rather than as a fraction because the denominator is not fixed: most addon templates are gated on their own `addons.<name>.enabled`, so how many releases a run has at all depends on what it enabled. That follows from the rule above rather than working around it, but it is worth stating, since the breadth of the reading and the breadth of the gate are not the same number.
7070
- **The guard runs after the assertions, so the failure paths need the reading without the verdict.** Every deadline a child remediation surfaces as — a node that never turns Ready, a HelmRelease wait that expires, a PVC that never binds — ends the script under errexit well before the guard, so on exactly the runs a teardown explains, the guard never executes and the removal is named nowhere. `cozy_report_helmrelease_remediation` is the same classification with no verdict, called from the node-join collector and from the exit handler that covers every other failure. Arm that handler where the object it reads starts existing, not where the tenant becomes reachable: the parent HelmRelease is rendered from the CR at the apply, and everything between there and the tenant LB is a management-cluster wait on that release or on what it installs, so a handler armed after those waits misses the window a helm-wait budget expiring produces. It fails nothing: the run is already red, and the tenant crust-gather snapshot hangs off the caller's own exit status. It walks the releases under both the caller's diagnostics phase budget and one of its own, and names whatever it stopped short of. Both budgets decide whether the next read may *start*, not when the walk ends, so neither is a worst case: the ceiling is the budget plus the reads already committed to when it expires. Write it that way in any collector you add — a budget described as a total is a number a later reader will trust for sizing the step around it.
7171
- **Report `failed` rather than failing on it.** `helmrelease_has_remediation_cycle` is the broader `failed`-or-`uninstalled` reading; it catches the upgrade remediation that rolls back rather than uninstalls, but it is not proof anything was removed, since a retried install under `RetryOnFailure` leaves `failed` behind with its manifests still applied.
72-
- **Do not put an unmeasured pass condition on the gate.** The tenant addons that set no strategy pair the default one with `remediation.retries: -1`, a configuration that declares repeated uninstall-and-reinstall their recovery path. Nothing in the suite measures how often they take it, and failing a 25-minute bringup on a release doing what it is configured to do is how a guard ends up switched off. That the configuration is itself a defect is a separate matter from what the guard asserts. Print the finding, name the release, leave the run green, and tighten when a measured baseline or a strategy change makes the condition knowable.
72+
- **Do not put an unmeasured pass condition on the gate.** The tenant addons that set no strategy pair the default one with `remediation.retries: -1`, a configuration that declares repeated uninstall-and-reinstall their recovery path. Nothing in the suite measures how often they take it, and failing a 25-minute bringup on a release doing what it is configured to do is how a guard ends up switched off. That the configuration is itself a defect is a separate matter from what the guard asserts, tracked in cozystack/cozystack#3555. Print the finding, name the release, leave the run green, and tighten when a measured baseline or a strategy change makes the condition knowable.
7373
- **Do not infer a child release's state from the parent's.** The Kubernetes `ApplicationDefinition` sets `release.cozystack.io/helm-install-disable-wait`, which the apiserver turns into `DisableWait` on both actions, so the parent reaches Ready without waiting for anything it applied — and the suite waits on some addons by name while never naming others. An empty `.status.history` is therefore judged against that release's own Ready condition: Ready with no history is a status shape the helper can no longer read and fails the run, not-Ready with no history is a release that never completed an action and is reported as not inspected. Read readiness before the history, not after, or a release that finishes installing between the two reads presents as Ready with an empty history and fails the run for a shape it never had. That ordering closes the truncation half of the window and not all of it: `HelmReleaseStatus.ClearHistory()` empties history outright, and the controller that calls it is not a module this tree carries.
7474

7575
### 7. Test-Impact Analysis (TIA), default-on

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 25 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2588,20 +2588,6 @@ cozy_report_node_join_failure() {
25882588
kubectl -n tenant-test logs -l kamaji.clastix.io/name="kubernetes-${test_name}" \
25892589
-c talos-csr-signer --tail=200 --prefix
25902590

2591-
# (d) What the tenant HelmReleases did to themselves. The table printed at the
2592-
# top of this block says which of them are Ready; this says whether one was
2593-
# torn down and put back on the way here, which is the same question the guard
2594-
# asks after the assertions and cannot ask from there, because this block ends
2595-
# in the exit that keeps it from ever running. A teardown of the tenant CNI or
2596-
# CSI shows up as exactly this failure -- no node ever turns Ready -- and
2597-
# without this the removal that preceded it is named nowhere in the run.
2598-
#
2599-
# After (c) because (c) is two reads and this is a handful per release, and
2600-
# ahead of the collectors below because those cost minutes. It carries its own
2601-
# wall clock as well, so a namespace whose apiserver answers slowly costs a
2602-
# stated number of seconds rather than however many releases are in it.
2603-
cozy_report_helmrelease_remediation tenant-test "kubernetes-${test_name}" "(d) " || true
2604-
26052591
# Order below is load-bearing and the phase budget is why. The budget declines
26062592
# whatever has not started when it runs out, so what runs last is what gets
26072593
# declined -- and (c) is the discriminator for mode 2b, the failure this whole
@@ -2670,6 +2656,29 @@ cozy_report_node_join_failure() {
26702656
cozy_capture_tenant_serial_console 'node-join failed: captured after fewer than 2 tenant nodes became Ready inside the deadline' 6 || true
26712657
fi
26722658

2659+
# (e) What the tenant HelmReleases did to themselves. The table printed at the
2660+
# top of this block says which of them are Ready; this says whether one was
2661+
# torn down and put back on the way here, which is the same question the guard
2662+
# asks after the assertions and cannot ask from there, because this block ends
2663+
# in the exit that keeps it from ever running. A teardown of the tenant CNI or
2664+
# CSI shows up as exactly this failure -- no node ever turns Ready -- and
2665+
# without this the removal that preceded it is named nowhere in the run.
2666+
#
2667+
# Below the console rather than above it, and the ordering rule is the
2668+
# evidence, not the price. The console is the only artefact that survives a
2669+
# worker which never reached apid, which is the dominant shape of this
2670+
# failure: nothing else in this block describes that worker at all. This
2671+
# classification is cheap and it discriminates, but what it reads lives on the
2672+
# management cluster and outlives the run, so a walk the budget declines here
2673+
# can be answered afterwards from the objects themselves. An irreplaceable
2674+
# artefact is collected before a replaceable one, whichever is cheaper.
2675+
#
2676+
# Its own wall clock still applies, so a namespace whose apiserver answers
2677+
# slowly costs a stated number of seconds rather than however many releases
2678+
# happen to be in it. A decline is announced rather than silent, because a
2679+
# missing section reads like a namespace that had nothing to report.
2680+
cozy_report_helmrelease_remediation tenant-test "kubernetes-${test_name}" "(e) " || true
2681+
26732682
# (d) What the workers got, what they were denied, and what the sandbox nodes
26742683
# under them lost to their own hypervisor. Its question has no other answer
26752684
# here: the ceiling is already carried by (a), but whether the guest ever met
@@ -2902,8 +2911,8 @@ _cozy_guard_kubectl() {
29022911
# declares uninstall-and-reinstall their recovery path; the run has no measurement of how
29032912
# often they take it, and failing a 25-minute bringup on a release doing what it
29042913
# is configured to do is how a guard ends up switched off. That the configuration
2905-
# is itself the defect is true and is not this guard's to assert - it is filed
2906-
# separately. A "failed" Snapshot is reported for the same reason and one more:
2914+
# is itself the defect is true and is not this guard's to assert - it is
2915+
# cozystack/cozystack#3555. A "failed" Snapshot is reported for the same reason and one more:
29072916
# under RetryOnFailure it is not even remediation, just an attempt that kept its
29082917
# manifests and was retried.
29092918
#

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ spec:
3434
# node-join diagnostics reads
3535
# worker network counters
3636
# serial-console family (~6m25s at worst, ~4m05s at the pool's minimum)
37+
# HelmRelease remediation footprint (90s of its own; below the console because the console alone covers a worker that never reached apid, while these releases stay readable after the run)
3738
# worker CPU usage and throttling counters, two samples
3839
# sandbox node CPU time, two samples
3940
# (those two alternate rather than run to completion in turn: both

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ spec:
2828
# node-join diagnostics reads
2929
# worker network counters
3030
# serial-console family (~6m25s at worst, ~4m05s at the pool's minimum)
31+
# HelmRelease remediation footprint (90s of its own; below the console because the console alone covers a worker that never reached apid, while these releases stay readable after the run)
3132
# worker CPU usage and throttling counters, two samples
3233
# sandbox node CPU time, two samples
3334
# (those two alternate rather than run to completion in turn: both

‎hack/run-kubernetes-node-join_test.bats‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1256,9 +1256,18 @@ assert_file_lacks_pattern() {
12561256
echo "found no gated collector at all, so this guard checked nothing" >&2
12571257
return 1
12581258
fi
1259+
# Same block filter as the spend-order test below, and for the same reason:
1260+
# the exit handler calls one of these collectors from outside any phase, and
1261+
# a call that spends no phase budget cannot spend it ahead of the console.
1262+
block=$(grep -n '^cozy_report_node_join_failure()' "$lib" | cut -d: -f1)
1263+
if [ -z "$block" ]; then
1264+
echo "expected to find the node-join diagnostics block in $lib" >&2
1265+
return 1
1266+
fi
12591267
for entry in $gated; do
12601268
line=${entry%%:*}
12611269
fn=${entry#*:}
1270+
[ "$line" -gt "$block" ] || continue
12621271
[ "$line" -lt "$console" ] || continue
12631272
case "$fn" in
12641273
# Not behind the phase gate at all: it runs ahead of the headline so the
@@ -1424,6 +1433,7 @@ ${carrier}
14241433
# step. A function that is neither still fails below, which is what makes
14251434
# this a list of exemptions rather than a list of everything.
14261435
cozy_diag_read | _ghcr_mirror_bounded_read | _talos_image_cache_bounded_read) continue ;;
1436+
_cozy_guard_kubectl) continue ;;
14271437
_cozy_cadvisor_node_stream | _cozy_cadvisor_worker_nodes) continue ;;
14281438
cozy_report_node_join_failure | _talos_image_cache_deploy_state) continue ;;
14291439
*)
@@ -1470,15 +1480,29 @@ ${carrier}
14701480
echo "found no gated collector at all, so this test checked nothing" >&2
14711481
exit 1
14721482
fi
1483+
# Only the calls inside the diagnostics block. The same collector may also be
1484+
# called from elsewhere in the file -- the exit handler classifies the tenant
1485+
# HelmReleases on every other failing path -- and such a call belongs to no
1486+
# phase and has no place in this order. Matched on position rather than
1487+
# exempted by name, so a second call added INSIDE the block still has to
1488+
# appear in the list.
1489+
block=$(grep -n '^cozy_report_node_join_failure()' "$lib" | cut -d: -f1)
1490+
if [ -z "$block" ]; then
1491+
echo "expected to find the node-join diagnostics block in $lib" >&2
1492+
exit 1
1493+
fi
14731494
expected=
14741495
for entry in $gated; do
1496+
line=${entry%%:*}
14751497
fn=${entry#*:}
1498+
[ "$line" -gt "$block" ] || continue
14761499
case "$fn" in
14771500
cozy_capture_tenant_worker_cpu_throttle) phrase='worker CPU usage and throttling counters' ;;
14781501
cozy_capture_sandbox_node_cpu_time) phrase='sandbox node CPU time' ;;
14791502
cozy_capture_tenant_worker_network_counters) phrase='worker network counters' ;;
14801503
cozy_capture_tenant_serial_console) phrase='serial-console family' ;;
14811504
cozy_capture_tenant_talos) phrase='guest Talos capture' ;;
1505+
cozy_report_helmrelease_remediation) phrase='HelmRelease remediation footprint' ;;
14821506
ghcr_mirror_diagnose) phrase='ghcr-mirror state' ;;
14831507
talos_image_cache_diagnose) phrase='talos-image-cache diagnosis' ;;
14841508
# Called with the same suffix but not behind the phase gate: it runs ahead

0 commit comments

Comments
 (0)