You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 98f7d8c
Browse filesBrowse the repository at this point in the historyBrowse files
fix(e2e): collect the console before the HelmRelease classification
The classification ran ahead of the guest serial console in the node-join
block, on the rule that cheap reads precede collectors costing minutes.
That rule is wrong here, and the ordering follows the evidence instead.
The console is the only artefact covering a worker that never reached
apid, which is the dominant shape of this failure and one nothing else in
the block describes. Its measured worst case is most of the phase budget,
so anything ahead of it competes with it on exactly the slow runs it
exists for. The classification reads objects on the management cluster
that outlive the run, so a walk the budget declines can still be answered
afterwards. Irreplaceable evidence first, whichever is cheaper.
Its section letter moves to (e): the block already had a (d).
Both suites' spend-order comments list it in its new position, and the
two guards over that order now match calls by position inside the block
rather than by name. The same collector is called from the exit handler,
which belongs to no phase, and by-name matching either mistook that call
for one in the block or would have had to exempt the name outright,
which would also have exempted a future call genuinely placed ahead of
the console.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -66,10 +66,10 @@ A HelmRelease that hit its wait timeout, uninstalled, and reinstalled is a silen
66
66
- Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile, which is what makes it readable at all after the release is Ready. It does not survive an unbounded number of them: the controller truncates history on every in-sync reconcile, retaining the newest five Snapshots on a release whose strategy is a retry and, on any other, those down to the previous `deployed` or `superseded` Snapshot — falling back to the same newest-five cap when there is no such Snapshot to cut at. Only the truncation itself is checkable from here: it lives in `helm-controller/api`, which this repository pins in `go.mod`. Which of the two a release gets is decided in the controller, and that module is not a dependency at all, so that half rests on upstream source this tree does not carry. A single e2e run applies the release once and never upgrades it afterwards, so the footprint is still present when the guard reads it — but a guard placed after further release actions, or one asserting on history days later, is reading a window and needs to say so. Use the shared helpers in `hack/e2e-chainsaw/_lib/remediation-guard.sh` from a `script` step.
67
67
- **Fail on a teardown the release is configured not to perform; report everything else.** `helmrelease_has_teardown` reports an `uninstalled` Snapshot, which is proof the release was removed and put back. Whether the release could have performed that removal on itself is two fields, and `cozy_guard_helmrelease` reads both: `RetryOnFailure` keeps the manifests applied and retries in place, so it never uninstalls, and the default strategy uninstalls only between the retry attempts `.spec.install.remediation.retries` allows, so at a budget of `0` — which is also what an absent `remediation` block means, since the field is `omitempty` on an int — it performs none either. An `uninstalled` Snapshot on a release in either shape has no explanation inside it and fails the run. The tenant `cilium` and `csi` set `RetryOnFailure`, as does every Application HelmRelease, which the aggregated apiserver stamps that way (`pkg/registry/apps/application/rest.go`); the other eighteen addon releases the chart declares pair the default strategy with `retries: -1`, where the same Snapshot is that strategy doing its documented job and is printed instead. The only other configuration that writes the status is an upgrade remediation set to `strategy: uninstall`, which nothing under `packages/` uses.
68
68
- **The strategy is read through a jsonpath, so something has to notice when it stops resolving.** `kubectl -o jsonpath` prints nothing and exits 0 for a path that no longer matches, and empty takes the lenient branch — a rename of `spec.install.strategy` in a future helm-controller would report every teardown as an install remediation, on a green run, with the unit tests still passing because they feed the value from a fixture rather than through the jsonpath. An empty value is not by itself evidence of that, since most addons legitimately set no strategy, so the check lives on the one release whose strategy the cluster is known to set: the parent, where an empty value fails the run. That is enough, because the same rename empties the field on every release at once.
69
-
- **"Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on.** Three releases can fail a run on a *teardown*: the parent, and the tenant `cilium` and `csi`, which are the only ones this chart configures never to remove themselves to recover. On every other addon it renders, a teardown and a `failed` Snapshot alike are notes. That is the teardown rule and not the whole gate — the two conditions that red a run on any release, addons included, are a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read, and a read that failed, which is never folded into "no history". Stated as three rather than as a fraction because the denominator is not fixed: most addon templates are gated on their own `addons.<name>.enabled`, so how many releases a run has at all depends on what it enabled. That follows from the rule above rather than working around it, but it is worth stating, since the breadth of the reading and the breadth of the gate are not the same number.
69
+
- **"Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on.** Three releases can fail a run on a *teardown*: the parent, and the tenant `cilium` and `csi`, which are the only ones this chart configures never to remove themselves to recover. On every other addon it renders, a teardown and a `failed` Snapshot alike are notes. That is the teardown rule and not the whole gate. Five conditions red a run and only the first is that rule: a teardown with no explanation inside the release, which reaches the parent, `cilium` and `csi`; a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read; a read that failed, which is never folded into "no history"; an empty `.spec.install.strategy.name` on the parent, the one release whose strategy the cluster is known to set; and an addon prefix that selects no HelmRelease at all, which is about the selection rather than about any one release, and which the passing path fails on rather than passing silently over nothing. Stated as three rather than as a fraction because the denominator is not fixed: most addon templates are gated on their own `addons.<name>.enabled`, so how many releases a run has at all depends on what it enabled. That follows from the rule above rather than working around it, but it is worth stating, since the breadth of the reading and the breadth of the gate are not the same number.
70
70
- **The guard runs after the assertions, so the failure paths need the reading without the verdict.** Every deadline a child remediation surfaces as — a node that never turns Ready, a HelmRelease wait that expires, a PVC that never binds — ends the script under errexit well before the guard, so on exactly the runs a teardown explains, the guard never executes and the removal is named nowhere. `cozy_report_helmrelease_remediation` is the same classification with no verdict, called from the node-join collector and from the exit handler that covers every other failure. Arm that handler where the object it reads starts existing, not where the tenant becomes reachable: the parent HelmRelease is rendered from the CR at the apply, and everything between there and the tenant LB is a management-cluster wait on that release or on what it installs, so a handler armed after those waits misses the window a helm-wait budget expiring produces. It fails nothing: the run is already red, and the tenant crust-gather snapshot hangs off the caller's own exit status. It walks the releases under both the caller's diagnostics phase budget and one of its own, and names whatever it stopped short of. Both budgets decide whether the next read may *start*, not when the walk ends, so neither is a worst case: the ceiling is the budget plus the reads already committed to when it expires. Write it that way in any collector you add — a budget described as a total is a number a later reader will trust for sizing the step around it.
71
71
- **Report `failed` rather than failing on it.** `helmrelease_has_remediation_cycle` is the broader `failed`-or-`uninstalled` reading; it catches the upgrade remediation that rolls back rather than uninstalls, but it is not proof anything was removed, since a retried install under `RetryOnFailure` leaves `failed` behind with its manifests still applied.
72
-
- **Do not put an unmeasured pass condition on the gate.** The tenant addons that set no strategy pair the default one with `remediation.retries: -1`, a configuration that declares repeated uninstall-and-reinstall their recovery path. Nothing in the suite measures how often they take it, and failing a 25-minute bringup on a release doing what it is configured to do is how a guard ends up switched off. That the configuration is itself a defect is a separate matter from what the guard asserts. Print the finding, name the release, leave the run green, and tighten when a measured baseline or a strategy change makes the condition knowable.
72
+
- **Do not put an unmeasured pass condition on the gate.** The tenant addons that set no strategy pair the default one with `remediation.retries: -1`, a configuration that declares repeated uninstall-and-reinstall their recovery path. Nothing in the suite measures how often they take it, and failing a 25-minute bringup on a release doing what it is configured to do is how a guard ends up switched off. That the configuration is itself a defect is a separate matter from what the guard asserts, tracked in cozystack/cozystack#3555. Print the finding, name the release, leave the run green, and tighten when a measured baseline or a strategy change makes the condition knowable.
73
73
- **Do not infer a child release's state from the parent's.** The Kubernetes `ApplicationDefinition` sets `release.cozystack.io/helm-install-disable-wait`, which the apiserver turns into `DisableWait` on both actions, so the parent reaches Ready without waiting for anything it applied — and the suite waits on some addons by name while never naming others. An empty `.status.history` is therefore judged against that release's own Ready condition: Ready with no history is a status shape the helper can no longer read and fails the run, not-Ready with no history is a release that never completed an action and is reported as not inspected. Read readiness before the history, not after, or a release that finishes installing between the two reads presents as Ready with an empty history and fails the run for a shape it never had. That ordering closes the truncation half of the window and not all of it: `HelmReleaseStatus.ClearHistory()` empties history outright, and the controller that calls it is not a module this tree carries.
Copy file name to clipboardExpand all lines: hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
+1Lines changed: 1 addition & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -34,6 +34,7 @@ spec:
34
34
# node-join diagnostics reads
35
35
# worker network counters
36
36
# serial-console family (~6m25s at worst, ~4m05s at the pool's minimum)
37
+
# HelmRelease remediation footprint (90s of its own; below the console because the console alone covers a worker that never reached apid, while these releases stay readable after the run)
37
38
# worker CPU usage and throttling counters, two samples
38
39
# sandbox node CPU time, two samples
39
40
# (those two alternate rather than run to completion in turn: both
Copy file name to clipboardExpand all lines: hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
+1Lines changed: 1 addition & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -28,6 +28,7 @@ spec:
28
28
# node-join diagnostics reads
29
29
# worker network counters
30
30
# serial-console family (~6m25s at worst, ~4m05s at the pool's minimum)
31
+
# HelmRelease remediation footprint (90s of its own; below the console because the console alone covers a worker that never reached apid, while these releases stay readable after the run)
31
32
# worker CPU usage and throttling counters, two samples
32
33
# sandbox node CPU time, two samples
33
34
# (those two alternate rather than run to completion in turn: both
0 commit comments