Skip to content

Commit 415430b

Browse files
authored
fix(e2e): fail on a failed release of cilium and csi as on the parent, and report addon remediation (cozystack#3594)
## What this PR does The history guard in the tenant Kubernetes e2e suite checked only the parent `kubernetes-<test>` HelmRelease. A failed action on one of the addon releases it installs left nothing the suite looked at, so a run showed only the downstream symptom, usually the node-join deadline, and never named the release behind it. The guard now reads every addon release of the parent. It selects them from the cluster by name prefix, so a release added to the chart later is covered too. The rule: - A `failed` or `uninstalled` Snapshot fails the run on any release whose install strategy is `RetryOnFailure`. That is the parent (the aggregated apiserver stamps it on every Application HelmRelease) and the tenant `cilium` and `csi`. - The same footprint on the other addons, which run the default strategy with `retries: -1`, is printed as a note and the run stays green. For the CRD addons that configuration is tracked in cozystack#3555. A transient `cilium` or `csi` failure that later recovers on a retry still turns the run red, by design, as long as its Snapshot still reads `failed`. One recovery leaves no such Snapshot: when an install fails and the very next attempt succeeds, Helm marks the failed install `superseded`, because there is no deployed release to supersede, and the history reads like an ordinary upgrade. The guard cannot see that case from the statuses alone. The code and the docs now say so, a test pins it, and detection is left to a follow-up. An earlier revision of this PR tried to tell a teardown apart from a retried failure and made only `uninstalled` fatal. That split does not hold. In helm-controller v1.5.0, the version the platform installs, a release with an active retry strategy never remediates. `atomic_release.go` returns an upgrade for a failed release before it reaches the remediation switch, and the two other paths that uninstall (a changed release target, and deletion) both clear history afterwards. So on these releases `uninstalled` never shows up, and `failed` is the only signal. The parent keeps the rule it already had on main, and `cilium` and `csi` now follow it too. The failure paths get the same reading. The guard runs after the assertions, while the deadlines a failed addon breaks end the script earlier, so the same classification is also printed, without a verdict, from the node-join collector and from the exit handler. That handler is now armed at the tenant CR apply, so the management-cluster waits before the tenant answers are covered too. Smaller changes that came with this: - Every guard read is bounded, and a failed read is never folded into "no history". - The parent must read back a non-empty strategy. An empty value there means the jsonpath stopped resolving, and without this check every footprint would quietly turn into a note. - The history read uses `HELMRELEASE_HISTORY_JSONPATH`, and the upstream-type pin in `internal/fluxcontract` now accepts the bounded wrapper the read goes through. `docs/agents/e2e-testing.md` §6 and checklist item 8 describe the new rule. On related issues: this adds the `local` that cozystack#3774 reports missing in `helmrelease_has_remediation_cycle`, but not the sweep over the other `_lib/` helpers that the issue also asks for. For cozystack#3773 it rewrites the guard comment the issue quotes. The waits without a backstop are not touched. Closes cozystack#3553. ### Screenshots Not a UI change. ### Downstream repositories I walked the trigger map against the diff. It touches `hack/e2e-chainsaw/_lib/`, `hack/*.bats`, one Go test under `internal/fluxcontract`, two spend-order comments in the kubernetes suites, and `docs/agents/e2e-testing.md`. No package, schema, CRD, make target or user-facing document changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): the HelmRelease history guard now also covers the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place; other addons get a note. The same classification is printed on the failure paths where the guard itself cannot run. ```
2 parents 0639ce3 + b42805b commit 415430b

11 files changed

Lines changed: 2337 additions & 71 deletions

‎docs/agents/e2e-testing.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,11 +84,16 @@ The install test fails if **any** HelmRelease is not Ready. A toothless gate (ba
8484
- Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR.
8585
- Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`.
8686

87-
### 6. Assert the parent HelmRelease did not remediate — via `status.history`
87+
### 6. Assert no HelmRelease failed an action it retries in place — via `status.history`
8888

89-
A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is a silent race we want to catch. **Do not** check `.status.installFailures` / `.status.upgradeFailures`: Flux's `ClearFailures()` zeroes those on every successful reconcile, so checking them after the HR is Ready is **vacuous** and passes against a reverted fix.
89+
A HelmRelease that hit its wait timeout and failed an action is a silent race we want to catch, and it stays one when a later retry recovers the release. **Do not** check `.status.installFailures` / `.status.upgradeFailures`: Flux's `ClearFailures()` zeroes those on every successful reconcile, so checking them after the HR is Ready is **vacuous** and passes against a reverted fix.
9090

91-
- Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile. Use the shared helper in `hack/e2e-chainsaw/_lib/remediation-guard.sh` (`helmrelease_has_remediation_cycle`) from a `script` step. Take the jsonpath from `HELMRELEASE_HISTORY_JSONPATH` in that same file rather than writing it out: `internal/fluxcontract` sources that file and runs the resulting value against the upstream Flux type, and a literal copied elsewhere is not covered by it.
91+
- Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile, which is what makes it readable at all after the release is Ready. It does not survive an unbounded number of them: the controller truncates history on every in-sync reconcile and again before it retries or remediates a failed release, retaining the newest five Snapshots on a release whose strategy is a retry and, on any other, those down to the previous `deployed` or `superseded` Snapshot — falling back to the same newest-five cap when there is no such Snapshot to cut at. An e2e run takes only a handful of actions on each release, so the footprint is still present when the guard reads it — but a guard placed after further release actions, or one asserting on history days later, is reading a window and needs to say so. Use the shared helper in `hack/e2e-chainsaw/_lib/remediation-guard.sh` (`helmrelease_has_remediation_cycle`) from a `script` step. Take the jsonpath from `HELMRELEASE_HISTORY_JSONPATH` in that same file rather than writing it out: `internal/fluxcontract` sources that file and runs the resulting value against the upstream Flux type, and a literal copied elsewhere is not covered by it.
92+
- **Fail on any footprint of a release that retries in place; report it on the others.** Under `RetryOnFailure` helm-controller never remediates: a failed action is retried as an upgrade with its manifests left in place, so the release never writes an `uninstalled` Snapshot of its own, and the `failed` one is the only record of the attempt. `cozy_guard_helmrelease` therefore fails the run on either status on a release whose `.spec.install.strategy.name` is `RetryOnFailure`: the parent, which the aggregated apiserver stamps that way like every Application HelmRelease (`pkg/registry/apps/application/rest.go`), and the tenant `cilium` and `csi`, whose templates set it. A retry that recovered still fails the run, by design, while the Snapshot still reads `failed`: the failed attempt happened, and on these releases nothing else records it. It does not after a failed install that the very next attempt recovers. With no deployed release to upgrade from, Helm upgrades from the failed one and marks it `superseded` on success, helm-controller rewrites that Snapshot in place, and the history reads `deployed`, `superseded` like any ordinary upgrade, so the guard passes. A failed upgrade keeps its `failed` Snapshot, since Helm supersedes the last deployed release instead, and so does an install that fails twice before recovering. That makes a late node join a red run here even when it lands inside the 29m node-join deadline: the tenant `cilium` install cannot complete before a worker registers, because its operator Deployment has nowhere to schedule, so workers arriving more than two of its 10m install timeouts after the install started leave a `failed` Snapshot, and the guard fails the run on it. This is accepted rather than carved out, since the chart itself treats an install outrunning its budget for want of a node as a problem, but the cause is the join and not the CNI. The guard's message says so and points at the Snapshot times in the describe output it prints. The other addon releases pair the default strategy with `retries: -1`, a configuration that declares repeated remediation their recovery path. Nothing in the suite measures how often they take it, and failing a whole bringup on a release doing what it is configured to do is how a guard ends up switched off, so the same footprint there is printed as a note and the run stays green. For the CRD addons that configuration is itself a defect, tracked in cozystack/cozystack#3555; tighten when a measured baseline or a strategy change makes the condition knowable.
93+
- **The strategy is read through a jsonpath, so something has to notice when it stops resolving.** `kubectl -o jsonpath` prints nothing and exits 0 for a path that no longer matches, and empty takes the lenient branch — a rename of `spec.install.strategy` in a future helm-controller would report every footprint on the parent, `cilium` and `csi` as a note, on a green run, with the unit tests still passing because they feed the value from a fixture rather than through the jsonpath. An empty value is not by itself evidence of that, since most addons legitimately set no strategy, so the check lives on the one release whose strategy the cluster is known to set: the parent, where an empty value, or any value other than `RetryOnFailure`, fails the run. That is enough, because the same rename empties the field on every release at once.
94+
- **"Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on.** Five conditions red a run: a `failed` or `uninstalled` Snapshot on a release that retries in place; a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read; a read that failed, which is never folded into "no history"; an empty or non-`RetryOnFailure` `.spec.install.strategy.name` on the parent; and an addon prefix that selects no HelmRelease at all, which the passing path fails on rather than passing silently over nothing. Every other addon footprint is a note. Which releases a run has at all depends on what it enabled, since most addon templates are gated on their own `addons.<name>.enabled`, so the breadth of the reading and the breadth of the gate are not the same number.
95+
- **The guard runs after the assertions, so the failure paths need the reading without the verdict.** Every deadline a failed child action surfaces as — a node that never turns Ready, a HelmRelease wait that expires, a PVC that never binds — ends the script under errexit well before the guard, so on exactly the runs such an action explains, the guard never executes and the history is named nowhere. `cozy_report_helmrelease_remediation` is the same classification with no verdict, called from the node-join collector and from the exit handler that covers every other failure. Arm that handler where the object it reads starts existing, not where the tenant becomes reachable: the parent HelmRelease is rendered from the CR at the apply, and everything between there and the tenant LB is a management-cluster wait on that release or on what it installs, so a handler armed after those waits misses the window in which an install action times out. It fails nothing: the run is already red, and the tenant crust-gather snapshot hangs off the caller's own exit status. It walks the releases under both the caller's diagnostics phase budget and one of its own, and names whatever it stopped short of. Both budgets decide whether the next read may *start*, not when the walk ends, so neither is a worst case: the ceiling is the budget plus the reads already committed to when it expires. Write it that way in any collector you add — a budget described as a total is a number a later reader will trust for sizing the step around it.
96+
- **Do not infer a child release's state from the parent's.** The Kubernetes `ApplicationDefinition` sets `release.cozystack.io/helm-install-disable-wait`, which the apiserver turns into `DisableWait` on both actions, so the parent reaches Ready without waiting for the manifests it applied, the child HelmReleases included (Helm still waits on its post-install hook Job) — and the suite waits on some addons by name while never naming others. An empty `.status.history` is therefore judged against that release's own Ready condition: Ready with no history is a status shape the helper can no longer read and fails the run, not-Ready with no history is a release that never completed an action and is reported as not inspected. Read readiness before the history, not after, or a release that finishes installing between the two reads presents as Ready with an empty history and fails the run for a shape it never had. That ordering closes the truncation half of the window and not all of it: `HelmReleaseStatus.ClearHistory()` empties history outright, and the controller that calls it is not a module this tree carries.
9297

9398
### 7. Test-Impact Analysis (TIA), default-on
9499

@@ -198,7 +203,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
198203
5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between.
199204
6. An HR-Ready assert has a floor: the release's own install timeout, not only the app's boot time. Below it a failed install reports as a bare assert timeout, and above it by less than a start margin it can, so **a number under the floor buys a shorter red run at the price of the diagnosis**. Whether to leave a suite under it is a per-suite call, made against what that suite's red runs cost, unless no budget inside the E2E job's cap can clear the floor at all, as with kafka's 7h install timeout; either way, a Chainsaw assert left under it says why in-line. Derive the floor from [Sizing an HR-Ready budget](#sizing-an-hr-ready-budget) rather than copying a neighbour's number, and justify a raise in-line. The same goes for any other wait longer than the ones around it, HR-Ready or not: it carries its reason in-line.
200205
7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`.
201-
8. If it touches parent-HR behavior, add the `status.history` remediation guard (`hack/e2e-chainsaw/_lib/remediation-guard.sh`).
206+
8. If it touches HelmRelease install/upgrade behavior, add the `status.history` guard: fail the run on a `failed` or `uninstalled` Snapshot of a release whose install strategy is `RetryOnFailure`, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in `hack/e2e-chainsaw/_lib/remediation-guard.sh`; the policy that weighs the footprint against `.spec.install.strategy.name`, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in `cozy_guard_helmrelease` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path `cozy_report_helmrelease_remediation` as well, or the runs the guard exists for are the runs it does not see.
202207
9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch.
203208
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`.
204209

‎hack/e2e-chainsaw/_lib/remediation-guard.sh‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,28 @@
1313
# tracks the Helm release state: deployed, superseded, failed, uninstalled,
1414
# and so on. A remediation cycle leaves the footprint behind: a snapshot
1515
# with status "uninstalled" (from install/upgrade remediation) or "failed"
16-
# (Helm release failure that remediation then uninstalled). Those stay in
17-
# history even after a subsequent successful reinstall.
16+
# (a Helm action that failed). Those stay in history after a subsequent
17+
# successful reinstall or retry, with one exception: a failed install that
18+
# the next upgrade recovers is marked superseded by Helm, so its Snapshot
19+
# no longer reads "failed" and nothing here can see it.
20+
#
21+
# "Bounded" is literal. The controller truncates history on every in-sync
22+
# reconcile, and again before it retries or remediates a failed release:
23+
# on a release whose strategy is a retry it keeps the newest
24+
# five Snapshots, and on any other it keeps those down to the previous
25+
# deployed or superseded one, falling back to that same five when history
26+
# holds no such Snapshot to cut at. A footprint therefore outlives the
27+
# reconcile that follows it, not an arbitrary number of them. Reading
28+
# history right after the actions that wrote it, which is what an e2e run
29+
# does, sees it; reading it after the release has moved on may not.
1830
#
1931
# helmrelease_has_remediation_cycle takes a newline-delimited list of
2032
# snapshot statuses (whatever the caller extracted via kubectl -o jsonpath
2133
# or equivalent) and returns 0 (detected) when any entry is "failed" or
2234
# "uninstalled", 1 otherwise. Empty input is treated as "no history yet,
23-
# no cycle observed".
35+
# no cycle observed". Whether a detected footprint fails a run is the
36+
# caller's decision, because it depends on the release's own strategy:
37+
# see cozy_guard_helmrelease in run-kubernetes.sh.
2438

2539
# The jsonpath a caller hands to kubectl to read those statuses, one per
2640
# line. It lives here rather than inline at the call site so that it is a
@@ -35,6 +49,7 @@
3549
HELMRELEASE_HISTORY_JSONPATH='{range .status.history[*]}{.status}{"\n"}{end}'
3650

3751
helmrelease_has_remediation_cycle() {
52+
local statuses
3853
statuses="$1"
3954
if [ -z "${statuses}" ]; then
4055
return 1

0 commit comments

Comments
 (0)