You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 415430b
Browse filesBrowse the repository at this point in the historyBrowse files
fix(e2e): fail on a failed release of cilium and csi as on the parent, and report addon remediation (cozystack#3594)
## What this PR does
The history guard in the tenant Kubernetes e2e suite checked only the
parent `kubernetes-<test>` HelmRelease. A failed action on one of the
addon releases it installs left nothing the suite looked at, so a run
showed only the downstream symptom, usually the node-join deadline, and
never named the release behind it.
The guard now reads every addon release of the parent. It selects them
from the cluster by name prefix, so a release added to the chart later
is covered too. The rule:
- A `failed` or `uninstalled` Snapshot fails the run on any release
whose install strategy is `RetryOnFailure`. That is the parent (the
aggregated apiserver stamps it on every Application HelmRelease) and the
tenant `cilium` and `csi`.
- The same footprint on the other addons, which run the default strategy
with `retries: -1`, is printed as a note and the run stays green. For
the CRD addons that configuration is tracked in cozystack#3555.
A transient `cilium` or `csi` failure that later recovers on a retry
still turns the run red, by design, as long as its Snapshot still reads
`failed`. One recovery leaves no such Snapshot: when an install fails
and the very next attempt succeeds, Helm marks the failed install
`superseded`, because there is no deployed release to supersede, and the
history reads like an ordinary upgrade. The guard cannot see that case
from the statuses alone. The code and the docs now say so, a test pins
it, and detection is left to a follow-up.
An earlier revision of this PR tried to tell a teardown apart from a
retried failure and made only `uninstalled` fatal. That split does not
hold. In helm-controller v1.5.0, the version the platform installs, a
release with an active retry strategy never remediates.
`atomic_release.go` returns an upgrade for a failed release before it
reaches the remediation switch, and the two other paths that uninstall
(a changed release target, and deletion) both clear history afterwards.
So on these releases `uninstalled` never shows up, and `failed` is the
only signal. The parent keeps the rule it already had on main, and
`cilium` and `csi` now follow it too.
The failure paths get the same reading. The guard runs after the
assertions, while the deadlines a failed addon breaks end the script
earlier, so the same classification is also printed, without a verdict,
from the node-join collector and from the exit handler. That handler is
now armed at the tenant CR apply, so the management-cluster waits before
the tenant answers are covered too. Smaller changes that came with this:
- Every guard read is bounded, and a failed read is never folded into
"no history".
- The parent must read back a non-empty strategy. An empty value there
means the jsonpath stopped resolving, and without this check every
footprint would quietly turn into a note.
- The history read uses `HELMRELEASE_HISTORY_JSONPATH`, and the
upstream-type pin in `internal/fluxcontract` now accepts the bounded
wrapper the read goes through.
`docs/agents/e2e-testing.md` §6 and checklist item 8 describe the new
rule.
On related issues: this adds the `local` that cozystack#3774 reports missing in
`helmrelease_has_remediation_cycle`, but not the sweep over the other
`_lib/` helpers that the issue also asks for. For cozystack#3773 it rewrites the
guard comment the issue quotes. The waits without a backstop are not
touched.
Closescozystack#3553.
### Screenshots
Not a UI change.
### Downstream repositories
I walked the trigger map against the diff. It touches
`hack/e2e-chainsaw/_lib/`, `hack/*.bats`, one Go test under
`internal/fluxcontract`, two spend-order comments in the kubernetes
suites, and `docs/agents/e2e-testing.md`. No package, schema, CRD, make
target or user-facing document changes.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(e2e): the HelmRelease history guard now also covers the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place; other addons get a note. The same classification is printed on the failure paths where the guard itself cannot run.
```
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+9-4Lines changed: 9 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -84,11 +84,16 @@ The install test fails if **any** HelmRelease is not Ready. A toothless gate (ba
84
84
- Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR.
85
85
- Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`.
86
86
87
-
### 6. Assert the parent HelmRelease did not remediate — via `status.history`
87
+
### 6. Assert no HelmRelease failed an action it retries in place — via `status.history`
88
88
89
-
A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is a silent race we want to catch. **Do not** check `.status.installFailures` / `.status.upgradeFailures`: Flux's `ClearFailures()` zeroes those on every successful reconcile, so checking them after the HR is Ready is **vacuous** and passes against a reverted fix.
89
+
A HelmRelease that hit its wait timeoutand failed an action is a silent race we want to catch, and it stays one when a later retry recovers the release. **Do not** check `.status.installFailures` / `.status.upgradeFailures`: Flux's `ClearFailures()` zeroes those on every successful reconcile, so checking them after the HR is Ready is **vacuous** and passes against a reverted fix.
90
90
91
-
- Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile. Use the shared helper in `hack/e2e-chainsaw/_lib/remediation-guard.sh` (`helmrelease_has_remediation_cycle`) from a `script` step. Take the jsonpath from `HELMRELEASE_HISTORY_JSONPATH` in that same file rather than writing it out: `internal/fluxcontract`sources that file and runs the resulting value against the upstream Flux type, and a literal copied elsewhere is not covered by it.
91
+
- Inspect `.status.history` instead — a `failed` or `uninstalled` Snapshot survives a later successful reconcile, which is what makes it readable at all after the release is Ready. It does not survive an unbounded number of them: the controller truncates history on every in-sync reconcile and again before it retries or remediates a failed release, retaining the newest five Snapshots on a release whose strategy is a retry and, on any other, those down to the previous `deployed` or `superseded` Snapshot — falling back to the same newest-five cap when there is no such Snapshot to cut at. An e2e run takes only a handful of actions on each release, so the footprint is still present when the guard reads it — but a guard placed after further release actions, or one asserting on history days later, is reading a window and needs to say so. Use the shared helper in `hack/e2e-chainsaw/_lib/remediation-guard.sh` (`helmrelease_has_remediation_cycle`) from a `script` step. Take the jsonpath from `HELMRELEASE_HISTORY_JSONPATH` in that same file rather than writing it out: `internal/fluxcontract` sources that file and runs the resulting value against the upstream Flux type, and a literal copied elsewhere is not covered by it.
92
+
- **Fail on any footprint of a release that retries in place; report it on the others.** Under `RetryOnFailure` helm-controller never remediates: a failed action is retried as an upgrade with its manifests left in place, so the release never writes an `uninstalled` Snapshot of its own, and the `failed` one is the only record of the attempt. `cozy_guard_helmrelease` therefore fails the run on either status on a release whose `.spec.install.strategy.name` is `RetryOnFailure`: the parent, which the aggregated apiserver stamps that way like every Application HelmRelease (`pkg/registry/apps/application/rest.go`), and the tenant `cilium` and `csi`, whose templates set it. A retry that recovered still fails the run, by design, while the Snapshot still reads `failed`: the failed attempt happened, and on these releases nothing else records it. It does not after a failed install that the very next attempt recovers. With no deployed release to upgrade from, Helm upgrades from the failed one and marks it `superseded` on success, helm-controller rewrites that Snapshot in place, and the history reads `deployed`, `superseded` like any ordinary upgrade, so the guard passes. A failed upgrade keeps its `failed` Snapshot, since Helm supersedes the last deployed release instead, and so does an install that fails twice before recovering. That makes a late node join a red run here even when it lands inside the 29m node-join deadline: the tenant `cilium` install cannot complete before a worker registers, because its operator Deployment has nowhere to schedule, so workers arriving more than two of its 10m install timeouts after the install started leave a `failed` Snapshot, and the guard fails the run on it. This is accepted rather than carved out, since the chart itself treats an install outrunning its budget for want of a node as a problem, but the cause is the join and not the CNI. The guard's message says so and points at the Snapshot times in the describe output it prints. The other addon releases pair the default strategy with `retries: -1`, a configuration that declares repeated remediation their recovery path. Nothing in the suite measures how often they take it, and failing a whole bringup on a release doing what it is configured to do is how a guard ends up switched off, so the same footprint there is printed as a note and the run stays green. For the CRD addons that configuration is itself a defect, tracked in cozystack/cozystack#3555; tighten when a measured baseline or a strategy change makes the condition knowable.
93
+
- **The strategy is read through a jsonpath, so something has to notice when it stops resolving.** `kubectl -o jsonpath` prints nothing and exits 0 for a path that no longer matches, and empty takes the lenient branch — a rename of `spec.install.strategy` in a future helm-controller would report every footprint on the parent, `cilium` and `csi` as a note, on a green run, with the unit tests still passing because they feed the value from a fixture rather than through the jsonpath. An empty value is not by itself evidence of that, since most addons legitimately set no strategy, so the check lives on the one release whose strategy the cluster is known to set: the parent, where an empty value, or any value other than `RetryOnFailure`, fails the run. That is enough, because the same rename empties the field on every release at once.
94
+
- **"Covers the tenant addon HelmReleases" means reads and classifies, not can fail the run on.** Five conditions red a run: a `failed` or `uninstalled` Snapshot on a release that retries in place; a Ready HelmRelease reporting no history at all, which is a status shape the helper can no longer read; a read that failed, which is never folded into "no history"; an empty or non-`RetryOnFailure` `.spec.install.strategy.name` on the parent; and an addon prefix that selects no HelmRelease at all, which the passing path fails on rather than passing silently over nothing. Every other addon footprint is a note. Which releases a run has at all depends on what it enabled, since most addon templates are gated on their own `addons.<name>.enabled`, so the breadth of the reading and the breadth of the gate are not the same number.
95
+
- **The guard runs after the assertions, so the failure paths need the reading without the verdict.** Every deadline a failed child action surfaces as — a node that never turns Ready, a HelmRelease wait that expires, a PVC that never binds — ends the script under errexit well before the guard, so on exactly the runs such an action explains, the guard never executes and the history is named nowhere. `cozy_report_helmrelease_remediation` is the same classification with no verdict, called from the node-join collector and from the exit handler that covers every other failure. Arm that handler where the object it reads starts existing, not where the tenant becomes reachable: the parent HelmRelease is rendered from the CR at the apply, and everything between there and the tenant LB is a management-cluster wait on that release or on what it installs, so a handler armed after those waits misses the window in which an install action times out. It fails nothing: the run is already red, and the tenant crust-gather snapshot hangs off the caller's own exit status. It walks the releases under both the caller's diagnostics phase budget and one of its own, and names whatever it stopped short of. Both budgets decide whether the next read may *start*, not when the walk ends, so neither is a worst case: the ceiling is the budget plus the reads already committed to when it expires. Write it that way in any collector you add — a budget described as a total is a number a later reader will trust for sizing the step around it.
96
+
- **Do not infer a child release's state from the parent's.** The Kubernetes `ApplicationDefinition` sets `release.cozystack.io/helm-install-disable-wait`, which the apiserver turns into `DisableWait` on both actions, so the parent reaches Ready without waiting for the manifests it applied, the child HelmReleases included (Helm still waits on its post-install hook Job) — and the suite waits on some addons by name while never naming others. An empty `.status.history` is therefore judged against that release's own Ready condition: Ready with no history is a status shape the helper can no longer read and fails the run, not-Ready with no history is a release that never completed an action and is reported as not inspected. Read readiness before the history, not after, or a release that finishes installing between the two reads presents as Ready with an empty history and fails the run for a shape it never had. That ordering closes the truncation half of the window and not all of it: `HelmReleaseStatus.ClearHistory()` empties history outright, and the controller that calls it is not a module this tree carries.
92
97
93
98
### 7. Test-Impact Analysis (TIA), default-on
94
99
@@ -198,7 +203,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
198
203
5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between.
199
204
6. An HR-Ready assert has a floor: the release's own install timeout, not only the app's boot time. Below it a failed install reports as a bare assert timeout, and above it by less than a start margin it can, so **a number under the floor buys a shorter red run at the price of the diagnosis**. Whether to leave a suite under it is a per-suite call, made against what that suite's red runs cost, unless no budget inside the E2E job's cap can clear the floor at all, as with kafka's 7h install timeout; either way, a Chainsaw assert left under it says why in-line. Derive the floor from [Sizing an HR-Ready budget](#sizing-an-hr-ready-budget) rather than copying a neighbour's number, and justify a raise in-line. The same goes for any other wait longer than the ones around it, HR-Ready or not: it carries its reason in-line.
200
205
7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`.
201
-
8. If it touches parent-HR behavior, add the `status.history` remediation guard (`hack/e2e-chainsaw/_lib/remediation-guard.sh`).
206
+
8. If it touches HelmRelease install/upgrade behavior, add the `status.history` guard: fail the run on a `failed` or `uninstalled` Snapshot of a release whose install strategy is `RetryOnFailure`, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in `hack/e2e-chainsaw/_lib/remediation-guard.sh`; the policy that weighs the footprint against `.spec.install.strategy.name`, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in `cozy_guard_helmrelease` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path `cozy_report_helmrelease_remediation` as well, or the runs the guard exists for are the runs it does not see.
202
207
9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch.
203
208
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`.
0 commit comments