Repository navigation
Description
The status.history guard in hack/e2e-chainsaw/_lib/run-kubernetes.sh (cozy_guard_helmrelease, via helmrelease_has_remediation_cycle in hack/e2e-chainsaw/_lib/remediation-guard.sh) fails a run on a failed or uninstalled Snapshot of a RetryOnFailure release. It misses the case it is most often cited for: an install that fails once, for example on the install timeout, and recovers on the next attempt.
Mechanism, at helm-controller v1.5.0 (the version internal/fluxinstall/manifests/fluxcd.yaml installs) and Helm v4.1.1 (the version it builds against):
- Under
RetryOnFailure, a failed release is retried as an upgrade (internal/reconcile/atomic_release.go:469-474). - With no deployed release, Helm upgrades from the last release, which is the failed install (
pkg/action/upgrade.go:258-262), and marks itsupersededonce the upgrade succeeds (:511-512). - helm-controller records both observations and replaces the old Snapshot in place (
internal/reconcile/release.go:96-110).
The history then reads deployed, superseded, which by status looks the same as an ordinary upgrade, and the guard passes. A failed Snapshot survives only when a deployed release already existed (a failed upgrade), or when the install failed at least twice in a row.
Two places still claim otherwise and predate the guard rework:
docs/agents/e2e-testing.md:77says a timed-out attempt leaves afailedSnapshot that the guard fails on, "so a success arriving from a later attempt is one the suite has already failed on". After a single timed-out install that the next attempt recovers, the Snapshot issupersededand the suite passes.- The predicate itself, which matches statuses only.
Detection needs a signal beyond the status list. Counting history entries does not work: the OIDC path of run_kubernetes_test upgrades the parent on purpose (cozy_switch_and_assert_oidc_custom_config) before the guard runs. Candidates to evaluate against a real run: a superseded Snapshot whose successor has the same chartVersion, configDigest and ociDigest (a retry changes neither chart nor values, while a spec-driven upgrade changes the config digest), or the InstallFailed event on the HelmRelease, which is lossy and ages out. Either needs a measured run before it can fail a run, so that the guard does not go red on a legitimate upgrade.
hack/run-kubernetes-remediation_test.bats has a test, "single-release guard cannot see a failed install the next attempt recovered", that pins the current pass. It is the test to flip when detection lands.
Activity