Repository navigation
Commit 0c4c11c
committed
fix(kubernetes): retry the tenant CSI install instead of uninstalling it
Flux defaults a HelmRelease to the RemediateOnFailure strategy, whose
install remediation is always an uninstall. A tenant CSI install that
misses its timeout was therefore torn down between attempts, taking the
CSI node plugin and the StorageClasses this release propagates with it,
and the next attempt restarted the DaemonSet rollout from nothing
instead of resuming one already under way.
That teardown races the rollout it is meant to recover. In one observed
run both DaemonSet pods were scheduled immediately and spent the whole
budget pulling images; the slower node started its last container six
seconds after the install timeout fired, and the uninstall removed it
one second later. Retrying in place keeps that work.
RetryOnFailure leaves the manifests applied and retries the failed
release. Both actions carry it because the controller picks the active
retry by .status.lastAttemptedReleaseAction and runs that retry as an
upgrade: set on install alone, the retry happens but its own failure
falls through to upgrade remediation, which rolls back by default with
no previous release to roll back to. The cost is that a real upgrade
failure now retries rather than rolls back, and that gives up no stable
safety net: with retries: -1 and the default rollback strategy, upgrade
remediation already rolled back and retried without end.
The remediation blocks stay for a separate reason. An absent release
consults install.remediation and an out-of-sync one consults upgrade,
both by retry count rather than by strategy, so retries: -1 is what
keeps those two branches acting.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>1 parent df157da commit 0c4c11c
2 files changed
Lines changed: 79 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
24 | 43 | | |
25 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
26 | 48 | | |
27 | 49 | | |
28 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
29 | 54 | | |
30 | 55 | | |
31 | 56 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
0 commit comments