Skip to content

Commit 0c4c11c

Browse files
committed
fix(kubernetes): retry the tenant CSI install instead of uninstalling it
Flux defaults a HelmRelease to the RemediateOnFailure strategy, whose install remediation is always an uninstall. A tenant CSI install that misses its timeout was therefore torn down between attempts, taking the CSI node plugin and the StorageClasses this release propagates with it, and the next attempt restarted the DaemonSet rollout from nothing instead of resuming one already under way. That teardown races the rollout it is meant to recover. In one observed run both DaemonSet pods were scheduled immediately and spent the whole budget pulling images; the slower node started its last container six seconds after the install timeout fired, and the uninstall removed it one second later. Retrying in place keeps that work. RetryOnFailure leaves the manifests applied and retries the failed release. Both actions carry it because the controller picks the active retry by .status.lastAttemptedReleaseAction and runs that retry as an upgrade: set on install alone, the retry happens but its own failure falls through to upgrade remediation, which rolls back by default with no previous release to roll back to. The cost is that a real upgrade failure now retries rather than rolls back, and that gives up no stable safety net: with retries: -1 and the default rollback strategy, upgrade remediation already rolled back and retried without end. The remediation blocks stay for a separate reason. An absent release consults install.remediation and an out-of-sync one consults upgrade, both by retry count rather than by strategy, so retries: -1 is what keeps those two branches acting. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent df157da commit 0c4c11c

2 files changed

Lines changed: 79 additions & 0 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/csi.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,36 @@ spec:
2121
key: super-admin.svc
2222
targetNamespace: cozy-csi
2323
storageNamespace: cozy-csi
24+
{{- /* Flux defaults to the RemediateOnFailure strategy, and install
25+
remediation is always an uninstall, so an install that outruns the
26+
spec.timeout above is torn down between attempts: the tenant loses
27+
its CSI node plugin and the StorageClasses this release propagates,
28+
and the next attempt restarts the DaemonSet rollout instead of
29+
resuming one already under way. RetryOnFailure keeps the manifests
30+
in place and retries the failed release. Both actions carry it
31+
because the controller picks the active retry by
32+
.status.lastAttemptedReleaseAction and performs that retry as an
33+
upgrade, so on install alone the retry's own failure falls through
34+
to upgrade remediation, a rollback with no previous release to
35+
reach. The cost is that a real upgrade failure retries rather than
36+
rolls back, which gives up no stable safety net: with retries: -1
37+
and the default rollback strategy, upgrade remediation already
38+
rolled back and retried without end. The remediation blocks stay
39+
for a different reason, that an absent release
40+
consults install and an out-of-sync one consults upgrade, both by
41+
retry count rather than by strategy, so retries: -1 keeps them
42+
open. */}}
2443
install:
2544
createNamespace: true
45+
strategy:
46+
name: RetryOnFailure
47+
retryInterval: 30s
2648
remediation:
2749
retries: -1
2850
upgrade:
51+
strategy:
52+
name: RetryOnFailure
53+
retryInterval: 30s
2954
remediation:
3055
retries: -1
3156
values:
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
suite: csi.yaml install retry strategy
2+
3+
templates:
4+
- templates/helmreleases/csi.yaml
5+
6+
values:
7+
- values/common.yaml
8+
9+
tests:
10+
# Install remediation is an uninstall in every case Flux offers, so under the
11+
# default strategy an install that misses its budget takes the tenant's CSI
12+
# node plugin and propagated StorageClasses away until a later attempt lands
13+
# them again. RetryOnFailure keeps the manifests applied and retries instead.
14+
- it: retries a failed install instead of uninstalling the tenant CSI
15+
release:
16+
name: kubernetes-test
17+
namespace: tenant-test
18+
asserts:
19+
- equal:
20+
path: spec.install.strategy.name
21+
value: RetryOnFailure
22+
- equal:
23+
path: spec.install.strategy.retryInterval
24+
value: 30s
25+
26+
# The controller performs the retry of a failed install as an upgrade, so
27+
# without a strategy here that retry's own failure falls through to upgrade
28+
# remediation instead of retrying again.
29+
- it: carries the strategy on the upgrade action the retry runs as
30+
release:
31+
name: kubernetes-test
32+
namespace: tenant-test
33+
asserts:
34+
- equal:
35+
path: spec.upgrade.strategy.name
36+
value: RetryOnFailure
37+
- equal:
38+
path: spec.upgrade.strategy.retryInterval
39+
value: 30s
40+
41+
# The strategy governs a failed release. An absent release and an out-of-sync
42+
# one take different branches, which consult the retry count instead and stop
43+
# acting once it is exhausted.
44+
- it: keeps unlimited retries for the branches the strategy does not reach
45+
release:
46+
name: kubernetes-test
47+
namespace: tenant-test
48+
asserts:
49+
- equal:
50+
path: spec.install.remediation.retries
51+
value: -1
52+
- equal:
53+
path: spec.upgrade.remediation.retries
54+
value: -1

0 commit comments

Comments
 (0)