Skip to content

Commit b4e2031

Browse files
authored
fix(kubernetes): retry the tenant CSI install instead of uninstalling it (#3581)
## What this PR does The tenant CSI HelmRelease carried `remediation.retries: -1` with no strategy, so Flux applied its default, `RemediateOnFailure`. Install remediation is an uninstall in every case the API offers, so an install that missed its budget was torn down between attempts: the tenant lost its CSI node plugin and the StorageClasses this release propagates until a later attempt applied them again, and that attempt restarted the DaemonSet rollout instead of resuming one already under way. That teardown races the rollout it exists to recover, and the run recorded in #3576 shows how narrowly. Both DaemonSet pods were scheduled at the same second on both workers, so nothing was waiting on capacity; the ten minutes went to image pulls. The slower worker started its last container six seconds after the install timeout fired, and the uninstall removed it one second after that. `RetryOnFailure` leaves the manifests applied and retries the failed release. Both actions carry it because the controller picks the active retry from `.status.lastAttemptedReleaseAction` and performs that retry as an upgrade: set on install alone, the retry still happens, but its own failure is then handled by upgrade remediation, whose default is a rollback with no previous release to roll back to. The `remediation` blocks stay, because an absent release consults `install.remediation` and an out-of-sync one consults `upgrade`, both by retry count rather than by strategy. This is the change #3552 makes for the tenant CNI, and the reason for putting the strategy on both actions is the one given there. It is also the shape upstream prescribes: the "Recommended settings" block in the [HelmRelease spec shipped with the helm-controller this installs](https://github.com/fluxcd/helm-controller/blob/v1.5.1/docs/spec/v2/helmreleases.md#recommended-settings) puts `strategy.name: RetryOnFailure` under both `install` and `upgrade` for production deployments. Nothing changes on the test side, and that is deliberate rather than deferred. Because no manifest is removed at the attempt boundary, the rollout continues across it and the kubelet finishes what it started, while the retry re-applies the same manifests every 30s. The release therefore reports Ready once the rollout completes rather than once a whole attempt has fit inside the timeout, so what the existing e2e wait has to outlast is the rollout and no longer an attempt boundary that discards it. That widens what the wait tolerates without making it unbounded: a rollout slower than the wait itself still fails, and this does not claim otherwise. Scope. #3576 records more than this fixes, so this does not close it. The other tenant addon HelmReleases this chart renders keep the remediation-only shape; CSI is the one this changes. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: The trigger map was walked against the diff. The change is a HelmRelease template and a chart test, not a `values.schema.json` field, a `values.yaml` default, a version enum, a `release.prefix`, an `ApplicationDefinition`, a chart source kind or a namespace. ### Release note ```release-note fix(kubernetes): the tenant CSI HelmRelease now retries a failed install instead of being uninstalled between attempts, so a failed install no longer takes the tenant's CSI node plugin and its propagated StorageClasses away while it retries. The same strategy applies to the upgrade action, which it has to: a retried install runs as an upgrade. One consequence is operator-visible — a failed CSI upgrade in a tenant cluster now retries the new manifests every 30s instead of rolling back to the previous release. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when installing or upgrading the CSI component by automatically retrying failed actions every 30 seconds. * Retains unlimited remediation retries to help recover from transient failures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 0dae3ca + 0c4c11c commit b4e2031

2 files changed

Lines changed: 79 additions & 0 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/csi.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,36 @@ spec:
2121
key: super-admin.svc
2222
targetNamespace: cozy-csi
2323
storageNamespace: cozy-csi
24+
{{- /* Flux defaults to the RemediateOnFailure strategy, and install
25+
remediation is always an uninstall, so an install that outruns the
26+
spec.timeout above is torn down between attempts: the tenant loses
27+
its CSI node plugin and the StorageClasses this release propagates,
28+
and the next attempt restarts the DaemonSet rollout instead of
29+
resuming one already under way. RetryOnFailure keeps the manifests
30+
in place and retries the failed release. Both actions carry it
31+
because the controller picks the active retry by
32+
.status.lastAttemptedReleaseAction and performs that retry as an
33+
upgrade, so on install alone the retry's own failure falls through
34+
to upgrade remediation, a rollback with no previous release to
35+
reach. The cost is that a real upgrade failure retries rather than
36+
rolls back, which gives up no stable safety net: with retries: -1
37+
and the default rollback strategy, upgrade remediation already
38+
rolled back and retried without end. The remediation blocks stay
39+
for a different reason, that an absent release
40+
consults install and an out-of-sync one consults upgrade, both by
41+
retry count rather than by strategy, so retries: -1 keeps them
42+
open. */}}
2443
install:
2544
createNamespace: true
45+
strategy:
46+
name: RetryOnFailure
47+
retryInterval: 30s
2648
remediation:
2749
retries: -1
2850
upgrade:
51+
strategy:
52+
name: RetryOnFailure
53+
retryInterval: 30s
2954
remediation:
3055
retries: -1
3156
values:
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
suite: csi.yaml install retry strategy
2+
3+
templates:
4+
- templates/helmreleases/csi.yaml
5+
6+
values:
7+
- values/common.yaml
8+
9+
tests:
10+
# Install remediation is an uninstall in every case Flux offers, so under the
11+
# default strategy an install that misses its budget takes the tenant's CSI
12+
# node plugin and propagated StorageClasses away until a later attempt lands
13+
# them again. RetryOnFailure keeps the manifests applied and retries instead.
14+
- it: retries a failed install instead of uninstalling the tenant CSI
15+
release:
16+
name: kubernetes-test
17+
namespace: tenant-test
18+
asserts:
19+
- equal:
20+
path: spec.install.strategy.name
21+
value: RetryOnFailure
22+
- equal:
23+
path: spec.install.strategy.retryInterval
24+
value: 30s
25+
26+
# The controller performs the retry of a failed install as an upgrade, so
27+
# without a strategy here that retry's own failure falls through to upgrade
28+
# remediation instead of retrying again.
29+
- it: carries the strategy on the upgrade action the retry runs as
30+
release:
31+
name: kubernetes-test
32+
namespace: tenant-test
33+
asserts:
34+
- equal:
35+
path: spec.upgrade.strategy.name
36+
value: RetryOnFailure
37+
- equal:
38+
path: spec.upgrade.strategy.retryInterval
39+
value: 30s
40+
41+
# The strategy governs a failed release. An absent release and an out-of-sync
42+
# one take different branches, which consult the retry count instead and stop
43+
# acting once it is exhausted.
44+
- it: keeps unlimited retries for the branches the strategy does not reach
45+
release:
46+
name: kubernetes-test
47+
namespace: tenant-test
48+
asserts:
49+
- equal:
50+
path: spec.install.remediation.retries
51+
value: -1
52+
- equal:
53+
path: spec.upgrade.remediation.retries
54+
value: -1

0 commit comments

Comments
 (0)