Skip to content

Commit 80a2090

Browse files
committed
fix(kubernetes): retry the tenant CNI install instead of uninstalling it
The tenant cilium HelmRelease carried no install strategy, so Flux applied its default, RemediateOnFailure, which uninstalls the release between attempts. Cilium is the tenant cluster's only CNI, so every cycle takes it away and gives it back when the next install applies the manifests again, and retries: -1 lets that repeat without end. What sets a cycle off is an install that outruns its budget, which can happen while the tenant still has no registered node, since the operator Deployment cannot become Available until one exists. Remediation neither causes that delay nor is fixed by this change; what it adds is a repeated teardown of the CNI on top of a delay that resolves itself once the node arrives. This removes the amplifier, not the cause. RetryOnFailure keeps the applied manifests in place and retries a failed install as an upgrade, so the rollout completes on its own once a worker registers. Both actions carry the strategy because the retry of a failed install runs as an upgrade, which would otherwise fall back to upgrade remediation. It is the strategy and retry interval cozystack-api and cozystack-operator already set on every HelmRelease they generate, except that they leave remediation nil where this release keeps it. The other tenant addons the chart renders still carry the remediation-only shape; this one is where the teardown removes the cluster's only CNI rather than churning a single component. The cost is that a genuine upgrade failure no longer rolls back either, it retries on the failed manifests instead. That is forced, since the controller cannot tell a retried install from a real upgrade, and what it replaces was an unbounded rollback-and-retry flap rather than a stable safety net. The remediation blocks stay, since they cover the branches no strategy reaches: an absent release, which consults the install block, and an out-of-sync one, which consults upgrade. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent ee26a50 commit 80a2090

2 files changed

Lines changed: 61 additions & 0 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/cilium.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,11 +51,30 @@ spec:
5151
storageNamespace: cozy-cilium
5252
interval: 5m
5353
timeout: 10m
54+
{{- /* Flux's default install strategy remediates a failed install by
55+
uninstalling the release, which takes the tenant's only CNI away
56+
between attempts. RetryOnFailure drops that teardown: the manifests
57+
stay applied and the failed install is retried as an upgrade. Both
58+
actions carry the strategy because that retry runs as an upgrade and
59+
would otherwise fall back to upgrade remediation. The cost is that a
60+
genuine upgrade failure no longer rolls back either, which is forced,
61+
since the controller cannot tell a retried install from a real one,
62+
and what it replaces was an unbounded rollback-and-retry flap. This
63+
removes an amplifier, not a cause: an install that outruns its budget
64+
because no node has registered yet stays a problem. The remediation
65+
blocks cover the branches no strategy reaches, an absent release for
66+
install and an out-of-sync one for upgrade, so retries: -1 stays. */}}
5467
install:
5568
createNamespace: true
69+
strategy:
70+
name: RetryOnFailure
71+
retryInterval: 30s
5672
remediation:
5773
retries: -1
5874
upgrade:
75+
strategy:
76+
name: RetryOnFailure
77+
retryInterval: 30s
5978
remediation:
6079
retries: -1
6180
values:
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
suite: Cilium install retry strategy tests
2+
templates:
3+
- templates/helmreleases/cilium.yaml
4+
values:
5+
- values-ci.yaml
6+
tests:
7+
# Flux's default strategy remediates a failed install by uninstalling the
8+
# release, so a missed install budget on the tenant CNI takes the cluster's
9+
# only CNI away for the window until the next install applies the manifests
10+
# again, once per cycle for as long as retries allows. RetryOnFailure drops
11+
# the teardown and retries as an upgrade instead.
12+
- it: retries a failed install instead of uninstalling the CNI
13+
asserts:
14+
- equal:
15+
path: spec.install.strategy.name
16+
value: RetryOnFailure
17+
- equal:
18+
path: spec.install.strategy.retryInterval
19+
value: 30s
20+
21+
# The retry of a failed install runs as an upgrade, so without a strategy on
22+
# the upgrade action the second attempt falls back to upgrade remediation.
23+
- it: keeps retrying on the upgrade action the install retry runs as
24+
asserts:
25+
- equal:
26+
path: spec.upgrade.strategy.name
27+
value: RetryOnFailure
28+
- equal:
29+
path: spec.upgrade.strategy.retryInterval
30+
value: 30s
31+
32+
# The strategy governs a failed release. Two other branches consult the retry
33+
# count instead and refuse to act once it is exhausted: an absent release,
34+
# which reads the install block, and an out-of-sync one, which reads upgrade.
35+
- it: keeps unlimited retries for the branches no strategy reaches
36+
asserts:
37+
- equal:
38+
path: spec.install.remediation.retries
39+
value: -1
40+
- equal:
41+
path: spec.upgrade.remediation.retries
42+
value: -1

0 commit comments

Comments
 (0)