Skip to content

Commit f4dc5d4

Browse files
committed
fix(harbor): retry the nested system install instead of uninstalling it
The harbor app chart renders a nested HelmRelease that owns the whole workload: the harbor Deployments, the inline RedisFailover and its volume. It carried Flux's default install strategy, and install remediation is an uninstall in every case Flux offers, so a first install that went over budget was torn down between attempts and the next one re-provisioned the redis volume and re-ran the master election from zero. That is reachable on a loaded runner. harbor-core blocks on redis while redis waits for its volume and its master election, and the nginx Deployment reaches the default progressDeadlineSeconds of 600s, which Flux reads as a stalled resource and fails the install early. The e2e wait budget for the release is the same 600s, so one teardown cycle consumes all of it and the second attempt cannot finish however fast it converges. RetryOnFailure keeps the manifests applied and retries the failed release, so the convergence already under way continues instead of restarting. Both actions carry the strategy because the controller performs the retry of a failed install as an upgrade, so on install alone the retry's own failure falls through to upgrade remediation. The cost is that a real upgrade failure retries rather than rolls back, which gives up no stable safety net: with retries: -1 and the default strategy, upgrade remediation already rolled back and retried without end. The remediation blocks stay because an absent release consults install and an out-of-sync one consults upgrade, both by retry count rather than by strategy. The change lands on the nested release rather than on the parent: the parent is built from the Application by the apiserver, which already stamps this strategy on both actions, while the nested release is the one that owns the workload and the volume a remediation deletes. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 605030b commit f4dc5d4

2 files changed

Lines changed: 104 additions & 0 deletions

File tree

‎packages/apps/harbor/templates/harbor.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,38 @@ spec:
8989
namespace: cozy-system
9090
interval: 5m
9191
timeout: 15m
92+
{{- /* This nested release owns the whole harbor workload: the Deployments,
93+
the inline RedisFailover and its volume. Flux's default install
94+
strategy remediates a failed install by uninstalling, and install
95+
remediation is an uninstall in every case Flux offers, so a first
96+
install that goes over budget is torn down between attempts and the
97+
next one re-provisions the redis PVC and re-runs the master election
98+
from zero. It goes over budget on a busy runner: harbor-core blocks on
99+
redis, and the nginx Deployment reaches the default
100+
progressDeadlineSeconds of 600s, which Flux reads as a stalled
101+
resource and fails the install early. RetryOnFailure keeps the
102+
manifests applied and retries the failed release, so the convergence
103+
already under way continues instead of restarting. Both actions carry
104+
the strategy because the controller performs that retry as an upgrade
105+
and the retry's own failure would otherwise fall through to upgrade
106+
remediation. The cost is that a genuine upgrade failure retries rather
107+
than rolls back, which gives up no stable safety net: with retries: -1
108+
and the default strategy, upgrade remediation already rolled back and
109+
retried without end. The remediation blocks stay because an absent
110+
release consults install and an out-of-sync one consults upgrade, both
111+
by retry count rather than by strategy. The parent release that
112+
renders this one already carries the same strategy, stamped by the
113+
apiserver that builds it from the Application. */}}
92114
install:
115+
strategy:
116+
name: RetryOnFailure
117+
retryInterval: 30s
93118
remediation:
94119
retries: -1
95120
upgrade:
121+
strategy:
122+
name: RetryOnFailure
123+
retryInterval: 30s
96124
remediation:
97125
retries: -1
98126
valuesFrom:
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
suite: nested system HelmRelease install retry strategy
2+
3+
release:
4+
name: harbor
5+
namespace: tenant-test
6+
7+
# helm-unittest renders every template in the chart, so the platform-injected
8+
# globals consumed by the ingress/httproute/bucket templates must be supplied
9+
# even though only templates/harbor.yaml is asserted on.
10+
set:
11+
_namespace:
12+
host: example.org
13+
ingress: nginx
14+
gateway: ""
15+
seaweedfs: seaweedfs
16+
_cluster:
17+
solver: http01
18+
19+
# templates/harbor.yaml renders a Secret, this HelmRelease and three
20+
# WorkloadMonitors, so the release is selected by kind rather than by document
21+
# index: a template gaining a document ahead of it moves the index onto another
22+
# object, and these asserts would then red for a reason that has nothing to do
23+
# with the strategy they exist to pin. The selector names what it wants, so it
24+
# keeps pointing at the release wherever the release ends up, and a selector
25+
# that matches nothing fails loudly rather than quietly matching the wrong
26+
# document.
27+
tests:
28+
# The workload lives in this nested release: the harbor Deployments, the
29+
# inline RedisFailover and its PVC. Install remediation is an uninstall in
30+
# every case Flux offers, so under the default strategy an install that goes
31+
# over budget deletes all of it, and the next attempt re-provisions the redis
32+
# volume and re-runs the master election from zero.
33+
- it: retries a failed install instead of uninstalling the harbor workload
34+
template: templates/harbor.yaml
35+
documentSelector:
36+
path: kind
37+
value: HelmRelease
38+
asserts:
39+
- equal:
40+
path: spec.install.strategy.name
41+
value: RetryOnFailure
42+
- equal:
43+
path: spec.install.strategy.retryInterval
44+
value: 30s
45+
46+
# The controller performs the retry of a failed install as an upgrade, so
47+
# without a strategy here that retry's own failure falls through to upgrade
48+
# remediation instead of retrying again.
49+
- it: carries the strategy on the upgrade action the retry runs as
50+
template: templates/harbor.yaml
51+
documentSelector:
52+
path: kind
53+
value: HelmRelease
54+
asserts:
55+
- equal:
56+
path: spec.upgrade.strategy.name
57+
value: RetryOnFailure
58+
- equal:
59+
path: spec.upgrade.strategy.retryInterval
60+
value: 30s
61+
62+
# The strategy governs a failed release. An absent release and an out-of-sync
63+
# one take different branches, which consult the retry count instead and stop
64+
# acting once it is exhausted.
65+
- it: keeps unlimited retries for the branches the strategy does not reach
66+
template: templates/harbor.yaml
67+
documentSelector:
68+
path: kind
69+
value: HelmRelease
70+
asserts:
71+
- equal:
72+
path: spec.install.remediation.retries
73+
value: -1
74+
- equal:
75+
path: spec.upgrade.remediation.retries
76+
value: -1

0 commit comments

Comments
 (0)