Skip to content

Commit 79ceaf1

Browse files
authored
fix(harbor): retry the nested system install instead of uninstalling it (#3621)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Harbor's app chart renders a nested HelmRelease that owns the whole workload: the Deployments, the inline RedisFailover and its volume. It had no install strategy, so it got Flux's default, and install remediation is an uninstall in every case Flux offers. A first install that misses its budget gets torn down, and the next one provisions the redis volume and runs the master election again from scratch. That happens on a loaded runner. harbor-core waits on redis, redis waits on its volume and the master election, and the nginx Deployment hits the default `progressDeadlineSeconds` of 600s. Flux reads that as a stalled resource and fails the install early. The e2e wait budget is 10m, the same as the deadline, so after a teardown the second attempt has less time than a first install needs. `RetryOnFailure` keeps the manifests applied and retries as an upgrade. Nothing is deleted, so the volume, the elected master and harbor-core's startup all survive, and the retry costs 30s instead of a full install. A Deployment clears `ProgressDeadlineExceeded` by itself once its pods go ready, so the retry lands as soon as the stack converges. This drops the teardown, not the deadline. The suite applies the release in one step and waits for it in later ones, so part of the budget is still left when the fail-early hits. That leftover is useless to a fresh install and usable to a retry that resumes. Whether the deadline or the budget also needs moving is a separate question, and an early deadline only becomes safe once a failed install stops being torn down. In production the win holds without any of that: a slow first install no longer deletes the Redis volume and the database. Both actions carry the strategy. The controller runs the retry of a failed install as an upgrade, so with install alone the retry's own failure falls through to upgrade remediation. The remediation blocks stay: an absent release consults install, an out-of-sync one consults upgrade, both by retry count rather than by strategy. It goes on the nested release, not the parent. The parent is built from the Application by the apiserver, which already stamps this strategy on both actions. The nested one owns the workload and the volume a remediation deletes. Same shape as the tenant CNI and CSI releases in #3552 and #3581, and the unit test follows theirs. Relates to #3445. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(harbor): retry a failed Harbor install instead of uninstalling the release, so a slow first install no longer deletes the Redis volume and starts over ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Harbor installations and upgrades now automatically retry every 30 seconds when failures occur. * Existing unlimited remediation retries remain available for both operations. * **Tests** * Added automated coverage to verify Harbor retry behavior and remediation settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents d956536 + f4dc5d4 commit 79ceaf1

2 files changed

Lines changed: 104 additions & 0 deletions

File tree

‎packages/apps/harbor/templates/harbor.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,38 @@ spec:
8989
namespace: cozy-system
9090
interval: 5m
9191
timeout: 15m
92+
{{- /* This nested release owns the whole harbor workload: the Deployments,
93+
the inline RedisFailover and its volume. Flux's default install
94+
strategy remediates a failed install by uninstalling, and install
95+
remediation is an uninstall in every case Flux offers, so a first
96+
install that goes over budget is torn down between attempts and the
97+
next one re-provisions the redis PVC and re-runs the master election
98+
from zero. It goes over budget on a busy runner: harbor-core blocks on
99+
redis, and the nginx Deployment reaches the default
100+
progressDeadlineSeconds of 600s, which Flux reads as a stalled
101+
resource and fails the install early. RetryOnFailure keeps the
102+
manifests applied and retries the failed release, so the convergence
103+
already under way continues instead of restarting. Both actions carry
104+
the strategy because the controller performs that retry as an upgrade
105+
and the retry's own failure would otherwise fall through to upgrade
106+
remediation. The cost is that a genuine upgrade failure retries rather
107+
than rolls back, which gives up no stable safety net: with retries: -1
108+
and the default strategy, upgrade remediation already rolled back and
109+
retried without end. The remediation blocks stay because an absent
110+
release consults install and an out-of-sync one consults upgrade, both
111+
by retry count rather than by strategy. The parent release that
112+
renders this one already carries the same strategy, stamped by the
113+
apiserver that builds it from the Application. */}}
92114
install:
115+
strategy:
116+
name: RetryOnFailure
117+
retryInterval: 30s
93118
remediation:
94119
retries: -1
95120
upgrade:
121+
strategy:
122+
name: RetryOnFailure
123+
retryInterval: 30s
96124
remediation:
97125
retries: -1
98126
valuesFrom:
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
suite: nested system HelmRelease install retry strategy
2+
3+
release:
4+
name: harbor
5+
namespace: tenant-test
6+
7+
# helm-unittest renders every template in the chart, so the platform-injected
8+
# globals consumed by the ingress/httproute/bucket templates must be supplied
9+
# even though only templates/harbor.yaml is asserted on.
10+
set:
11+
_namespace:
12+
host: example.org
13+
ingress: nginx
14+
gateway: ""
15+
seaweedfs: seaweedfs
16+
_cluster:
17+
solver: http01
18+
19+
# templates/harbor.yaml renders a Secret, this HelmRelease and three
20+
# WorkloadMonitors, so the release is selected by kind rather than by document
21+
# index: a template gaining a document ahead of it moves the index onto another
22+
# object, and these asserts would then red for a reason that has nothing to do
23+
# with the strategy they exist to pin. The selector names what it wants, so it
24+
# keeps pointing at the release wherever the release ends up, and a selector
25+
# that matches nothing fails loudly rather than quietly matching the wrong
26+
# document.
27+
tests:
28+
# The workload lives in this nested release: the harbor Deployments, the
29+
# inline RedisFailover and its PVC. Install remediation is an uninstall in
30+
# every case Flux offers, so under the default strategy an install that goes
31+
# over budget deletes all of it, and the next attempt re-provisions the redis
32+
# volume and re-runs the master election from zero.
33+
- it: retries a failed install instead of uninstalling the harbor workload
34+
template: templates/harbor.yaml
35+
documentSelector:
36+
path: kind
37+
value: HelmRelease
38+
asserts:
39+
- equal:
40+
path: spec.install.strategy.name
41+
value: RetryOnFailure
42+
- equal:
43+
path: spec.install.strategy.retryInterval
44+
value: 30s
45+
46+
# The controller performs the retry of a failed install as an upgrade, so
47+
# without a strategy here that retry's own failure falls through to upgrade
48+
# remediation instead of retrying again.
49+
- it: carries the strategy on the upgrade action the retry runs as
50+
template: templates/harbor.yaml
51+
documentSelector:
52+
path: kind
53+
value: HelmRelease
54+
asserts:
55+
- equal:
56+
path: spec.upgrade.strategy.name
57+
value: RetryOnFailure
58+
- equal:
59+
path: spec.upgrade.strategy.retryInterval
60+
value: 30s
61+
62+
# The strategy governs a failed release. An absent release and an out-of-sync
63+
# one take different branches, which consult the retry count instead and stop
64+
# acting once it is exhausted.
65+
- it: keeps unlimited retries for the branches the strategy does not reach
66+
template: templates/harbor.yaml
67+
documentSelector:
68+
path: kind
69+
value: HelmRelease
70+
asserts:
71+
- equal:
72+
path: spec.install.remediation.retries
73+
value: -1
74+
- equal:
75+
path: spec.upgrade.remediation.retries
76+
value: -1

0 commit comments

Comments
 (0)