Repository navigation
Commit f4dc5d4
committed
fix(harbor): retry the nested system install instead of uninstalling it
The harbor app chart renders a nested HelmRelease that owns the whole
workload: the harbor Deployments, the inline RedisFailover and its
volume. It carried Flux's default install strategy, and install
remediation is an uninstall in every case Flux offers, so a first
install that went over budget was torn down between attempts and the
next one re-provisioned the redis volume and re-ran the master election
from zero.
That is reachable on a loaded runner. harbor-core blocks on redis while
redis waits for its volume and its master election, and the nginx
Deployment reaches the default progressDeadlineSeconds of 600s, which
Flux reads as a stalled resource and fails the install early. The e2e
wait budget for the release is the same 600s, so one teardown cycle
consumes all of it and the second attempt cannot finish however fast it
converges.
RetryOnFailure keeps the manifests applied and retries the failed
release, so the convergence already under way continues instead of
restarting. Both actions carry the strategy because the controller
performs the retry of a failed install as an upgrade, so on install
alone the retry's own failure falls through to upgrade remediation. The
cost is that a real upgrade failure retries rather than rolls back,
which gives up no stable safety net: with retries: -1 and the default
strategy, upgrade remediation already rolled back and retried without
end. The remediation blocks stay because an absent release consults
install and an out-of-sync one consults upgrade, both by retry count
rather than by strategy.
The change lands on the nested release rather than on the parent: the
parent is built from the Application by the apiserver, which already
stamps this strategy on both actions, while the nested release is the
one that owns the workload and the volume a remediation deletes.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>1 parent 605030b commit f4dc5d4
2 files changed
Lines changed: 104 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
92 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
93 | 118 | | |
94 | 119 | | |
95 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
96 | 124 | | |
97 | 125 | | |
98 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
0 commit comments