Repository navigation
Commit a27c341
committed
fix(tenant): retry failed monitoring and gateway installs in place
Both releases carry Flux's default install strategy, and install
remediation is an uninstall in every case Flux offers. Neither disables
hooks on it, so a remediated install runs the app chart's post-delete
cleanup Job against a release whose only fault was missing a readiness
budget.
The monitoring Job deletes PVCs labelled
apps.cozystack.io/application.name=<release>-system. That label is not
incidental: the metrics and logs claim templates stamp it precisely so
the operator copies it onto the storage PVCs, and the cleanup Job
selects on it to stay release-scoped. A slow first install therefore
deletes the tenant's metrics and logs, and nothing reconstructs them.
The gateway Job deletes the cozystack-acme-account Secret. That key
regenerates on the next install, so this one costs no data. It costs a
re-registration against the ACME provider, and repeated cycles run into
its rate limits, which can hold certificate issuance for the tenant down
for hours.
RetryOnFailure keeps the manifests applied and retries the failed
release, so no uninstall happens and neither cleanup Job runs. Both
actions carry the strategy because the controller performs that retry as
an upgrade, so on install alone the retry's own failure falls through to
upgrade remediation. Two things are traded away. A real upgrade failure
now retries rather than rolls back, which gives up no stable safety net,
since with retries: -1 and the default strategy upgrade remediation
already rolled back and retried without end. And the uninstall was the
only automatic clean slate, so an install wedged on a bad resource state
retries against that state until an operator removes the release. The
remediation blocks stay because an absent release consults install and
an out-of-sync one consults upgrade, both by retry count rather than by
strategy.
The chart's etcd and seaweedfs releases lose storage the same way and
are outside this change. Beyond those four, computeplane, ingress and
info remain on the default strategy, but their charts ship no
destructive post-delete hook, so remediation costs them a wasteful
teardown rather than data. That is a different question from this one.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>1 parent 605030b commit a27c341
3 files changed
Lines changed: 154 additions & 0 deletions
File tree
- packages/apps/tenant
- templates
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
22 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
23 | 45 | | |
24 | 46 | | |
25 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
26 | 51 | | |
27 | 52 | | |
28 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
33 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
34 | 56 | | |
35 | 57 | | |
36 | 58 | | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
37 | 62 | | |
38 | 63 | | |
39 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
0 commit comments