Skip to content

Commit a27c341

Browse files
committed
fix(tenant): retry failed monitoring and gateway installs in place
Both releases carry Flux's default install strategy, and install remediation is an uninstall in every case Flux offers. Neither disables hooks on it, so a remediated install runs the app chart's post-delete cleanup Job against a release whose only fault was missing a readiness budget. The monitoring Job deletes PVCs labelled apps.cozystack.io/application.name=<release>-system. That label is not incidental: the metrics and logs claim templates stamp it precisely so the operator copies it onto the storage PVCs, and the cleanup Job selects on it to stay release-scoped. A slow first install therefore deletes the tenant's metrics and logs, and nothing reconstructs them. The gateway Job deletes the cozystack-acme-account Secret. That key regenerates on the next install, so this one costs no data. It costs a re-registration against the ACME provider, and repeated cycles run into its rate limits, which can hold certificate issuance for the tenant down for hours. RetryOnFailure keeps the manifests applied and retries the failed release, so no uninstall happens and neither cleanup Job runs. Both actions carry the strategy because the controller performs that retry as an upgrade, so on install alone the retry's own failure falls through to upgrade remediation. Two things are traded away. A real upgrade failure now retries rather than rolls back, which gives up no stable safety net, since with retries: -1 and the default strategy upgrade remediation already rolled back and retried without end. And the uninstall was the only automatic clean slate, so an install wedged on a bad resource state retries against that state until an operator removes the release. The remediation blocks stay because an absent release consults install and an out-of-sync one consults upgrade, both by retry count rather than by strategy. The chart's etcd and seaweedfs releases lose storage the same way and are outside this change. Beyond those four, computeplane, ingress and info remain on the default strategy, but their charts ship no destructive post-delete hook, so remediation costs them a wasteful teardown rather than data. That is a different question from this one. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 605030b commit a27c341

3 files changed

Lines changed: 154 additions & 0 deletions

File tree

‎packages/apps/tenant/templates/gateway.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,35 @@ spec:
1919
namespace: cozy-system
2020
interval: 5m
2121
timeout: 10m
22+
{{- /* Install remediation is an uninstall in every case Flux offers, and this
23+
release does not disable hooks on it, so a remediated install runs the
24+
gateway chart's post-delete cleanup Job. That Job deletes the
25+
cozystack-acme-account Secret. The key regenerates on the next install,
26+
so unlike the storage-owning releases this costs no data: it costs a
27+
re-registration against the ACME provider, and repeated cycles run into
28+
its rate limits, which can hold certificate issuance for the tenant
29+
down for hours. RetryOnFailure keeps the manifests applied and retries
30+
the failed release instead, so no uninstall happens and the cleanup Job
31+
never runs. Both actions carry the strategy because the controller
32+
performs that retry as an upgrade. Two things are traded away: a real
33+
upgrade failure now retries rather than rolls back, which gives up no
34+
stable safety net, since with retries: -1 the default strategy already
35+
rolled back and retried without end; and the uninstall was the only
36+
automatic clean slate, so an install wedged on a bad resource state
37+
retries against it until an operator removes the release. The
38+
remediation blocks stay because an absent release consults install and
39+
an out-of-sync one consults upgrade, both by retry count rather than by
40+
strategy. */}}
2241
install:
42+
strategy:
43+
name: RetryOnFailure
44+
retryInterval: 30s
2345
remediation:
2446
retries: -1
2547
upgrade:
48+
strategy:
49+
name: RetryOnFailure
50+
retryInterval: 30s
2651
remediation:
2752
retries: -1
2853
valuesFrom:

‎packages/apps/tenant/templates/monitoring.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,35 @@ spec:
3030
namespace: cozy-system
3131
interval: 5m
3232
timeout: 10m
33+
{{- /* Install remediation is an uninstall in every case Flux offers, and this
34+
release does not disable hooks on it, so a remediated install runs the
35+
monitoring chart's post-delete cleanup Job. That Job deletes PVCs by
36+
apps.cozystack.io/application.name=<release>-system, which is the label
37+
the metrics and logs claim templates stamp so the operator copies it
38+
onto the storage PVCs. An install that only missed its readiness budget
39+
therefore takes the tenant's metrics and logs with it, and nothing
40+
reconstructs them. RetryOnFailure keeps the manifests applied and
41+
retries the failed release instead, so no uninstall happens and the
42+
cleanup Job never runs. Both actions carry the strategy because the
43+
controller performs that retry as an upgrade. Two things are traded
44+
away: a real upgrade failure now retries rather than rolls back, which
45+
gives up no stable safety net, since with retries: -1 the default
46+
strategy already rolled back and retried without end; and the uninstall
47+
was the only automatic clean slate, so an install wedged on a bad
48+
resource state retries against it until an operator removes the
49+
release. The remediation blocks stay because an absent release consults
50+
install and an out-of-sync one consults upgrade, both by retry count
51+
rather than by strategy. */}}
3352
install:
53+
strategy:
54+
name: RetryOnFailure
55+
retryInterval: 30s
3456
remediation:
3557
retries: -1
3658
upgrade:
59+
strategy:
60+
name: RetryOnFailure
61+
retryInterval: 30s
3762
remediation:
3863
retries: -1
3964
valuesFrom:
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
suite: monitoring and gateway tenant HelmReleases retry a failed install in place
2+
3+
# Named after the two releases rather than after a class. They do not share one:
4+
# monitoring loses storage and gateway loses a regenerable key. The tenant's
5+
# etcd and seaweedfs releases lose storage as well and are outside this suite,
6+
# so a class-shaped title here would print as PASS over members it never
7+
# looked at.
8+
#
9+
# Restrict rendering to the two templates asserted on, so a template this suite
10+
# makes no claim about cannot fail it. That is not hypothetical: at the time of
11+
# writing, dropping the restriction reds all four tests on keycloakgroups.yaml,
12+
# which errors with "index of untyped nil" on the `.Values._cluster` the
13+
# platform injects and this suite has no reason to set. Adding a template to
14+
# the list below is safe.
15+
templates:
16+
- templates/monitoring.yaml
17+
- templates/gateway.yaml
18+
19+
release:
20+
name: tenant-test
21+
namespace: tenant-test
22+
23+
set:
24+
monitoring: true
25+
gateway: true
26+
27+
tests:
28+
# Install remediation is an uninstall in every case Flux offers, and neither
29+
# release disables hooks on it, so a remediated install runs the app chart's
30+
# post-delete cleanup Job. The monitoring one deletes PVCs by the
31+
# apps.cozystack.io/application.name=<release>-system label, and the metrics
32+
# and logs claim templates stamp exactly that label so the operator copies it
33+
# onto the storage PVCs. An install that misses its budget therefore deletes
34+
# the tenant's metrics and logs, which nothing reconstructs.
35+
- it: retries a failed monitoring install instead of deleting the storage
36+
asserts:
37+
- template: templates/monitoring.yaml
38+
equal:
39+
path: spec.install.strategy.name
40+
value: RetryOnFailure
41+
- template: templates/monitoring.yaml
42+
equal:
43+
path: spec.install.strategy.retryInterval
44+
value: 30s
45+
46+
# The gateway cleanup Job deletes the cozystack-acme-account Secret. That key
47+
# regenerates on the next install, so the cost is re-registration against the
48+
# ACME provider and its rate limits rather than lost data. Same mechanism,
49+
# smaller stake, and worth fixing for the same reason: remediation invokes a
50+
# teardown written for an honest uninstall.
51+
- it: retries a failed gateway install instead of discarding the ACME account
52+
asserts:
53+
- template: templates/gateway.yaml
54+
equal:
55+
path: spec.install.strategy.name
56+
value: RetryOnFailure
57+
- template: templates/gateway.yaml
58+
equal:
59+
path: spec.install.strategy.retryInterval
60+
value: 30s
61+
62+
# The controller performs the retry of a failed install as an upgrade, so
63+
# without a strategy here that retry's own failure falls through to upgrade
64+
# remediation instead of retrying again.
65+
- it: carries the strategy on the upgrade action the retry runs as
66+
asserts:
67+
- template: templates/monitoring.yaml
68+
equal:
69+
path: spec.upgrade.strategy.name
70+
value: RetryOnFailure
71+
- template: templates/monitoring.yaml
72+
equal:
73+
path: spec.upgrade.strategy.retryInterval
74+
value: 30s
75+
- template: templates/gateway.yaml
76+
equal:
77+
path: spec.upgrade.strategy.name
78+
value: RetryOnFailure
79+
- template: templates/gateway.yaml
80+
equal:
81+
path: spec.upgrade.strategy.retryInterval
82+
value: 30s
83+
84+
# The strategy governs a failed release. An absent release and an out-of-sync
85+
# one take different branches, which consult the retry count instead and stop
86+
# acting once it is exhausted.
87+
- it: keeps unlimited retries for the branches the strategy does not reach
88+
asserts:
89+
- template: templates/monitoring.yaml
90+
equal:
91+
path: spec.install.remediation.retries
92+
value: -1
93+
- template: templates/monitoring.yaml
94+
equal:
95+
path: spec.upgrade.remediation.retries
96+
value: -1
97+
- template: templates/gateway.yaml
98+
equal:
99+
path: spec.install.remediation.retries
100+
value: -1
101+
- template: templates/gateway.yaml
102+
equal:
103+
path: spec.upgrade.remediation.retries
104+
value: -1

0 commit comments

Comments
 (0)