Skip to content

Commit 92ec49e

Browse files
authored
fix(tenant): retry failed monitoring and gateway installs in place (#3633)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The tenant chart renders monitoring and gateway with no install strategy, so they get Flux's default, and install remediation is an uninstall in every case Flux offers. Neither sets `spec.uninstall.disableHooks`, which defaults to false. So a release that missed nothing but a readiness budget gets uninstalled, and the app chart's `post-delete` cleanup Job runs against it. Those Jobs delete on purpose, and they are right to. The monitoring one removes PVCs labelled `apps.cozystack.io/application.name=<release>-system`. That label is not incidental either: the metrics and logs claim templates stamp it so the operator copies it onto the storage PVCs, and the Job selects on it to stay release-scoped. Both ends were built to meet. The result is that a slow first install deletes the tenant's metrics and logs, and nothing reconstructs them. The gateway Job deletes the `cozystack-acme-account` Secret. That key regenerates, so this one costs no data. It costs a re-registration against the ACME provider, and repeated cycles run into its rate limits, which can hold certificate issuance for the tenant down for hours. Smaller stake, same mechanism, and the two are not one class: only one of them owns storage. `RetryOnFailure` keeps the manifests applied and retries the failed release, so no uninstall happens and neither Job runs. Both actions carry the strategy because the controller performs that retry as an upgrade, so with install alone the retry's own failure falls through to upgrade remediation. The remediation blocks stay: an absent release consults install, an out-of-sync one consults upgrade, both by retry count rather than by strategy. Two things are traded away. A real upgrade failure now retries instead of rolling back, which gives up no stable safety net, since with `retries: -1` the default strategy was already rolling back and re-upgrading without end. And the uninstall was the only automatic clean slate, so an install wedged on a bad resource state retries against that state until an operator removes the release. For a release that owns data that is the right exchange. This removes the consequence rather than the trigger, and for monitoring the trigger is worth naming separately: the parent HelmRelease here waits `10m`, while the child `monitoring-system` release sets `15m` deliberately, with a comment explaining that 10m is too short for the OIDC users-Job inside its `activeDeadlineSeconds: 900`. The parent's wait covers the child, so the parent can fail while the child is behaving exactly as designed. That mismatch is untouched here and wants its own change. The chart's etcd and seaweedfs releases lose storage the same way and are outside this change. Beyond those four, computeplane, ingress and info stay on the default strategy, but their charts ship no destructive `post-delete` hook, so remediation costs them a wasteful teardown rather than data. Same shape as #3552 and #3581. Relates to #3624. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(tenant): retry a failed monitoring or gateway install instead of uninstalling the release, so a slow first install no longer runs the cleanup hooks that delete the tenant's metrics and logs volumes or its ACME account key ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Reliability Improvements** * Monitoring and gateway deployments now automatically retry failed installs and upgrades every 30 seconds. * Existing unlimited remediation retries remain available, improving recovery from deployment issues. * **Validation** * Added automated coverage to verify retry behavior for both monitoring and gateway deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 36f83fa + a27c341 commit 92ec49e

3 files changed

Lines changed: 154 additions & 0 deletions

File tree

‎packages/apps/tenant/templates/gateway.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,35 @@ spec:
1919
namespace: cozy-system
2020
interval: 5m
2121
timeout: 10m
22+
{{- /* Install remediation is an uninstall in every case Flux offers, and this
23+
release does not disable hooks on it, so a remediated install runs the
24+
gateway chart's post-delete cleanup Job. That Job deletes the
25+
cozystack-acme-account Secret. The key regenerates on the next install,
26+
so unlike the storage-owning releases this costs no data: it costs a
27+
re-registration against the ACME provider, and repeated cycles run into
28+
its rate limits, which can hold certificate issuance for the tenant
29+
down for hours. RetryOnFailure keeps the manifests applied and retries
30+
the failed release instead, so no uninstall happens and the cleanup Job
31+
never runs. Both actions carry the strategy because the controller
32+
performs that retry as an upgrade. Two things are traded away: a real
33+
upgrade failure now retries rather than rolls back, which gives up no
34+
stable safety net, since with retries: -1 the default strategy already
35+
rolled back and retried without end; and the uninstall was the only
36+
automatic clean slate, so an install wedged on a bad resource state
37+
retries against it until an operator removes the release. The
38+
remediation blocks stay because an absent release consults install and
39+
an out-of-sync one consults upgrade, both by retry count rather than by
40+
strategy. */}}
2241
install:
42+
strategy:
43+
name: RetryOnFailure
44+
retryInterval: 30s
2345
remediation:
2446
retries: -1
2547
upgrade:
48+
strategy:
49+
name: RetryOnFailure
50+
retryInterval: 30s
2651
remediation:
2752
retries: -1
2853
valuesFrom:

‎packages/apps/tenant/templates/monitoring.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,35 @@ spec:
3030
namespace: cozy-system
3131
interval: 5m
3232
timeout: 10m
33+
{{- /* Install remediation is an uninstall in every case Flux offers, and this
34+
release does not disable hooks on it, so a remediated install runs the
35+
monitoring chart's post-delete cleanup Job. That Job deletes PVCs by
36+
apps.cozystack.io/application.name=<release>-system, which is the label
37+
the metrics and logs claim templates stamp so the operator copies it
38+
onto the storage PVCs. An install that only missed its readiness budget
39+
therefore takes the tenant's metrics and logs with it, and nothing
40+
reconstructs them. RetryOnFailure keeps the manifests applied and
41+
retries the failed release instead, so no uninstall happens and the
42+
cleanup Job never runs. Both actions carry the strategy because the
43+
controller performs that retry as an upgrade. Two things are traded
44+
away: a real upgrade failure now retries rather than rolls back, which
45+
gives up no stable safety net, since with retries: -1 the default
46+
strategy already rolled back and retried without end; and the uninstall
47+
was the only automatic clean slate, so an install wedged on a bad
48+
resource state retries against it until an operator removes the
49+
release. The remediation blocks stay because an absent release consults
50+
install and an out-of-sync one consults upgrade, both by retry count
51+
rather than by strategy. */}}
3352
install:
53+
strategy:
54+
name: RetryOnFailure
55+
retryInterval: 30s
3456
remediation:
3557
retries: -1
3658
upgrade:
59+
strategy:
60+
name: RetryOnFailure
61+
retryInterval: 30s
3762
remediation:
3863
retries: -1
3964
valuesFrom:
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
suite: monitoring and gateway tenant HelmReleases retry a failed install in place
2+
3+
# Named after the two releases rather than after a class. They do not share one:
4+
# monitoring loses storage and gateway loses a regenerable key. The tenant's
5+
# etcd and seaweedfs releases lose storage as well and are outside this suite,
6+
# so a class-shaped title here would print as PASS over members it never
7+
# looked at.
8+
#
9+
# Restrict rendering to the two templates asserted on, so a template this suite
10+
# makes no claim about cannot fail it. That is not hypothetical: at the time of
11+
# writing, dropping the restriction reds all four tests on keycloakgroups.yaml,
12+
# which errors with "index of untyped nil" on the `.Values._cluster` the
13+
# platform injects and this suite has no reason to set. Adding a template to
14+
# the list below is safe.
15+
templates:
16+
- templates/monitoring.yaml
17+
- templates/gateway.yaml
18+
19+
release:
20+
name: tenant-test
21+
namespace: tenant-test
22+
23+
set:
24+
monitoring: true
25+
gateway: true
26+
27+
tests:
28+
# Install remediation is an uninstall in every case Flux offers, and neither
29+
# release disables hooks on it, so a remediated install runs the app chart's
30+
# post-delete cleanup Job. The monitoring one deletes PVCs by the
31+
# apps.cozystack.io/application.name=<release>-system label, and the metrics
32+
# and logs claim templates stamp exactly that label so the operator copies it
33+
# onto the storage PVCs. An install that misses its budget therefore deletes
34+
# the tenant's metrics and logs, which nothing reconstructs.
35+
- it: retries a failed monitoring install instead of deleting the storage
36+
asserts:
37+
- template: templates/monitoring.yaml
38+
equal:
39+
path: spec.install.strategy.name
40+
value: RetryOnFailure
41+
- template: templates/monitoring.yaml
42+
equal:
43+
path: spec.install.strategy.retryInterval
44+
value: 30s
45+
46+
# The gateway cleanup Job deletes the cozystack-acme-account Secret. That key
47+
# regenerates on the next install, so the cost is re-registration against the
48+
# ACME provider and its rate limits rather than lost data. Same mechanism,
49+
# smaller stake, and worth fixing for the same reason: remediation invokes a
50+
# teardown written for an honest uninstall.
51+
- it: retries a failed gateway install instead of discarding the ACME account
52+
asserts:
53+
- template: templates/gateway.yaml
54+
equal:
55+
path: spec.install.strategy.name
56+
value: RetryOnFailure
57+
- template: templates/gateway.yaml
58+
equal:
59+
path: spec.install.strategy.retryInterval
60+
value: 30s
61+
62+
# The controller performs the retry of a failed install as an upgrade, so
63+
# without a strategy here that retry's own failure falls through to upgrade
64+
# remediation instead of retrying again.
65+
- it: carries the strategy on the upgrade action the retry runs as
66+
asserts:
67+
- template: templates/monitoring.yaml
68+
equal:
69+
path: spec.upgrade.strategy.name
70+
value: RetryOnFailure
71+
- template: templates/monitoring.yaml
72+
equal:
73+
path: spec.upgrade.strategy.retryInterval
74+
value: 30s
75+
- template: templates/gateway.yaml
76+
equal:
77+
path: spec.upgrade.strategy.name
78+
value: RetryOnFailure
79+
- template: templates/gateway.yaml
80+
equal:
81+
path: spec.upgrade.strategy.retryInterval
82+
value: 30s
83+
84+
# The strategy governs a failed release. An absent release and an out-of-sync
85+
# one take different branches, which consult the retry count instead and stop
86+
# acting once it is exhausted.
87+
- it: keeps unlimited retries for the branches the strategy does not reach
88+
asserts:
89+
- template: templates/monitoring.yaml
90+
equal:
91+
path: spec.install.remediation.retries
92+
value: -1
93+
- template: templates/monitoring.yaml
94+
equal:
95+
path: spec.upgrade.remediation.retries
96+
value: -1
97+
- template: templates/gateway.yaml
98+
equal:
99+
path: spec.install.remediation.retries
100+
value: -1
101+
- template: templates/gateway.yaml
102+
equal:
103+
path: spec.upgrade.remediation.retries
104+
value: -1

0 commit comments

Comments
 (0)