Skip to content

Commit 50ed307

Browse files
fix(backupstrategy-controller): gate the default MongoDB Strategy on its CRD
Upgrading a cluster to a build that carries the new MongoDB backup driver wedges the backupstrategy-controller release permanently: Helm upgrade failed: resource mapping not found for name: "cozy-default-mongodb" ... no matches for kind "MongoDB" in version "strategy.backups.cozystack.io/v1alpha1"; ensure CRDs are installed first This chart ships its strategy.backups.cozystack.io CRDs as ordinary templates, and Helm resolves every document in a release manifest through the cluster's RESTMapper in Build() BEFORE applying any of them. A Strategy CR whose CRD is not served yet therefore fails Build(), so the release applies ZERO objects — including the CRD that would have made the mapping resolve — and every retry renders the same manifest. Nothing recovers on its own. The chart already knows about this trap: it is why every default Strategy gates on the bucket name. That gate covers the first install of a fresh cluster, where nothing is resolvable and no Strategy renders at all. It cannot cover shipping a NEW kind to a chart that is already installed, which is what happened here: the bucket name resolved releases ago, so the first upgraded revision renders the new CRD and the new Strategy CR together and wedges. Gate the CR on its CRD being served, via a crdEstablished helper that asks about the CRD OBJECT rather than listing the custom kind — `lookup` on a kind the apiserver does not serve is a render error, not an empty result, which would trade this wedge for a different permanent failure. apiextensions.k8s.io/v1 is always served, so that lookup can only answer yes or no. Established=True rather than mere existence, because that is when the kind enters the RESTMapper. Convergence afterwards is the mechanism the bucket gate already relies on and is NOT the HelmRelease interval: once the CRD is served, the controller's DefaultObjectsGate sees the kind mapped and the CR absent and forces a real Helm upgrade whose render includes it. Kinds that do not map at all are skipped there (meta.IsNoMatchError), which is what stops the gate forcing upgrades against a render that cannot yet produce the object. Found by the release upgrade E2E lane on the v1.6.2 → main path, where `mongodbs.strategy.backups.cozystack.io` is the only definition added since the tag. Verified against a live cluster that is in exactly the pre-upgrade state (seven older strategy CRDs, no mongodbs): `helm template --dry-run=server` of main's chart fails with the error above and produces zero documents, while the same render with this change succeeds with 21 documents and the Strategy correctly withheld. The helper's two branches were probed the same way — a served CRD answers true, an absent one answers empty. The new unit case pins the gate offline through the external-S3 path, where the bucket name resolves without the offline override so only the CRD gate can hold the CR back; removing the gate turns exactly that case red. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent c8e4d97 commit 50ed307

3 files changed

Lines changed: 118 additions & 14 deletions

File tree

‎packages/system/backupstrategy-controller/templates/_helpers.tpl‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,3 +122,66 @@
122122
{{- end -}}
123123
{{- end -}}
124124
{{- end -}}
125+
126+
{{/*
127+
Answer, for one CRD this chart ships as a template, whether the apiserver
128+
already serves its kind.
129+
130+
Why a default Strategy needs this on top of the bucket-name gate. This chart
131+
ships its strategy.backups.cozystack.io CRDs as ordinary templates
132+
(templates/crds.yaml globs definitions/*.yaml), and Helm resolves EVERY
133+
document in a release manifest through the cluster's RESTMapper in Build()
134+
before it applies any of them. A Strategy CR whose CRD is not served yet
135+
therefore fails Build(), and the release applies ZERO objects — including the
136+
CRDs that would have made the mapping resolve. Every retry renders the same
137+
manifest, so the failure is permanent rather than a race.
138+
139+
The bucket-name gate covers the FIRST INSTALL of a new cluster, where nothing
140+
is resolvable yet and no Strategy renders at all. It cannot cover the case
141+
this helper exists for: adding a NEW strategy kind to a chart that is already
142+
installed. There the bucket name resolved long ago, so the first upgraded
143+
revision renders the new CRD and the new Strategy CR together and wedges the
144+
release exactly as above. That is not hypothetical — it is how the MongoDB
145+
driver reached main, and the upgrade E2E lane caught it on the v1.6.2 → main
146+
path with `no matches for kind "MongoDB"`.
147+
148+
Asking about the CRD OBJECT rather than listing the custom kind is
149+
deliberate. `lookup` on a kind the apiserver does not serve is not an empty
150+
result — it is a render ERROR ("unable to get apiresource"), which would trade
151+
the wedge for a different permanent failure. apiextensions.k8s.io/v1 is always
152+
served, so this lookup can only answer yes or no. Established=True rather than
153+
mere existence, because a CRD is only in the RESTMapper once it is; a
154+
not-yet-established CRD costs one retried revision here instead of a failed
155+
one, and Flux retries.
156+
157+
Convergence after a skip is the same mechanism the bucket gate relies on and
158+
is NOT the HelmRelease interval (see the bucketName helper): once the CRD is
159+
served, the controller's DefaultObjectsGate sees the kind mapped and the CR
160+
absent and forces a real Helm upgrade, whose render then includes it. The gate
161+
skips kinds that do not map at all (meta.IsNoMatchError), which is what keeps
162+
it from forcing upgrades against a render that cannot yet produce the object.
163+
164+
bucketNameOverride short-circuits the lookup for the same reason it does in
165+
bucketName: it marks an offline render — `helm template`, a CI diff, a
166+
helm-unittest case — where there is no apiserver to ask and the operator has
167+
declared that cluster-derived state should be treated as resolved. Live
168+
deploys go through Flux and never set it.
169+
170+
Every default Strategy added from here on should gate on this helper as well
171+
as on the bucket name. The siblings that predate it are not wedged today only
172+
because their CRDs already shipped in an earlier release.
173+
*/}}
174+
{{- define "backupstrategy-controller.crdEstablished" -}}
175+
{{- if .root.Values.backupStorage.bucketNameOverride -}}
176+
true
177+
{{- else -}}
178+
{{- $crd := lookup "apiextensions.k8s.io/v1" "CustomResourceDefinition" "" .crd -}}
179+
{{- if and $crd $crd.status -}}
180+
{{- range $c := (default (list) $crd.status.conditions) -}}
181+
{{- if and (eq $c.type "Established") (eq (toString $c.status) "True") -}}
182+
true
183+
{{- end -}}
184+
{{- end -}}
185+
{{- end -}}
186+
{{- end -}}
187+
{{- end -}}

‎packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml‎

Lines changed: 27 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -6,32 +6,45 @@
66
lives on the MongoDB application (declared by the chart as "s3-storage" when
77
backup.enabled=true), not here.
88

9-
It is gated on $bucketName all the same, and NOT because it needs the value.
10-
The gate is what keeps this chart installable on a fresh cluster. This chart
9+
It is gated all the same, and NOT because it needs a bucket value. This chart
1110
ships its own strategy.backups.cozystack.io CRDs as ordinary templates
1211
(templates/crds.yaml globs definitions/*.yaml), and Helm resolves every
1312
document in a release manifest through the cluster's RESTMapper in Build()
1413
BEFORE it applies any of them. An ungated Strategy CR therefore has no
15-
mapping on the first install, Build() fails with "no matches for kind", and
16-
the release applies ZERO objects — including the CRDs that would have made
17-
the mapping resolve. Nothing recovers from that: every retry renders the same
18-
manifest, so the failure is permanent rather than a race, and Helm's kind
19-
sorter cannot help because it orders the apply, not the Build.
14+
mapping while its CRD is still absent, Build() fails with "no matches for
15+
kind", and the release applies ZERO objects — including the CRDs that would
16+
have made the mapping resolve. Nothing recovers from that: every retry
17+
renders the same manifest, so the failure is permanent rather than a race,
18+
and Helm's kind sorter cannot help because it orders the apply, not the
19+
Build.
2020

21-
So every default Strategy in this chart must be absent from the first render.
22-
$bucketName is empty until the COSI BucketClaim status is reconciled (see
23-
templates/_helpers.tpl), which is exactly the window where the CRDs are being
24-
applied; by the time DefaultObjectsGate forces a second revision the CRDs
25-
exist and the mapping resolves. The seven sibling strategies got this for
26-
free by needing the bucket name. This one has to ask for it.
21+
So this Strategy must be absent from any render whose cluster cannot yet map
22+
it, and there are TWO such renders, not one.
23+
24+
A fresh install is the first. $bucketName is empty until the COSI BucketClaim
25+
status is reconciled (see templates/_helpers.tpl), which is exactly the window
26+
where the CRDs are being applied; by the time DefaultObjectsGate forces a
27+
second revision the CRDs exist and the mapping resolves. The seven sibling
28+
strategies get that for free by needing the bucket name.
29+
30+
Adding this kind to an ALREADY-INSTALLED chart is the second, and the bucket
31+
gate is no help there: the bucket name resolved releases ago, so the first
32+
upgraded revision renders the new CRD and this CR together and wedges the
33+
release. That is not a hypothetical — this Strategy shipped to main that way,
34+
and the upgrade E2E lane caught it on the v1.6.2 → main path. Hence the
35+
explicit CRD gate below, which asks whether the apiserver serves the kind and
36+
answers for both renders; see the crdEstablished helper for why it asks about
37+
the CRD object rather than listing the kind, and for how the release converges
38+
afterwards.
2739

2840
Precondition (surfaced by the driver as a clear Ready=False, not a silent
2941
hang): the MongoDB application must be deployed with backup.enabled=true so
3042
the percona-backup-mongodb agents run and the "s3-storage" entry exists. See
3143
docs/operations/backup-classes.md.
3244
*/}}
3345
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
34-
{{- if $bucketName -}}
46+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "mongodbs.strategy.backups.cozystack.io") -}}
47+
{{- if and $bucketName $crdEstablished -}}
3548
apiVersion: strategy.backups.cozystack.io/v1alpha1
3649
kind: MongoDB
3750
metadata:

‎packages/system/backupstrategy-controller/tests/rendering_test.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,10 @@ tests:
9090
template: templates/backupclass-default.yaml
9191

9292
- it: "all Strategy CRs and the Velero BSL render once a bucket name resolves"
93+
# bucketNameOverride marks an offline render, and the MongoDB count below is
94+
# what pins that the CRD gate honours it: without that short-circuit no
95+
# `helm template` or CI diff could ever produce this Strategy, since neither
96+
# has an apiserver to ask about the CRD.
9397
set:
9498
backupStorage.bucketNameOverride: test-bucket
9599
asserts:
@@ -150,6 +154,30 @@ tests:
150154
value: ext-bucket
151155
template: templates/strategy-mariadb-default.yaml
152156

157+
- it: "MongoDB Strategy gates on its CRD too, so shipping the kind to an installed chart cannot wedge the release"
158+
# provisionBucket=false is what makes this case load-bearing: it resolves
159+
# $bucketName WITHOUT the offline bucketNameOverride, which is the shape of a
160+
# cluster that has been running for releases — bucket gate open, and only the
161+
# CRD gate still able to hold the CR back. helm-unittest renders with no
162+
# apiserver, so the CRD lookup answers "absent", which is the same answer a
163+
# real cluster gives on the upgrade that first ships that CRD. Rendering the
164+
# CR there fails Build() for the WHOLE release ("no matches for kind
165+
# MongoDB") and applies zero objects, including the CRD that would have
166+
# fixed it, so a count of 1 here is that wedge, green.
167+
set:
168+
backupStorage.provisionBucket: false
169+
backupStorage.bucketName: ext-bucket
170+
asserts:
171+
- hasDocuments:
172+
count: 0
173+
template: templates/strategy-mongodb-default.yaml
174+
# A sibling whose CRD shipped in an earlier release keeps rendering, which
175+
# is what makes the count above an assertion about the MongoDB gate rather
176+
# than about a chart-wide render failure.
177+
- hasDocuments:
178+
count: 1
179+
template: templates/strategy-cnpg-default.yaml
180+
153181
- it: "velero.bslEnabled=false: Velero strategies and BSL drop out while other strategies keep rendering"
154182
set:
155183
backupStorage.bucketNameOverride: test-bucket

0 commit comments

Comments
 (0)