Skip to content

Commit 8bd6d91

Browse files
fix(backups): gate every default Strategy CR on its CRD (#4929)
## What this PR does Upgrade from v1.6.x to v1.7 leaves `backupstrategy-controller` HelmRelease in `UpgradeFailed`: `no matches for kind "Kafka" in version "strategy.backups.cozystack.io/v1alpha1"`, same for Rabbitmq and Redis, and none of the new strategy CRDs get created. CRDs of this chart are ordinary templates, and Helm maps every object of a release through the RESTMapper before applying any of them. Default Strategy CRs for kafka, rabbitmq and redis were gated only on the bucket name, so they render next to their not yet served CRDs, Build() fails, nothing is applied and every retry renders the same manifest. Same happens on a first install with external S3 (`provisionBucket: false`) for all older default strategies, bucket name comes from values there. MongoDB default strategy already has the `crdEstablished` gate, this PR adds it to all other default strategies. On an installed cluster nothing changes. Otherwise first revision creates the CRDs and skips the CRs, then DefaultObjectsGate forces the follow-up upgrade that renders them. CRDs stay templates so later CRD changes still reach the cluster on upgrade. Short explanation is added to `docs/operations/backup-classes.md`. To keep this from coming back there is `hack/backupstrategy-crd-gate.bats`: it renders the chart with the bucket gate open and no CRD served and fails on any Strategy CR from any template, so a new `strategy-*-default.yaml` without the gate goes red without anyone updating a list. It also checks that every CR gates on the CRD this chart ships for its kind. helm-unittest covers both halves of the gate for all 11 templates. Checked on a throwaway kind cluster: - v1.6.4 installed, then upgraded: current main fails with the error above, this branch passes with all 11 CRDs Established and the follow-up upgrade creates the rest - modified CRD is applied on a later upgrade - fresh install converges, and fresh install with external S3 converges here while main fails with "no matches for kind" for eight kinds Fixes #4925. Needs backport to release-1.7, both commits apply there cleanly. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): backupstrategy-controller release no longer gets stuck in UpgradeFailed on upgrade from v1.6 and no longer fails a fresh install with external S3, strategy CRDs are created first and default strategies follow on the next reconcile ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Default backup strategies are now created only after their required resource types are available and the backup bucket is resolved. This helps prevent installation and upgrade issues, including when using externally managed S3 storage. * **Documentation** * Clarified when default backup strategies are created during installation and upgrades. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 11b1963 + f47bab7 commit 8bd6d91

14 files changed

Lines changed: 355 additions & 29 deletions

‎docs/operations/backup-classes.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,8 @@ The same trap sits one release earlier, on the Secret everything else depends on
209209

210210
Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the **bucket's** `bucket-<name>-system` HelmRelease. Once the bucket name resolves it checks that every object `cozy-default` routes to exists and stamps the same pair on the **`backupstrategy-controller`** HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready.
211211

212+
Each default `Strategy` CR also waits for its own CRD. The chart ships the `strategy.backups.cozystack.io` CRDs as templates in the same release, and Helm resolves every object's kind before it applies any, so a CR rendered next to a CRD the apiserver does not serve yet would fail the whole release, CRDs included. Each strategy template therefore renders only once its CRD is `Established`. That holds the CRs back on the first render of every install, including an external-S3 one where the bucket name is known up front, and on the upgrade that first ships a new strategy kind: that revision creates the CRDs, and the gate's forced upgrade above renders the CRs.
213+
212214
The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while `spec.suspend` is true, so it logs the skip and leaves the force counter alone. A release left suspended (`cozyhr suspend`) is therefore never repaired until it is resumed.
213215

214216
Watch it with:

‎hack/backupstrategy-crd-gate.bats‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
#!/usr/bin/env bats
2+
# backupstrategy-controller ships its strategy.backups.cozystack.io CRDs as
3+
# ordinary templates in the same release as the default Strategy CRs. Helm maps
4+
# every document of a release through the RESTMapper before it applies any, so
5+
# one Strategy CR whose CRD is not served yet fails the whole release with "no
6+
# matches for kind" and applies nothing, the CRDs included. That happens on the
7+
# upgrade that first ships a kind, and on the first render of an external-S3
8+
# install, where the bucket gate is already open. Every default Strategy must
9+
# therefore also gate on its CRD being established (the crdEstablished helper).
10+
#
11+
# helm-unittest can only pin the templates someone remembered to list. These
12+
# cases discover the templates from the render itself, so a new
13+
# strategy-*-default.yaml without the CRD gate fails here without anyone
14+
# touching a test.
15+
#
16+
# Harness compatibility note: CI runs this file under Bats through
17+
# `make bats-unit-tests`. It also remains compatible with the narrower legacy
18+
# `hack/cozytest.sh` translator, so it avoids `run`, `$status`, `$output`,
19+
# `skip`, and setup()/teardown(). Paths stay repo-root-relative for both
20+
# runners.
21+
#
22+
# Run with: bats hack/backupstrategy-crd-gate.bats
23+
24+
load test_helper
25+
26+
CHART=packages/system/backupstrategy-controller
27+
STRATEGY_API='strategy.backups.cozystack.io/'
28+
29+
# Every Strategy CR in the render under $1, one "<template> <kind>/<name>" line
30+
# each. --output-dir keeps one file per template, which is what names the
31+
# offender.
32+
strategy_crs() {
33+
find "$1" -name '*.yaml' | sort | while read -r f; do
34+
yq eval-all "select(.apiVersion // \"\" | test(\"^${STRATEGY_API}\")) | .kind + \"/\" + .metadata.name" "$f" \
35+
| sed "s#^#$(basename "$f") #"
36+
done
37+
}
38+
39+
@test "no Strategy CR renders while no strategy CRD is established, for any template" {
40+
tmp=$(mktemp -d)
41+
42+
# provisionBucket=false resolves the bucket name from values, so every bucket
43+
# gate is open, while the render has no apiserver and every CRD lookup
44+
# answers "absent": the state of an upgrade that ships a new kind, and of the
45+
# first external-S3 install.
46+
helm template t "$CHART" -n cozy-backup-controller \
47+
--set backupStorage.provisionBucket=false \
48+
--set backupStorage.bucketName=ext-bucket \
49+
--output-dir "$tmp/gated" >/dev/null
50+
strategy_crs "$tmp/gated" > "$tmp/gated.txt"
51+
52+
# Positive control: the same render with only the CRD gate opened (the
53+
# offline bucketNameOverride short-circuits it) must produce Strategy CRs,
54+
# or the assertion above it would hold for an empty render.
55+
helm template t "$CHART" -n cozy-backup-controller \
56+
--set backupStorage.provisionBucket=false \
57+
--set backupStorage.bucketName=ext-bucket \
58+
--set backupStorage.bucketNameOverride=ext-bucket \
59+
--output-dir "$tmp/open" >/dev/null
60+
strategy_crs "$tmp/open" > "$tmp/open.txt"
61+
62+
if [ -s "$tmp/gated.txt" ]; then
63+
echo "Strategy CRs rendered without an established CRD (missing crdEstablished gate):" >&2
64+
cat "$tmp/gated.txt" >&2
65+
false
66+
fi
67+
if [ ! -s "$tmp/open.txt" ]; then
68+
echo "control render produced no Strategy CR; the check above proves nothing" >&2
69+
false
70+
fi
71+
# A strategy template that renders nothing by default (behind an opt-in
72+
# value) would pass the first check without its gate ever being exercised.
73+
# Every one of them has to show up in the control render; one that is off by
74+
# default needs its switch added to both renders above.
75+
missing=""
76+
for f in "$CHART"/templates/strategy-*.yaml; do
77+
f=$(basename "$f")
78+
if ! grep -q "^$f " "$tmp/open.txt"; then
79+
missing="$missing $f"
80+
fi
81+
done
82+
if [ -n "$missing" ]; then
83+
echo "strategy templates absent from the control render, so unchecked:$missing" >&2
84+
false
85+
fi
86+
87+
rm -rf "$tmp"
88+
}
89+
90+
@test "every rendered Strategy CR gates on the CRD this chart ships for its kind" {
91+
tmp=$(mktemp -d)
92+
93+
# kind -> CRD name, from the CRDs this chart actually renders.
94+
for f in "$CHART"/definitions/*.yaml; do
95+
yq eval '.spec.names.kind + " " + .metadata.name' "$f"
96+
done > "$tmp/crds.txt"
97+
if [ ! -s "$tmp/crds.txt" ]; then
98+
echo "no CRDs read from $CHART/definitions" >&2
99+
false
100+
fi
101+
102+
helm template t "$CHART" -n cozy-backup-controller \
103+
--set backupStorage.bucketNameOverride=ext-bucket \
104+
--output-dir "$tmp/open" >/dev/null
105+
strategy_crs "$tmp/open" > "$tmp/open.txt"
106+
if [ ! -s "$tmp/open.txt" ]; then
107+
echo "render produced no Strategy CR; nothing was checked" >&2
108+
false
109+
fi
110+
111+
# The gate passes on any established CRD it is given, so a gate naming the
112+
# wrong CRD (a typo, a copied sibling) leaves the CR out of every live render
113+
# while the case above stays green. The name has to be the one serving the
114+
# kind.
115+
bad=""
116+
while read -r tpl obj; do
117+
kind=${obj%%/*}
118+
crd=$(awk -v k="$kind" '$1 == k { print $2 }' "$tmp/crds.txt")
119+
if [ -z "$crd" ]; then
120+
bad="$bad
121+
$tpl: $obj has no CRD in $CHART/definitions"
122+
continue
123+
fi
124+
if ! grep -qF "\"crd\" \"$crd\"" "$CHART/templates/$tpl"; then
125+
bad="$bad
126+
$tpl: $obj does not gate on crdEstablished for $crd"
127+
fi
128+
done < "$tmp/open.txt"
129+
if [ -n "$bad" ]; then
130+
echo "Strategy CR templates with a wrong or missing CRD gate:$bad" >&2
131+
false
132+
fi
133+
134+
rm -rf "$tmp"
135+
}

‎packages/system/backupstrategy-controller/templates/_helpers.tpl‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -167,9 +167,11 @@
167167
declared that cluster-derived state should be treated as resolved. Live
168168
deploys go through Flux and never set it.
169169
170-
Every default Strategy added from here on should gate on this helper as well
171-
as on the bucket name. The siblings that predate it are not wedged today only
172-
because their CRDs already shipped in an earlier release.
170+
Every default Strategy gates on this helper as well as on the bucket name,
171+
including the kinds whose CRDs shipped long ago: the bucket gate is also open
172+
on the very first render of an external-S3 install (provisionBucket=false),
173+
where no CRD exists yet. hack/backupstrategy-crd-gate.bats fails CI on any
174+
template that renders a Strategy CR without it.
173175
*/}}
174176
{{- define "backupstrategy-controller.crdEstablished" -}}
175177
{{- if .root.Values.backupStorage.bucketNameOverride -}}

‎packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@
1212
inner expression at chart install time and fail.
1313
*/}}
1414
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
15-
{{- if $bucketName -}}
15+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "altinities.strategy.backups.cozystack.io") -}}
16+
{{- if and $bucketName $crdEstablished -}}
1617
apiVersion: strategy.backups.cozystack.io/v1alpha1
1718
kind: Altinity
1819
metadata:

‎packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
2-
{{- if $bucketName -}}
2+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "cnpgs.strategy.backups.cozystack.io") -}}
3+
{{- if and $bucketName $crdEstablished -}}
34
apiVersion: strategy.backups.cozystack.io/v1alpha1
45
kind: CNPG
56
metadata:

‎packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
2-
{{- if $bucketName -}}
2+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "etcds.strategy.backups.cozystack.io") -}}
3+
{{- if and $bucketName $crdEstablished -}}
34
apiVersion: strategy.backups.cozystack.io/v1alpha1
45
kind: Etcd
56
metadata:

‎packages/system/backupstrategy-controller/templates/strategy-foundationdb-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
{{- $endpointHost := $endpoint | trimPrefix "https://" | trimPrefix "http://" -}}
33
{{- $secureConn := ternary "1" "0" (hasPrefix "https://" $endpoint) -}}
44
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
5-
{{- if $bucketName -}}
5+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "foundationdbs.strategy.backups.cozystack.io") -}}
6+
{{- if and $bucketName $crdEstablished -}}
67
apiVersion: strategy.backups.cozystack.io/v1alpha1
78
kind: FoundationDB
89
metadata:

‎packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,8 @@
1717
docs/operations/backup-classes.md and the example README.
1818
*/}}
1919
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
20-
{{- if $bucketName -}}
20+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "kafkas.strategy.backups.cozystack.io") -}}
21+
{{- if and $bucketName $crdEstablished -}}
2122
apiVersion: strategy.backups.cozystack.io/v1alpha1
2223
kind: Kafka
2324
metadata:

‎packages/system/backupstrategy-controller/templates/strategy-mariadb-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
{{- $endpoint := include "backupstrategy-controller.endpoint" . -}}
22
{{- $endpointHost := $endpoint | trimPrefix "https://" | trimPrefix "http://" -}}
33
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
4-
{{- if $bucketName -}}
4+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "mariadbs.strategy.backups.cozystack.io") -}}
5+
{{- if and $bucketName $crdEstablished -}}
56
apiVersion: strategy.backups.cozystack.io/v1alpha1
67
kind: MariaDB
78
metadata:

‎packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@
2020
a full-data backup is a Velero volume snapshot, not this strategy.
2121
*/}}
2222
{{- $bucketName := include "backupstrategy-controller.bucketName" . -}}
23-
{{- if $bucketName -}}
23+
{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "rabbitmqs.strategy.backups.cozystack.io") -}}
24+
{{- if and $bucketName $crdEstablished -}}
2425
apiVersion: strategy.backups.cozystack.io/v1alpha1
2526
kind: Rabbitmq
2627
metadata:

0 commit comments

Comments
 (0)