Skip to content

Commit 1752d28

Browse files
fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does Refs #5205. The S3 endpoint the platform backups use is copied twice, and neither copy is refreshed after `publishing.host` changes. The COSI sidecar writes `BucketInfo` once per `BucketAccess`, so the credentials Secret `bucket-cozy-backups-system-credentials` rendered from it keeps the old endpoint. The Velero BSL and the default Strategy CRs keep whatever endpoint the last render of `backupstrategy-controller` saw. The default objects gate only acted on missing objects, so backups kept going to the old host and nothing reported it. The gate now also forces a render when a copy is stale: - the bucket's `<bucket>-system` release, when the credentials Secret's endpoint differs from the COSI Secret it is rendered from; - `backupstrategy-controller`, when the endpoint an object was rendered with differs from the credentials Secret's. The chart stamps the rendered endpoint on the BSL and on every default Strategy as `backups.cozystack.io/endpoint`, because the endpoint itself sits at a different path in every kind. Objects without the annotation count as stale until a render stamps it; `MinForceInterval` bounds the forces. Stale credentials are forced before the BackupClass and the routed objects are read, so an absent BackupClass or an unreadable Strategy still fails the check but does not keep the old endpoint in the credentials Secret. A `BucketInfo` without an endpoint is not compared, since rendering from it would write an empty endpoint or fail the bucket release. Repairing missing objects keeps working while the credentials comparison is stale or fails, and a `systemSecretName` that is not derived from a COSI Secret (no `-credentials` suffix, or the raw COSI Secret itself) is not compared at all. RBAC is unchanged: the controller already holds cluster-wide `get` on Secrets. No new metric: nothing would consume it. What this does not do: the COSI Secret still holds the endpoint the `BucketAccess` was granted with, so after a host change the endpoint follows only once an operator deletes the `BucketAccess` and forces `bucket-cozy-backups` to recreate it (this issues new S3 keys). From there the gate replaces the remaining manual steps of the recovery described in the issue. Automatic re-grant is left for a separate issue, and the operator runbook goes to the website, hence `Refs` rather than `Fixes`. Known limitation: the credentials comparison runs only when `backupStorage.systemSecretName` is `bucket-<bucketName>-system-credentials`, the Secret the forced `bucket-<bucketName>-system` release renders. If an operator sets `bucketName` and `systemSecretName` inconsistently, the comparison is switched off rather than forcing a release that never rewrites the compared Secret, so a stale endpoint in that Secret is not repaired. Repairing a missing credentials Secret still forces the release derived from `bucketName`, as before this PR. Tested with unit tests for the gate (including the comparison normalisation shared with the chart helpers), helm-unittest for the annotation and the wiring, and a bats guard that every rendered Strategy CR and the BSL carry the annotation. On a live cluster, a server-side render of the chart produced exactly the annotation the gate derives from the real credentials Secret, so the steady state does not force anything. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: an operator runbook for changing `publishing.host` with platform backups enabled (re-grant the `BucketAccess`, then let the gate converge; restart ClickHouse pods that read the credentials from environment variables). Not filed yet, pending approval of the text. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup strategies and storage locations now record their storage endpoint, enabling detection and repair of outdated endpoint settings after bucket access changes. * Bucket credentials and backup objects are refreshed independently when their endpoints are missing or out of date. * **Documentation** * Clarified when endpoint changes trigger updates and how suspended releases are handled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: 9e8e10c
2 parents a14ad50 + 490eab5 commit 1752d28

21 files changed

Lines changed: 958 additions & 63 deletions

‎cmd/backupstrategy-controller/main.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,7 @@ func main() {
243243
Namespace: os.Getenv("BACKUP_CREDENTIALS_HELMRELEASE_NAMESPACE"),
244244
Name: os.Getenv("BACKUP_CREDENTIALS_HELMRELEASE_NAME"),
245245
},
246+
CredentialsCOSISecretName: os.Getenv("BACKUP_CREDENTIALS_COSI_SECRET_NAME"),
246247
}
247248
if err := gate.SetupWithManager(mgr); err != nil {
248249
setupLog.Error(err, "unable to add DefaultObjectsGate runnable")

‎hack/backupstrategy-crd-gate.bats‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,3 +133,39 @@ $tpl: $obj does not gate on crdEstablished for $crd"
133133

134134
rm -rf "$tmp"
135135
}
136+
137+
# The controller's DefaultObjectsGate tells an object rendered with a stale S3
138+
# endpoint (publishing.host changed after the render) by the
139+
# backups.cozystack.io/endpoint annotation, because the endpoint itself sits at
140+
# a different path in every kind. An object the chart renders without it
141+
# counts as stale on every check, so a new template that forgets it would
142+
# force this release every MinForceInterval forever.
143+
@test "every rendered Strategy CR and the Velero BSL carry the rendered endpoint annotation" {
144+
tmp=$(mktemp -d)
145+
146+
helm template t "$CHART" -n cozy-backup-controller \
147+
--set backupStorage.bucketNameOverride=ext-bucket \
148+
--set backupStorage.endpoint=https://s3.example.org \
149+
--output-dir "$tmp/open" >/dev/null
150+
151+
objs=$(find "$tmp/open" -name '*.yaml' | sort | while read -r f; do
152+
yq eval-all "select((.apiVersion // \"\" | test(\"^${STRATEGY_API}\")) or .kind == \"BackupStorageLocation\") | .kind + \"/\" + .metadata.name + \" \" + (.metadata.annotations[\"backups.cozystack.io/endpoint\"] // \"<none>\")" "$f" \
153+
| sed "s#^#$(basename "$f") #"
154+
done)
155+
if ! echo "$objs" | grep -q " BackupStorageLocation/"; then
156+
echo "render produced no BSL; nothing was checked for it" >&2
157+
false
158+
fi
159+
if ! echo "$objs" | grep -q " CNPG/"; then
160+
echo "render produced no Strategy CR; nothing was checked" >&2
161+
false
162+
fi
163+
bad=$(echo "$objs" | awk '$3 != "https://s3.example.org"')
164+
if [ -n "$bad" ]; then
165+
echo "objects without the rendered endpoint in backups.cozystack.io/endpoint:" >&2
166+
echo "$bad" >&2
167+
false
168+
fi
169+
170+
rm -rf "$tmp"
171+
}

‎internal/backupcontroller/default_objects_gate.go‎

Lines changed: 189 additions & 46 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)