Skip to content

Commit 6338b0b

Browse files
fix(migrations): derive the etcd-adoption snapshot target from the projected bucket creds (#3335)
## What this PR does Migration 50 takes a mandatory S3 safety snapshot before adopting live etcd onto the v1alpha2 operator. It is a **`pre-upgrade` hook**, so every chart-rendered resource it reads still belongs to the version being upgraded **from**. It resolved the snapshot target from the live `cozy-default-etcd` Strategy CR — and on a v1.5.x cluster that CR cannot be trusted to describe a reachable target. The platform upgrade is gated behind the snapshot, so nothing can self-heal first: the corrected `backupstrategy-controller.endpoint` helper only reconciles **after** the upgrade this hook blocks. ### Two failure modes, both a hard block **1. The Strategy CR does not exist.** `strategy-etcd-default.yaml` is guarded by `{{- if $bucketName -}}`, whose helper `lookup`s the BucketClaim's `.status.bucketName`. That BucketClaim is created by the *same chart*, so on first render the lookup is empty and the Strategy is silently skipped. Helm `lookup` only runs at install/upgrade time, so a cluster that reached 1.5.x in one hop never grows the CR on reconcile — a forced reconcile does not produce it. **2. The Strategy CR exists carrying `http://seaweedfs-s3.tenant-root.svc.cozy.local:8333`** (the static v1.5.x `backupStorage.endpoint` default). Cozystack ships SeaweedFS with `enableSecurity=true`, so `:8333` is a **TLS listener** — plaintext against it fails the handshake and every snapshot dies. Fixing the *scheme* is not sufficient: as `backupstrategy-controller.endpoint` documents, the Etcd Strategy's S3 schema has no `caCert`/`insecureSkipVerify`, so the self-signed in-cluster endpoint is unusable at **any** scheme. A trusted-cert endpoint is required. ## The fix Take the coordinates from `cozy-backups-creds` instead. The projector writes it from the **COSI-provisioned bucket's system credentials** — bucket provisioning, not chart values — so it describes the live bucket regardless of which version rendered the charts. It carries `endpoint`, `bucketName`, `region` and `forcePathStyle` alongside the AWS keys, and `resolve_platform_backup_args` already read this Secret to validate those keys, so preferring it adds no new dependency. Both modes disappear: mode 1 because the CR is no longer needed, mode 2 because its endpoint is ignored. ### Classifying COSI vs. external S3 The projected endpoint is a bare host fronted by the always-TLS S3 ingress, so on the platform-managed path we force `https://` — exactly what `backupstrategy-controller.endpoint` does. But the admin-managed external-S3 case (`provisionBucket: false`) must keep `.Values.backupStorage.endpoint` verbatim, since it may legitimately be plaintext against a private store. The two therefore have to be told apart. **Endpoint presence cannot do that.** `ProjectBackupCredentials` substitutes `BACKUP_STORAGE_ENDPOINT` when the source Secret is silent and then *fails loud* rather than projecting an endpoint-less Secret, so a projected `endpoint` key is **always** present and **always** scheme-stripped — for external S3 too. Classifying on its absence would force `https://` onto a plaintext external store and never consult the CR, which is a regression against a supported, documented configuration. Instead the hook asks whether the bucket the credentials name is one **COSI actually provisioned**: it matches the projected `bucketName` against `.status.bucketName` of any BucketClaim. That is keyed on data rather than names, so it is independent of `backupStorage.namespace` / `.bucketName` / `bucketNameOverride` — all supported Package-CR overrides the hook cannot see — and a claim-name lookup would misclassify a renamed bucket as external. Resolution is then: | cluster state | endpoint | other coordinates | |---|---|---| | a **live** BucketClaim claims the bucket | projected host, forced `https://` | projected Secret | | **no** claim claims the bucket (external S3) | Strategy CR, scheme verbatim | Strategy CR | | only a **Terminating** claim claims it | refuse | — | | BucketClaim API unreadable | refuse | — | On the external path every coordinate comes from the CR, never a Secret/CR hybrid: `strategy-etcd-default.yaml` pairs `bucket` with `credentialsSecretRef: cozy-backups-creds`, so CR coordinates plus the projected credentials is exactly the pairing the cluster's real etcd BackupJobs already use. ### Deliberate behaviour changes Two states now refuse where `main` proceeded. Both are cases where the classifier cannot read its input, and adopting live etcd is irreversible, so it fails closed with an actionable message rather than guessing: - **A Terminating BucketClaim on the projected bucket.** Reachable via COSI → external reusing a retained bucket name, where the `bucketclaim-protection` finalizer wedges the corpse permanently. Both readings are reachable and guessing wrong in either direction also ends in a block — just with a confusing TLS/handshake error instead of a precise one. Remedy: let COSI reap the claim, or clear its finalizer, then re-run (migration 50 is idempotent). - **A sustained BucketClaim API read failure** on a cluster that has COSI installed. A genuinely absent CRD is still a definitive, zero-cost "external" answer and costs no retries. ### The escape hatch is now reachable The script has always supported `ETCD_ADOPT_SKIP_BACKUP=1` for clusters that intentionally have no backup storage, but `migration-hook.yaml` passed no such variable, so the failure paths advised an action the platform made impossible. It is now plumbed as `migrations.etcdAdoptSkipBackup`, an explicit opt-in defaulting to off, and the operator-facing messages name that knob rather than a variable nobody can set. It also unblocks the two refuse states above. The template normalises to the exact string the script matches: a bare bool renders `true`, which the script's `= "1"` test would accept into the Job spec and then silently ignore — the worst outcome for a valve reached for under duress. Anything unrecognised resolves to `"0"`, so taking the hatch requires an affirmative spelling and never a typo. A Platform Package predating the key renders cleanly as off. The script's own check was widened alongside it to accept `1|true|yes` case-insensitively, so a hand-run image honours what an operator actually types. That is a behaviour change beyond plumbing, and it has a cost worth naming: if the template's normalisation were ever dropped, a raw bool would previously have failed safe (snapshot taken) and now fails unsafe (snapshot skipped). The rendered value is pinned by a unit test to prevent that. `migration-hook.yaml` had no test coverage at all before this: the Job renders only when a `lookup` of `cozystack-version` reports a version below target, and `lookup` returns nil under `helm template`, so lint and CI never rendered the Job. That is structurally why an unreachable escape hatch shipped unnoticed. The new suite mocks the ConfigMap so the Job is rendered and its env asserted by value. ## Tests The bats fixture modelled `cozy-backups-creds` as carrying only the AWS keys, and test 1 asserted the broken plaintext endpoint **as expected output** — written against the implementation rather than the requirement, against a mock that did not match a real cluster, which is why this shipped green. The fixture now models what the projector and COSI really produce. Added, each verified to fail before the corresponding fix and pass after: - platform path derives `https://` from the projected coordinates, and the in-cluster host never appears - absent Strategy CR still resolves (mode 1) - external S3 keeps the CR endpoint's scheme verbatim - external S3 takes **every** coordinate from the CR, never a Secret/CR hybrid - overridden bucket coordinates still classify as COSI - a large multi-tenant claim list still classifies correctly (the match must not depend on list size or position) - a Terminating claim on the projected bucket refuses, and never scales or adopts - an unreadable BucketClaim CRD / list refuses - a live claim wins over an unrelated Terminating one - COSI keeps the projected bucket even when the CR names another - the skip-backup hatch is honoured, and an unrecognised value does not skip Suite: 15 → 28, plus a new helm-unittest suite covering the hook's rendered env (chart: 81 → 88 tests). ## Validation Reproduced end-to-end on a throwaway 3-node stand (fresh v1.4.5 → v1.5.3 → this branch's build). Before: mode 1, version stamp stuck, `cozy-backups` Bucket Ready, `backupstrategy-controller` Ready, creds projected, **zero** Strategy CRs. After: the stamp advanced and the hook completed, with endpoint, bucket, region and path-style all derived from the projected Secret and no Strategy CR in existence. Failure was safe in both cases — it refuses before any mutation: etcd stayed 3/3 Running, the legacy CR un-adopted, no scale-down. The derived `https://` endpoint reproduces the 1.6 chart helper's own output byte-for-byte, and the projected `bucketName` equals the BucketClaim's `.status.bucketName` on a live stand. ## Follow-ups this PR deliberately does not do - **Fix the `if $bucketName` lookup race** in `strategy-etcd-default.yaml`, so the Strategy exists at all on affected clusters. This PR makes migration 50 independent of it, but the missing CR likely means **default etcd backups are silently broken on 1.5.x clusters**, upgrade or not — the backup e2e is `.disabled`, so CI cannot see it. Possibly related to #3256. Worth its own issue. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Backup destination for migration 50 adoption can now derive from projected credentials when available. * Added `migrations.etcdAdoptSkipBackup` to control whether the migration job skips pre-adoption snapshot safety checks. * **Bug Fixes** * Improved COSI vs external S3 classification and endpoint normalization; clearer failures when destination can’t be resolved. * `ETCD_ADOPT_SKIP_BACKUP` now honors true/false and rejects unrecognized values (does not skip by accident). * **Tests** * Expanded end-to-end coverage for projected vs external S3 behavior, precedence rules, and non-destructive failure scenarios. * Added migration hook tests validating `ETCD_ADOPT_SKIP_BACKUP` env-var mapping and defaulting. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 1d2418e + efa85a4 commit 6338b0b

6 files changed

Lines changed: 983 additions & 26 deletions

File tree

0 commit comments

Comments
 (0)