You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f6a03e9
Browse filesBrowse the repository at this point in the historyBrowse files
fix(etcd): complete v1alpha2 transition for in-cluster 1.5→1.6 upgrades (#3270)
Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 →
1.6 upgrades** into a single PR, rebased on current `main`. Supersedes
#3265 and #3261 — their commits are carried here (authorship preserved),
so those PRs can be closed once this lands. Fresh installs use the new
shapes directly and are unaffected, so fresh-install CI does not catch
these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression
guard.
## What this PR does
### 1. Keep the legacy `etcd-headless` Service alive during adoption —
`packages/extra/etcd`
`etcd-migrate` adopts legacy clusters **in place**: the Pods keep their
original `spec.subdomain: etcd-headless` and are dialed at
`etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's
native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only
creates the native `etcd` Service and the legacy `etcd-headless` Service
is pruned, so those per-pod names stop resolving (`no such host`),
`MemberList` fails, and `status.readyMembers` never populates — the
`EtcdCluster` never goes `Ready` even though etcd is healthy and in
quorum. We ship a chart-managed transitional headless `etcd-headless`
Service (selector mirrors the operator's native `etcd` Service via
`etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`)
— the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN
already kept for this window. Removable together with that SAN once
members roll onto the native subdomain.
### 2. Survive the immutable controller-Deployment selector on upgrade —
`packages/system/etcd-operator` (#3242)
#2859 replaced the upstream etcd-operator chart with the
cozystack-authored one, changing `Deployment.spec.selector.matchLabels`.
`spec.selector` is immutable, so `helm upgrade` cannot patch the
existing Deployment and the whole HelmRelease upgrade fails (`field is
immutable`). A **pre-upgrade hook**
(`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role +
RoleBinding + Job) deletes the Deployment **only** when its live
selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when
the selector already matches, never runs on fresh install.
### 3. Raise the operator's memory cold-start floor —
`packages/system/etcd-operator`
Steady-state working set is ~250Mi; the static `limits.memory: 128Mi`
OOMKills a Pod that starts before the VPA admission webhook rewrites it.
Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in
depth so the operator never depends on VPA timing to avoid crashing.
### 4. Make migration 50 (etcd adoption) robust in-cluster —
`packages/core/platform/images/migrations/migrations/50` (was #3261)
Exact server/peer cert-SAN match, Secret-gated wait on the adoption
Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration
script drives the adoption reliably from inside the cluster. Covered by
`hack/migration-50-etcd-adopt.bats`.
### 5. Hardening / review fixes (this PR's original scope)
- **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set
`runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s
image user is the non-numeric name `nonroot`, which the kubelet cannot
verify against `runAsNonRoot` — so the hook Pod fails admission and
silently blocks the very upgrade it exists to unblock. Adds `runAsUser:
65532`, matching the postgres-operator webhook-ready hook that runs the
same image.
- **Digest-pin the kubectl image.** The values comment claimed
digest-pinning but shipped a floating `v1.32` tag. Reuses the digest
postgres-operator vendors, adds the `renovate` annotation, and templates
`repo:tag@digest`.
- **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook
wiring, weight ordering, namespaced least-privilege RBAC,
numeric-non-root security context, digest-pinned image),
`etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor),
`extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless`
Service). Each assertion was mutation-tested.
### 6. Derive the default S3 endpoint from the provisioned bucket —
`packages/system/backupstrategy-controller`
Derive the default S3 endpoint (and per-driver scheme / TLS /
`secure_connection`) from the provisioned bucket Secret instead of
requiring it to be hand-set, so etcd (and other) backup strategies get a
working endpoint by default. Docs in
`docs/operations/backup-classes.md`; covered by
`tests/endpoint_form_test.yaml`.
### Verification
- `helm unittest` green on current `main`: etcd-operator **18/18**,
extra/etcd **18/18**, backupstrategy-controller **11/11**.
- Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 →
1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless`
Service took the adopted cluster to `readyMembers=3 / Available=True`.
```release-note
fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved backup storage endpoint handling across supported backup
drivers, including provisioned and external S3 storage.
* Added compatibility support for legacy etcd pod discovery during
migration.
* Added an automated upgrade safeguard for etcd operator deployments.
* **Bug Fixes**
* Improved certificate SAN detection and etcd migration authentication.
* Increased the etcd operator’s minimum startup memory to prevent early
restarts.
* **Documentation**
* Clarified backup endpoint, TLS, and driver-specific behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Copy file name to clipboardExpand all lines: docs/operations/backup-classes.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,7 +27,7 @@ The FoundationDB strategy CR is rendered by the chart so admins can reference it
27
27
28
28
### Endpoint format per driver
29
29
30
-
Different operators expect different endpoint shapes; the strategy templates rendered by `backupstrategy-controller` adapt the single `backupStorage.endpoint` value (a full URL like `http://seaweedfs-s3.tenant-root.svc:8333`) to each consumer's contract:
30
+
Different operators expect different endpoint shapes; the strategy templates rendered by `backupstrategy-controller` resolve one S3 endpoint (via the `backupstrategy-controller.endpoint` helper) and adapt it to each consumer's contract. For a provisioned bucket (`provisionBucket: true`, the default) the endpoint is **derived from the COSI bucket's system-credentials Secret** (`backupStorage.systemSecretName`) and forced to `https://` — the external S3 ingress with an ACME cert, the only endpoint the backup operators can verify. SeaweedFS's in-cluster S3 serves TLS on `:8333` behind the self-signed "SeaweedFS CA", and the Etcd Strategy S3 schema has no `caCert` field, so the in-cluster endpoint cannot be targeted directly. The `backupStorage.endpoint` chart value (a full URL like `http://seaweedfs-s3.tenant-root.svc:8333`) is the **fallback**, used for external S3 (`provisionBucket: false`) and for offline `helm template`/pre-reconcile renders where the Secret lookup returns nothing. The resolved endpoint is adapted per consumer:
31
31
32
32
| Driver | Strategy template field | Form |
33
33
|--------|-------------------------|------|
@@ -38,7 +38,7 @@ Different operators expect different endpoint shapes; the strategy templates ren
38
38
| Velero |`BackupStorageLocation.spec.config.s3Url`| full URL (scheme preserved) |
The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL pull from `backupStorage.endpoint` in chart values, not from the Secret.
41
+
The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced `https://`) for a provisioned bucket, or the `backupStorage.endpoint` fallback for external S3.
42
42
43
43
VM-driven (Velero) backups land in the same `cozy-backups` bucket under the `velero/` prefix. A `BackupStorageLocation` named `cozy-default` is shipped by the `backupstrategy-controller` chart (`packages/system/backupstrategy-controller/templates/velero-bsl.yaml`) so endpoint/bucket/region come from the same `backupStorage` values block used by Strategy CRs and the projector.
44
44
@@ -135,7 +135,7 @@ spec:
135
135
| `provisionBucket` | Toggle creation of the in-cluster `apps.cozystack.io/Bucket` CR. Set `false` for external S3 (see [Disabling the platform-managed bucket](#disabling-the-platform-managed-bucket)). |
136
136
| `bucketName` | K8s name of the Bucket CR + lookup key for the COSI BucketClaim. The actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. |
137
137
| `bucketNameOverride` | Escape hatch for offline `helm template` renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. |
138
-
| `endpoint` | S3 endpoint baked into every default strategy CR + the Velero BSL. Switching to `https://` silently enables TLS in the MariaDB strategy — ensure the CA bundle is reachable to the relevant operator/driver Pods before flipping it. |
138
+
| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies — ensure the CA bundle is reachable to the relevant operator/driver Pods first. |
139
139
| `region` | Re-projected into `cozy-backups-creds` on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). |
| `systemSecretName` | Name of the human-friendly Secret produced by the Bucket app (or pre-created manually for external S3). The projector also accepts the raw COSI Secret format. |
0 commit comments