Skip to content

Commit f6a03e9

Browse files
fix(etcd): complete v1alpha2 transition for in-cluster 1.5→1.6 upgrades (#3270)
Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 → 1.6 upgrades** into a single PR, rebased on current `main`. Supersedes #3265 and #3261 — their commits are carried here (authorship preserved), so those PRs can be closed once this lands. Fresh installs use the new shapes directly and are unaffected, so fresh-install CI does not catch these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression guard. ## What this PR does ### 1. Keep the legacy `etcd-headless` Service alive during adoption — `packages/extra/etcd` `etcd-migrate` adopts legacy clusters **in place**: the Pods keep their original `spec.subdomain: etcd-headless` and are dialed at `etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only creates the native `etcd` Service and the legacy `etcd-headless` Service is pruned, so those per-pod names stop resolving (`no such host`), `MemberList` fails, and `status.readyMembers` never populates — the `EtcdCluster` never goes `Ready` even though etcd is healthy and in quorum. We ship a chart-managed transitional headless `etcd-headless` Service (selector mirrors the operator's native `etcd` Service via `etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`) — the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN already kept for this window. Removable together with that SAN once members roll onto the native subdomain. ### 2. Survive the immutable controller-Deployment selector on upgrade — `packages/system/etcd-operator` (#3242) #2859 replaced the upstream etcd-operator chart with the cozystack-authored one, changing `Deployment.spec.selector.matchLabels`. `spec.selector` is immutable, so `helm upgrade` cannot patch the existing Deployment and the whole HelmRelease upgrade fails (`field is immutable`). A **pre-upgrade hook** (`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role + RoleBinding + Job) deletes the Deployment **only** when its live selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when the selector already matches, never runs on fresh install. ### 3. Raise the operator's memory cold-start floor — `packages/system/etcd-operator` Steady-state working set is ~250Mi; the static `limits.memory: 128Mi` OOMKills a Pod that starts before the VPA admission webhook rewrites it. Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in depth so the operator never depends on VPA timing to avoid crashing. ### 4. Make migration 50 (etcd adoption) robust in-cluster — `packages/core/platform/images/migrations/migrations/50` (was #3261) Exact server/peer cert-SAN match, Secret-gated wait on the adoption Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration script drives the adoption reliably from inside the cluster. Covered by `hack/migration-50-etcd-adopt.bats`. ### 5. Hardening / review fixes (this PR's original scope) - **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set `runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s image user is the non-numeric name `nonroot`, which the kubelet cannot verify against `runAsNonRoot` — so the hook Pod fails admission and silently blocks the very upgrade it exists to unblock. Adds `runAsUser: 65532`, matching the postgres-operator webhook-ready hook that runs the same image. - **Digest-pin the kubectl image.** The values comment claimed digest-pinning but shipped a floating `v1.32` tag. Reuses the digest postgres-operator vendors, adds the `renovate` annotation, and templates `repo:tag@digest`. - **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook wiring, weight ordering, namespaced least-privilege RBAC, numeric-non-root security context, digest-pinned image), `etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor), `extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless` Service). Each assertion was mutation-tested. ### 6. Derive the default S3 endpoint from the provisioned bucket — `packages/system/backupstrategy-controller` Derive the default S3 endpoint (and per-driver scheme / TLS / `secure_connection`) from the provisioned bucket Secret instead of requiring it to be hand-set, so etcd (and other) backup strategies get a working endpoint by default. Docs in `docs/operations/backup-classes.md`; covered by `tests/endpoint_form_test.yaml`. ### Verification - `helm unittest` green on current `main`: etcd-operator **18/18**, extra/etcd **18/18**, backupstrategy-controller **11/11**. - Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 → 1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless` Service took the adopted cluster to `readyMembers=3 / Available=True`. ```release-note fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved backup storage endpoint handling across supported backup drivers, including provisioned and external S3 storage. * Added compatibility support for legacy etcd pod discovery during migration. * Added an automated upgrade safeguard for etcd operator deployments. * **Bug Fixes** * Improved certificate SAN detection and etcd migration authentication. * Increased the etcd operator’s minimum startup memory to prevent early restarts. * **Documentation** * Clarified backup endpoint, TLS, and driver-specific behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 3fb8ec5 + 4af5a82 commit f6a03e9

19 files changed

Lines changed: 707 additions & 33 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ The FoundationDB strategy CR is rendered by the chart so admins can reference it
2727

2828
### Endpoint format per driver
2929

30-
Different operators expect different endpoint shapes; the strategy templates rendered by `backupstrategy-controller` adapt the single `backupStorage.endpoint` value (a full URL like `http://seaweedfs-s3.tenant-root.svc:8333`) to each consumer's contract:
30+
Different operators expect different endpoint shapes; the strategy templates rendered by `backupstrategy-controller` resolve one S3 endpoint (via the `backupstrategy-controller.endpoint` helper) and adapt it to each consumer's contract. For a provisioned bucket (`provisionBucket: true`, the default) the endpoint is **derived from the COSI bucket's system-credentials Secret** (`backupStorage.systemSecretName`) and forced to `https://` — the external S3 ingress with an ACME cert, the only endpoint the backup operators can verify. SeaweedFS's in-cluster S3 serves TLS on `:8333` behind the self-signed "SeaweedFS CA", and the Etcd Strategy S3 schema has no `caCert` field, so the in-cluster endpoint cannot be targeted directly. The `backupStorage.endpoint` chart value (a full URL like `http://seaweedfs-s3.tenant-root.svc:8333`) is the **fallback**, used for external S3 (`provisionBucket: false`) and for offline `helm template`/pre-reconcile renders where the Secret lookup returns nothing. The resolved endpoint is adapted per consumer:
3131

3232
| Driver | Strategy template field | Form |
3333
|--------|-------------------------|------|
@@ -38,7 +38,7 @@ Different operators expect different endpoint shapes; the strategy templates ren
3838
| Velero | `BackupStorageLocation.spec.config.s3Url` | full URL (scheme preserved) |
3939
| ClickHouse sidecar | `S3_ENDPOINT` env | bare host:port (from projected Secret) |
4040

41-
The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL pull from `backupStorage.endpoint` in chart values, not from the Secret.
41+
The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced `https://`) for a provisioned bucket, or the `backupStorage.endpoint` fallback for external S3.
4242

4343
VM-driven (Velero) backups land in the same `cozy-backups` bucket under the `velero/` prefix. A `BackupStorageLocation` named `cozy-default` is shipped by the `backupstrategy-controller` chart (`packages/system/backupstrategy-controller/templates/velero-bsl.yaml`) so endpoint/bucket/region come from the same `backupStorage` values block used by Strategy CRs and the projector.
4444

@@ -135,7 +135,7 @@ spec:
135135
| `provisionBucket` | Toggle creation of the in-cluster `apps.cozystack.io/Bucket` CR. Set `false` for external S3 (see [Disabling the platform-managed bucket](#disabling-the-platform-managed-bucket)). |
136136
| `bucketName` | K8s name of the Bucket CR + lookup key for the COSI BucketClaim. The actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. |
137137
| `bucketNameOverride` | Escape hatch for offline `helm template` renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. |
138-
| `endpoint` | S3 endpoint baked into every default strategy CR + the Velero BSL. Switching to `https://` silently enables TLS in the MariaDB strategy — ensure the CA bundle is reachable to the relevant operator/driver Pods before flipping it. |
138+
| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies — ensure the CA bundle is reachable to the relevant operator/driver Pods first. |
139139
| `region` | Re-projected into `cozy-backups-creds` on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). |
140140
| `forcePathStyle` | Path-style addressing; SeaweedFS S3 requires it, AWS S3 typically doesn't. |
141141
| `systemSecretName` | Name of the human-friendly Secret produced by the Bucket app (or pre-created manually for external S3). The projector also accepts the raw COSI Secret format. |

‎hack/migration-50-etcd-adopt.bats‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,3 +289,53 @@ lineno() {
289289
! grep -qF -- "ETCD-MIGRATE --apply" "$FAKE_CMDLOG"
290290
rm -rf "$WORK"
291291
}
292+
293+
@test "cert SAN check is exact: a superstring dnsName does not skip the re-issue patch" {
294+
prep
295+
# The Certificate's spec.dnsNames already carries *.etcd.<ns>.svc.cluster.local,
296+
# which CONTAINS the native wildcard *.etcd.<ns>.svc as a substring but is not
297+
# an exact match. A substring match (the bug) would treat the wildcard as
298+
# already present and skip the patch; the exact match must still re-issue.
299+
export FAKE_CERTS=1
300+
export FAKE_CERT_SUPERSTRING=1
301+
rc=0
302+
bash "$MIG" >"$WORK/out" 2>&1 || rc=$?
303+
cat "$WORK/out"
304+
cat "$FAKE_CMDLOG"
305+
[ "$rc" -eq 0 ]
306+
# Both Certificates are still patched despite the substring-only SAN.
307+
grep -qF -- "PATCH-CERT etcd-server" "$FAKE_CMDLOG"
308+
grep -qF -- "PATCH-CERT etcd-peer" "$FAKE_CMDLOG"
309+
rm -rf "$WORK"
310+
}
311+
312+
@test "etcd-migrate authenticates via a synthesized in-cluster kubeconfig at kubernetes.default.svc" {
313+
prep
314+
# etcd-migrate only reads a kubeconfig FILE (no in-cluster SA fallback), so the
315+
# script synthesizes one from the mounted ServiceAccount. Point the SA dir at a
316+
# fixture and capture the written kubeconfig.
317+
sa="$WORK/sa"; mkdir -p "$sa"
318+
printf 'faketoken' > "$sa/token"
319+
printf 'fakeca' > "$sa/ca.crt"
320+
printf 'cozy-system' > "$sa/namespace"
321+
export ETCD_ADOPT_SA_DIR="$sa"
322+
export ETCD_MIGRATE_KUBECONFIG="$WORK/in-cluster.kubeconfig"
323+
# A bare IPv6 host that must NOT end up in the (unbracketed, invalid) server URL.
324+
export KUBERNETES_SERVICE_HOST="fd00::1"
325+
export KUBERNETES_SERVICE_PORT="443"
326+
rc=0
327+
bash "$MIG" >"$WORK/out" 2>&1 || rc=$?
328+
cat "$WORK/out"
329+
cat "$FAKE_CMDLOG"
330+
[ "$rc" -eq 0 ]
331+
# Both the dry-run and --apply invocations carry --kubeconfig=<synthesized>.
332+
[ "$(grep -cF -- "--kubeconfig=$WORK/in-cluster.kubeconfig" "$FAKE_CMDLOG")" -ge 2 ]
333+
# The kubeconfig is IP-family-agnostic (DNS name, not the bare IPv6 host) and
334+
# authenticates via the SA token file + CA.
335+
[ -s "$ETCD_MIGRATE_KUBECONFIG" ]
336+
grep -qF -- "server: https://kubernetes.default.svc" "$ETCD_MIGRATE_KUBECONFIG"
337+
! grep -qF -- "fd00::1" "$ETCD_MIGRATE_KUBECONFIG"
338+
grep -qF -- "tokenFile: $sa/token" "$ETCD_MIGRATE_KUBECONFIG"
339+
grep -qF -- "certificate-authority: $sa/ca.crt" "$ETCD_MIGRATE_KUBECONFIG"
340+
rm -rf "$WORK"
341+
}

‎hack/testdata/migration-50/kubectl‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,20 @@ case "$args" in
6969
cert="$(arg_after certificate.cert-manager.io "$@")"
7070
: > "$state_dir/reissued-${cert}-tls"
7171
log "PATCH-CERT $cert"; exit 0 ;;
72+
# jsonpath '{.spec.dnsNames[*]}' probe used by _san_present: emit a
73+
# SPACE-separated SAN list (matches real kubectl jsonpath[*] output), NOT the
74+
# -o json blob below. FAKE_CERT_SUPERSTRING=1 injects a longer SAN that
75+
# CONTAINS the native wildcard as a substring (*.etcd.<ns>.svc inside
76+
# *.etcd.<ns>.svc.cluster.local) but is not an exact match — exercises the
77+
# exact-match guard (a substring match would wrongly skip the re-issue patch).
78+
*"get certificate.cert-manager.io"*"dnsNames"*)
79+
ns="$(arg_after -n "$@")"
80+
if [ "${FAKE_CERT_SUPERSTRING:-0}" = "1" ]; then
81+
printf 'etcd etcd.%s.svc *.etcd.%s.svc.cluster.local\n' "$ns" "$ns"
82+
else
83+
printf 'etcd etcd.%s.svc\n' "$ns"
84+
fi
85+
exit 0 ;;
7286
*"get certificate.cert-manager.io"*"-o json"*)
7387
ns="$(arg_after -n "$@")"
7488
printf '{"spec":{"dnsNames":["etcd","etcd.%s.svc"]}}\n' "$ns"; exit 0 ;;

‎packages/core/platform/images/migrations/migrations/50‎

Lines changed: 94 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,61 @@ CREDS_TENANT_LABEL="internal.cozystack.io/tenantresource"
114114
# templates declare only the short forms, so adding FQDN SANs here would be
115115
# stripped on the next HelmRelease reconcile — adoption and chart output must
116116
# converge to the same fixed point on the cert SANs.
117+
# _san_present <ns> <secret> <cert> <native>
118+
# Returns 0 iff <native> is present in the issued Secret's cert-manager.io/alt-names
119+
# annotation OR the Certificate's spec.dnsNames (the source of truth for the SANs
120+
# cert-manager will issue). ROBUSTNESS: a plain `kubectl get ... 2>/dev/null` can
121+
# transiently return an EMPTY string on an API hiccup (discovery refresh, apiserver
122+
# blip, throttling), which is indistinguishable from "SAN genuinely absent" and
123+
# makes a correctly-configured cert look missing — the failure mode that wedges the
124+
# cert wait below. Neither field is ever legitimately empty on a real object, so we
125+
# treat an empty read from BOTH sources as a transient error and retry, only
126+
# trusting a non-empty read to decide presence/absence.
127+
_san_present() {
128+
local ns="$1" secret="$2" cert="$3" native="$4" a d try
129+
for try in 1 2 3 4 5; do
130+
a=$(kubectl -n "$ns" get secret "$secret" \
131+
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null || true)
132+
# {.spec.dnsNames[*]} yields a SPACE-separated list (one SAN per token);
133+
# {.spec.dnsNames} would be a bracketed JSON blob that tr ',' cannot split,
134+
# forcing a substring match. Split on spaces and match EXACTLY (grep -qxF) —
135+
# same as the Secret annotation above — so a wildcard that is a substring of
136+
# a longer SAN (e.g. *.etcd.<ns>.svc inside *.etcd.<ns>.svc.cluster.local)
137+
# cannot false-positive and skip the patch.
138+
d=$(kubectl -n "$ns" get certificate.cert-manager.io "$cert" \
139+
-o jsonpath='{.spec.dnsNames[*]}' 2>/dev/null || true)
140+
if [ -n "$a" ] || [ -n "$d" ]; then
141+
printf '%s' "$a" | tr ',' '\n' | grep -qxF "$native" && return 0
142+
printf '%s' "$d" | tr ' ' '\n' | grep -qxF "$native" && return 0
143+
return 1
144+
fi
145+
sleep 1
146+
done
147+
return 1
148+
}
149+
150+
# _secret_has_san: exact-match check that $native is listed in the ISSUED
151+
# Secret's cert-manager alt-names annotation — the authoritative record of what
152+
# cert-manager has actually issued. Unlike _san_present it does NOT consult the
153+
# Certificate spec.dnsNames: after we patch the spec below, the spec carries the
154+
# wildcard immediately, but the Secret is only re-issued once cert-manager
155+
# reconciles; the wait loop must gate on the SECRET, not on the value we just
156+
# wrote. Same empty-read retry rationale as _san_present (a real issued Secret
157+
# never has an empty alt-names annotation), and the same exact grep -qxF match.
158+
_secret_has_san() {
159+
local ns="$1" secret="$2" native="$3" a try
160+
for try in 1 2 3 4 5; do
161+
a=$(kubectl -n "$ns" get secret "$secret" \
162+
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null || true)
163+
if [ -n "$a" ]; then
164+
printf '%s' "$a" | tr ',' '\n' | grep -qxF "$native"
165+
return
166+
fi
167+
sleep 1
168+
done
169+
return 1
170+
}
171+
117172
ensure_wildcard_sans() {
118173
local ns="$1" name="$2"
119174
local native="*.${name}.${ns}.svc"
@@ -129,9 +184,7 @@ ensure_wildcard_sans() {
129184
echo " - ${ns}/${cert}: Certificate not found; skipping (operator-managed or already pruned)"
130185
continue
131186
fi
132-
if kubectl -n "$ns" get secret "$secret" \
133-
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null \
134-
| tr ',' '\n' | grep -qxF "$native"; then
187+
if _san_present "$ns" "$secret" "$cert" "$native"; then
135188
echo " - ${ns}/${cert}: native wildcard already present; nothing to do"
136189
continue
137190
fi
@@ -148,10 +201,8 @@ ensure_wildcard_sans() {
148201
# Wait for cert-manager to re-issue the Secret with the new SANs.
149202
local i
150203
for i in $(seq 1 30); do
151-
if kubectl -n "$ns" get secret "$secret" \
152-
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null \
153-
| tr ',' '\n' | grep -qxF "$native"; then
154-
echo " ${ns}/${secret}: re-issued with native wildcard"
204+
if _secret_has_san "$ns" "$secret" "$native"; then
205+
echo " ${ns}/${secret}: re-issued Secret now carries the native wildcard"
155206
break
156207
fi
157208
[ "$i" = "30" ] && { echo " ERROR: ${ns}/${secret} not re-issued with ${native} in time" >&2; exit 1; }
@@ -331,9 +382,44 @@ if [ "$LEGACY_COUNT" -gt 0 ]; then
331382
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' 2>/dev/null)
332383
fi
333384

385+
# etcd-migrate (unlike kubectl) does NOT fall back to the in-cluster
386+
# ServiceAccount: it only reads a kubeconfig FILE (-k/--kubeconfig, default
387+
# /root/.kube/config), which does not exist in this Job pod. Synthesize one from
388+
# the mounted ServiceAccount so etcd-migrate authenticates as the hook SA.
389+
ETCD_MIGRATE_KUBECONFIG="${ETCD_MIGRATE_KUBECONFIG:-/tmp/in-cluster.kubeconfig}"
390+
_sa_dir="${ETCD_ADOPT_SA_DIR:-/var/run/secrets/kubernetes.io/serviceaccount}"
391+
if [ ! -f "$ETCD_MIGRATE_KUBECONFIG" ] && [ -f "$_sa_dir/token" ]; then
392+
# server is the in-cluster DNS name, NOT https://$KUBERNETES_SERVICE_HOST:PORT:
393+
# on an IPv6-only cluster KUBERNETES_SERVICE_HOST is a bare IPv6 address that
394+
# would need bracketing to be a valid URL. kubernetes.default.svc is
395+
# IP-family-agnostic, resolves in-cluster, and is validated by the mounted SA
396+
# CA — the same server URL kubectl itself synthesizes in-cluster.
397+
cat > "$ETCD_MIGRATE_KUBECONFIG" <<KUBECONFIG_EOF
398+
apiVersion: v1
399+
kind: Config
400+
clusters:
401+
- name: in-cluster
402+
cluster:
403+
certificate-authority: ${_sa_dir}/ca.crt
404+
server: https://kubernetes.default.svc
405+
contexts:
406+
- name: in-cluster
407+
context:
408+
cluster: in-cluster
409+
user: sa
410+
namespace: $(cat "$_sa_dir/namespace")
411+
current-context: in-cluster
412+
users:
413+
- name: sa
414+
user:
415+
tokenFile: ${_sa_dir}/token
416+
KUBECONFIG_EOF
417+
fi
418+
334419
# Always show the plan first.
335420
echo "=== etcd-migrate dry-run ==="
336421
etcd-migrate \
422+
--kubeconfig="${ETCD_MIGRATE_KUBECONFIG}" \
337423
--new-controller="${ETCD_OPERATOR_NS}/${ETCD_OPERATOR_DEPLOY}" \
338424
--skip-controller-check || true
339425

@@ -379,6 +465,7 @@ if [ "$LEGACY_COUNT" -gt 0 ]; then
379465
# 0 is still the LEGACY operator, so resolveAgentImage would otherwise bake the
380466
# wrong (aenix) image into the snapshot Job.
381467
etcd-migrate --apply --yes \
468+
--kubeconfig="${ETCD_MIGRATE_KUBECONFIG}" \
382469
--new-controller="${ETCD_OPERATOR_NS}/${ETCD_OPERATOR_DEPLOY}" \
383470
--skip-controller-check \
384471
--agent-image="${ETCD_OPERATOR_IMAGE}" \

‎packages/extra/etcd/templates/etcd-cluster.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -275,3 +275,50 @@ spec:
275275
app.kubernetes.io/managed-by: etcd-operator
276276
app.kubernetes.io/name: etcd
277277
version: {{ $.Chart.Version }}
278+
---
279+
# Transitional headless Service backing the LEGACY per-pod DNS names.
280+
#
281+
# `etcd-migrate` adopts legacy clusters IN PLACE — the existing Pods are never
282+
# recreated, so they keep their original `spec.subdomain: etcd-headless` (and
283+
# hostname etcd-<i>). Their stable DNS name therefore stays
284+
# `etcd-<i>.etcd-headless.<ns>.svc`, which the v1alpha2 operator dials for
285+
# MemberList/health until the members are eventually rolled onto its native
286+
# `<member>.etcd.<ns>.svc` domain (served by the operator-owned `etcd` Service).
287+
#
288+
# The v1alpha2 operator only creates the native `etcd` Service, and the legacy
289+
# `etcd-headless` Service (previously owned by the old operator) is pruned during
290+
# the transition. Without a Service named `etcd-headless`, the adopted Pods'
291+
# `etcd-<i>.etcd-headless.<ns>.svc` records stop resolving (`no such host`),
292+
# MemberList fails, and status.readyMembers never populates — the cluster never
293+
# goes Ready even though the etcd processes are healthy and in quorum. This is
294+
# the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` wildcard kept in
295+
# the server/peer cert SANs above for the same transition window.
296+
#
297+
# Selector mirrors the operator's native `etcd` Service
298+
# (etcd-operator.cozystack.io/cluster), so this resolves the same member Pods;
299+
# publishNotReadyAddresses keeps peer discovery working while a member is not
300+
# yet Ready. Safe to remove — together with the legacy cert SAN — once the
301+
# members have been rolled onto the native `etcd` subdomain.
302+
apiVersion: v1
303+
kind: Service
304+
metadata:
305+
name: etcd-headless
306+
namespace: {{ .Release.Namespace }}
307+
labels:
308+
app.kubernetes.io/name: etcd
309+
app.kubernetes.io/instance: {{ .Release.Name }}
310+
cozystack.io/service: etcd
311+
spec:
312+
clusterIP: None
313+
publishNotReadyAddresses: true
314+
selector:
315+
etcd-operator.cozystack.io/cluster: etcd
316+
ports:
317+
- name: client
318+
port: 2379
319+
targetPort: 2379
320+
protocol: TCP
321+
- name: peer
322+
port: 2380
323+
targetPort: 2380
324+
protocol: TCP

‎packages/extra/etcd/tests/etcd-cluster_test.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,40 @@ tests:
3838
- notEqual:
3939
path: spec.version
4040
value: "3.5.12"
41+
42+
- it: ships the transitional etcd-headless Service that keeps legacy per-pod DNS resolving during v1alpha2 adoption
43+
# Adopted Pods keep their legacy `etcd-<i>.etcd-headless.<ns>.svc` DNS (never
44+
# recreated), which the operator dials for MemberList until members roll onto
45+
# the native domain. It must be headless, publish not-ready addresses (so
46+
# peer discovery works before a member is Ready), select the operator's
47+
# member Pods, and expose the client/peer ports.
48+
documentSelector:
49+
path: kind
50+
value: Service
51+
asserts:
52+
- equal:
53+
path: metadata.name
54+
value: etcd-headless
55+
- equal:
56+
path: spec.clusterIP
57+
value: None
58+
- equal:
59+
path: spec.publishNotReadyAddresses
60+
value: true
61+
- equal:
62+
path: spec.selector["etcd-operator.cozystack.io/cluster"]
63+
value: etcd
64+
- contains:
65+
path: spec.ports
66+
content:
67+
name: client
68+
port: 2379
69+
targetPort: 2379
70+
protocol: TCP
71+
- contains:
72+
path: spec.ports
73+
content:
74+
name: peer
75+
port: 2380
76+
targetPort: 2380
77+
protocol: TCP

0 commit comments

Comments
 (0)