Skip to content

Commit b89b159

Browse files
committed
fix(platform): make etcd v1alpha2 adoption (migration 50) robust in-cluster
Migration 50 (etcd.aenix.io -> etcd-operator.cozystack.io/v1alpha2 adoption) had two defects that blocked every in-cluster 1.5 -> 1.6 upgrade on a cluster with an existing etcd: 1. Cert-SAN wait treated transient kubectl failures as "SAN absent". ensure_wildcard_sans checked/awaited the wildcard SAN with `kubectl get ... 2>/dev/null | grep`, so any transient GET failure (API discovery refresh, apiserver blip, throttling) produced an empty string indistinguishable from a genuine absence -> false miss, and the 120s wait never recovered. Replace the two ad-hoc checks with a _san_present helper that retries on an empty read (a real Certificate/Secret never has empty dnsNames/alt-names) and accepts the native wildcard from EITHER the issued Secret's cert-manager.io/alt-names annotation OR the Certificate spec.dnsNames (the source of truth for what cert-manager will issue). 2. etcd-migrate had no kubeconfig in-cluster. etcd-migrate only reads a kubeconfig file (-k/--kubeconfig, default /root/.kube/config) and, unlike kubectl, does not fall back to the mounted in-cluster ServiceAccount. The hook Job set no KUBECONFIG and passed no --kubeconfig, so both the dry-run and --apply aborted with "error building kubeconfig: stat /root/.kube/config: no such file". Synthesize an in-cluster kubeconfig from the mounted ServiceAccount and pass --kubeconfig to both etcd-migrate invocations. Verified end-to-end on a 1.5.2 -> 1.6.0-rc.1 upgrade: the adoption now completes in-place (pods never restarted, data intact) and the cluster reaches readyMembers=3 / Available=True. Refs: #3243, #3255 Signed-off-by: Andrey Kolkov <[email protected]>
1 parent f91841f commit b89b159

1 file changed

Lines changed: 60 additions & 6 deletions

File tree

  • packages/core/platform/images/migrations/migrations

‎packages/core/platform/images/migrations/migrations/50‎

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,33 @@ CREDS_TENANT_LABEL="internal.cozystack.io/tenantresource"
114114
# templates declare only the short forms, so adding FQDN SANs here would be
115115
# stripped on the next HelmRelease reconcile — adoption and chart output must
116116
# converge to the same fixed point on the cert SANs.
117+
# _san_present <ns> <secret> <cert> <native>
118+
# Returns 0 iff <native> is present in the issued Secret's cert-manager.io/alt-names
119+
# annotation OR the Certificate's spec.dnsNames (the source of truth for the SANs
120+
# cert-manager will issue). ROBUSTNESS: a plain `kubectl get ... 2>/dev/null` can
121+
# transiently return an EMPTY string on an API hiccup (discovery refresh, apiserver
122+
# blip, throttling), which is indistinguishable from "SAN genuinely absent" and
123+
# makes a correctly-configured cert look missing — the failure mode that wedges the
124+
# cert wait below. Neither field is ever legitimately empty on a real object, so we
125+
# treat an empty read from BOTH sources as a transient error and retry, only
126+
# trusting a non-empty read to decide presence/absence.
127+
_san_present() {
128+
local ns="$1" secret="$2" cert="$3" native="$4" a d try
129+
for try in 1 2 3 4 5; do
130+
a=$(kubectl -n "$ns" get secret "$secret" \
131+
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null || true)
132+
d=$(kubectl -n "$ns" get certificate.cert-manager.io "$cert" \
133+
-o jsonpath='{.spec.dnsNames}' 2>/dev/null || true)
134+
if [ -n "$a" ] || [ -n "$d" ]; then
135+
printf '%s' "$a" | tr ',' '\n' | grep -qxF "$native" && return 0
136+
printf '%s' "$d" | tr ',' '\n' | grep -qF "$native" && return 0
137+
return 1
138+
fi
139+
sleep 1
140+
done
141+
return 1
142+
}
143+
117144
ensure_wildcard_sans() {
118145
local ns="$1" name="$2"
119146
local native="*.${name}.${ns}.svc"
@@ -129,9 +156,7 @@ ensure_wildcard_sans() {
129156
echo " - ${ns}/${cert}: Certificate not found; skipping (operator-managed or already pruned)"
130157
continue
131158
fi
132-
if kubectl -n "$ns" get secret "$secret" \
133-
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null \
134-
| tr ',' '\n' | grep -qxF "$native"; then
159+
if _san_present "$ns" "$secret" "$cert" "$native"; then
135160
echo " - ${ns}/${cert}: native wildcard already present; nothing to do"
136161
continue
137162
fi
@@ -148,9 +173,7 @@ ensure_wildcard_sans() {
148173
# Wait for cert-manager to re-issue the Secret with the new SANs.
149174
local i
150175
for i in $(seq 1 30); do
151-
if kubectl -n "$ns" get secret "$secret" \
152-
-o jsonpath='{.metadata.annotations.cert-manager\.io/alt-names}' 2>/dev/null \
153-
| tr ',' '\n' | grep -qxF "$native"; then
176+
if _san_present "$ns" "$secret" "$cert" "$native"; then
154177
echo " ${ns}/${secret}: re-issued with native wildcard"
155178
break
156179
fi
@@ -331,9 +354,39 @@ if [ "$LEGACY_COUNT" -gt 0 ]; then
331354
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' 2>/dev/null)
332355
fi
333356

357+
# etcd-migrate (unlike kubectl) does NOT fall back to the in-cluster
358+
# ServiceAccount: it only reads a kubeconfig FILE (-k/--kubeconfig, default
359+
# /root/.kube/config), which does not exist in this Job pod. Synthesize one from
360+
# the mounted ServiceAccount so etcd-migrate authenticates as the hook SA.
361+
ETCD_MIGRATE_KUBECONFIG="/tmp/in-cluster.kubeconfig"
362+
_sa_dir="/var/run/secrets/kubernetes.io/serviceaccount"
363+
if [ ! -f "$ETCD_MIGRATE_KUBECONFIG" ] && [ -f "$_sa_dir/token" ]; then
364+
cat > "$ETCD_MIGRATE_KUBECONFIG" <<KUBECONFIG_EOF
365+
apiVersion: v1
366+
kind: Config
367+
clusters:
368+
- name: in-cluster
369+
cluster:
370+
certificate-authority: ${_sa_dir}/ca.crt
371+
server: https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT:-443}
372+
contexts:
373+
- name: in-cluster
374+
context:
375+
cluster: in-cluster
376+
user: sa
377+
namespace: $(cat "$_sa_dir/namespace")
378+
current-context: in-cluster
379+
users:
380+
- name: sa
381+
user:
382+
tokenFile: ${_sa_dir}/token
383+
KUBECONFIG_EOF
384+
fi
385+
334386
# Always show the plan first.
335387
echo "=== etcd-migrate dry-run ==="
336388
etcd-migrate \
389+
--kubeconfig="${ETCD_MIGRATE_KUBECONFIG}" \
337390
--new-controller="${ETCD_OPERATOR_NS}/${ETCD_OPERATOR_DEPLOY}" \
338391
--skip-controller-check || true
339392

@@ -379,6 +432,7 @@ if [ "$LEGACY_COUNT" -gt 0 ]; then
379432
# 0 is still the LEGACY operator, so resolveAgentImage would otherwise bake the
380433
# wrong (aenix) image into the snapshot Job.
381434
etcd-migrate --apply --yes \
435+
--kubeconfig="${ETCD_MIGRATE_KUBECONFIG}" \
382436
--new-controller="${ETCD_OPERATOR_NS}/${ETCD_OPERATOR_DEPLOY}" \
383437
--skip-controller-check \
384438
--agent-image="${ETCD_OPERATOR_IMAGE}" \

0 commit comments

Comments
 (0)