Skip to content

Commit efa85a4

Browse files
feat(platform): make the etcd-adoption skip-backup hatch reachable
Migration 50 has honoured ETCD_ADOPT_SKIP_BACKUP since it was written, and its failure paths tell operators to "re-run with ETCD_ADOPT_SKIP_BACKUP=1". But migration-hook.yaml passes only NAMESPACE/CURRENT_VERSION/ TARGET_VERSION, and the Job is a Helm hook the chart re-renders, so there has never been a way to set it. The script advertised an escape the platform made impossible. The endpoint-classification work landing alongside this makes that worse: it adds refuse-rather-than-guess paths, so more clusters can now legitimately stop here. Plumb it through migrations.etcdAdoptSkipBackup, default false. Skipping the pre-adoption snapshot rewires ownership of LIVE etcd storage with no way back, so it stays opt-in and deliberate — the flag is the last resort for a cluster that has nothing to fix, not the first response to a refusal. Render "1"/"0" rather than the bare bool. The script matches the literal string "1", so a bool would render "true", sail through the Job spec looking correct, and be ignored — a safety valve that is set but silently disregarded is worse than one that was never offered, because the operator believes they opted in. The value is normalised through a string, so a bool, a quoted "true" out of the Package CR's Values JSON, or a numeric 1 all land on the same answer, while nil or garbage resolves to "0": taking this hatch must require an affirmative spelling, never a typo. Resolved with `dig` so a Platform Package predating the key renders "0" instead of failing the render of the entire platform chart and bricking every upgrade. Widen the script's own check to accept 1/true/yes as well. The template guarantees the GitOps path, but an operator running this image by hand will reasonably type "true", and that is exactly the moment they are already stuck. The two layers cover independent failure modes. Emit the var unconditionally, including the "0" default, so the rendered Job states the cluster's safety posture outright rather than leaving an operator to diff values to find out whether a snapshot was skipped. Point the failure messages at the values path that actually exists, and describe it honestly: editing the Platform Package and letting Flux re-render is a real operation, not a one-liner, and the flag has to be reverted afterwards or the next etcd migration skips its snapshot too. Fixing the condition the migration names remains the recommended path — the migration is idempotent, so re-running after a fix is safe. templates/migration-hook.yaml had no coverage at all, because the Job only renders when a lookup of the cozystack-version ConfigMap reports a version below targetVersion, and lookup is nil under helm template. The new suite mocks that ConfigMap and asserts the RENDERED VALUE — enabled, disabled, key absent, stringified true, garbage, and the pre-existing three env vars — rather than merely that the variable exists, which is what let the trap above go unnoticed in the first place. Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 0ba5337 commit efa85a4

5 files changed

Lines changed: 264 additions & 6 deletions

File tree

‎hack/migration-50-etcd-adopt.bats‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,12 @@ lineno() {
207207
cat "$FAKE_CMDLOG"
208208
[ "$rc" -ne 0 ]
209209
grep -qF -- "refusing to adopt" "$WORK/out"
210+
# The refusal must tell the operator how to actually take the hatch. Naming
211+
# ETCD_ADOPT_SKIP_BACKUP alone was unactionable advice for the entire life of
212+
# this migration: nothing can set a var on a Helm-hook Job, so the message has
213+
# to name the Package CR path that renders it.
214+
grep -qF -- "package.cozystack.io cozystack.cozystack-platform" "$WORK/out"
215+
grep -qF -- "etcdAdoptSkipBackup: true" "$WORK/out"
210216
# Nothing destructive must have run: no scale-down, no --apply, no version stamp.
211217
! grep -qF -- "SCALE 0" "$FAKE_CMDLOG"
212218
! grep -qF -- "ETCD-MIGRATE --apply" "$FAKE_CMDLOG"
@@ -572,6 +578,41 @@ tenant-old bucket-dead bucket-dead-uuid 2026-07-17T09:00:00Z"
572578
rm -rf "$WORK"
573579
}
574580

581+
@test "ETCD_ADOPT_SKIP_BACKUP=true is honoured, not silently ignored" {
582+
prep
583+
# The platform renders this var through migrations.etcdAdoptSkipBackup and
584+
# always emits "1", but an operator setting it by hand will reasonably type
585+
# "true". An exact match on "1" would accept that, show it in the Job spec and
586+
# do nothing — a safety valve that is set but silently ignored is worse than
587+
# one that was never offered, because the operator believes they opted in.
588+
export ETCD_ADOPT_SKIP_BACKUP=true
589+
rc=0
590+
bash "$MIG" >"$WORK/out" 2>&1 || rc=$?
591+
cat "$WORK/out"
592+
[ "$rc" -eq 0 ]
593+
apply=$(grep -F -- "ETCD-MIGRATE --apply" "$FAKE_CMDLOG")
594+
echo "$apply" | grep -qF -- "--skip-backup"
595+
! echo "$apply" | grep -qF -- "--backup-s3-endpoint"
596+
! grep -qF -- "STAGE " "$FAKE_CMDLOG"
597+
rm -rf "$WORK"
598+
}
599+
600+
@test "an unrecognised ETCD_ADOPT_SKIP_BACKUP value does NOT skip the snapshot" {
601+
prep
602+
# Skipping the snapshot on live etcd must require an affirmative answer. A
603+
# typo, a stray value, or a future template bug rendering something unexpected
604+
# must fall back to the safe side and still take the snapshot.
605+
export ETCD_ADOPT_SKIP_BACKUP=banana
606+
rc=0
607+
bash "$MIG" >"$WORK/out" 2>&1 || rc=$?
608+
cat "$WORK/out"
609+
[ "$rc" -eq 0 ]
610+
apply=$(grep -F -- "ETCD-MIGRATE --apply" "$FAKE_CMDLOG")
611+
! echo "$apply" | grep -qF -- "--skip-backup"
612+
echo "$apply" | grep -qF -- "--backup-s3-endpoint=https://s3.example.com"
613+
rm -rf "$WORK"
614+
}
615+
575616
@test "ETCD_ADOPT_SKIP_BACKUP=1 unblocks adoption when no backup target resolves" {
576617
prep
577618
# The narrowing the reviewer asked for: a cluster with legacy etcd but no

‎packages/core/platform/images/migrations/migrations/50‎

Lines changed: 47 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,47 @@ DRY_RUN="${MIGRATION_DRY_RUN:-0}"
2929
# than turn "no backup storage" into a total upgrade block, an operator who
3030
# accepts adopting live etcd WITHOUT a snapshot can set ETCD_ADOPT_SKIP_BACKUP=1
3131
# to pass etcd-migrate's --skip-backup. Default 0 keeps the fail-loud behaviour.
32+
# The platform renders this from migrations.etcdAdoptSkipBackup
33+
# (templates/migration-hook.yaml), which is the only route an operator has:
34+
# nothing else can set a var on a Helm-hook Job. Keep the two in lockstep —
35+
# an unreachable hatch is what the failure messages below used to advertise.
3236
SKIP_BACKUP="${ETCD_ADOPT_SKIP_BACKUP:-0}"
3337

38+
# _is_true accepts the spellings an operator can plausibly arrive at. The
39+
# platform renders this var through migrations.etcdAdoptSkipBackup and always
40+
# emits "1"/"0", so the extra spellings are for the hand-set paths (running this
41+
# image directly, or a Job edited in place) where "true" is the natural thing to
42+
# type. A safety valve that is set but silently ignored is the worst outcome
43+
# here — worse than one that is not offered at all — so recognise the obvious
44+
# spellings rather than only the documented one. Everything else, including
45+
# typos, stays false: skipping the snapshot must need an affirmative answer.
46+
_is_true() {
47+
case "$(printf '%s' "${1:-}" | tr '[:upper:]' '[:lower:]')" in
48+
1 | true | yes) return 0 ;;
49+
*) return 1 ;;
50+
esac
51+
}
52+
53+
# _print_skip_backup_hatch spells out how to actually take the hatch. Naming the
54+
# env var alone is useless advice: this Job is a Helm hook rendered by the
55+
# platform chart, so an operator cannot set the variable directly and nothing
56+
# they can edit survives the next render. The Package CR is the only supported
57+
# route (it is cluster-scoped; the platform chart's own values sit under
58+
# components.platform). Deliberately not sold as a quick fix — it is a values
59+
# change plus a re-reconcile, and it must be reverted afterwards.
60+
_print_skip_backup_hatch() {
61+
echo " kubectl edit package.cozystack.io cozystack.cozystack-platform" >&2
62+
echo " spec:" >&2
63+
echo " components:" >&2
64+
echo " platform:" >&2
65+
echo " values:" >&2
66+
echo " migrations:" >&2
67+
echo " etcdAdoptSkipBackup: true" >&2
68+
echo " Flux then re-renders the platform chart and re-runs this hook, which adopts" >&2
69+
echo " every legacy etcd WITHOUT a snapshot. Set it back to false once the upgrade" >&2
70+
echo " completes, or the next etcd migration will skip its snapshot too." >&2
71+
}
72+
3473
# Shared cozystack-version stamp helper. Every migration >= 42 must stamp via
3574
# this (labeled, no-delete-guarded) helper rather than a hand-rolled apply — the
3675
# cozystack-version-stamp.bats guard enforces it. Provides stamp_cozystack_version.
@@ -571,10 +610,11 @@ fi
571610
# ETCD_ADOPT_SKIP_BACKUP=1 escape hatch; any inbound ETCD_MIGRATE_BACKUP_ARGS is
572611
# overwritten here on purpose (it can never suppress the snapshot).
573612
if [ "$LEGACY_COUNT" -gt 0 ]; then
574-
if [ "$SKIP_BACKUP" = "1" ]; then
575-
echo "WARNING: ETCD_ADOPT_SKIP_BACKUP=1 — adopting live etcd WITHOUT a pre-adoption" >&2
576-
echo " safety snapshot, by explicit operator request. A botched adoption on a" >&2
577-
echo " tenant control-plane etcd is UNRECOVERABLE with no snapshot." >&2
613+
if _is_true "$SKIP_BACKUP"; then
614+
echo "WARNING: migrations.etcdAdoptSkipBackup is set — adopting live etcd WITHOUT a" >&2
615+
echo " pre-adoption safety snapshot, by explicit operator request. A botched adoption" >&2
616+
echo " on a tenant control-plane etcd is UNRECOVERABLE with no snapshot. Reset the" >&2
617+
echo " flag to false once this upgrade is through." >&2
578618
ETCD_MIGRATE_BACKUP_ARGS="--skip-backup"
579619
else
580620
echo "Resolving platform-managed S3 snapshot target ('${PLATFORM_STRATEGY}' strategy)..."
@@ -590,7 +630,8 @@ if [ "$LEGACY_COUNT" -gt 0 ]; then
590630
echo " cozy-backups Bucket and the backupstrategy-controller are healthy, then re-run." >&2
591631
echo " If this cluster intentionally has no backup storage (backups disabled," >&2
592632
echo " external S3 without staged creds, or SeaweedFS not yet ready) and you accept" >&2
593-
echo " adopting live etcd WITHOUT a snapshot, re-run with ETCD_ADOPT_SKIP_BACKUP=1." >&2
633+
echo " adopting live etcd WITHOUT a snapshot, take the escape hatch:" >&2
634+
_print_skip_backup_hatch
594635
exit 1
595636
fi
596637
fi
@@ -659,7 +700,7 @@ if [ "$LEGACY_COUNT" -gt 0 ]; then
659700
# etcd-migrate's per-namespace snapshot Job mounts PLATFORM_CREDS_SECRET there.
660701
# Skipped entirely when ETCD_ADOPT_SKIP_BACKUP=1 (no snapshot Job runs, so no
661702
# creds are needed) — adoption below still proceeds.
662-
if [ "$SKIP_BACKUP" != "1" ]; then
703+
if ! _is_true "$SKIP_BACKUP"; then
663704
echo "=== staging snapshot credentials ('${PLATFORM_CREDS_SECRET}') into adopted namespaces ==="
664705
while read -r ns name; do
665706
[ -n "$ns" ] || continue

‎packages/core/platform/templates/migration-hook.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,20 @@
22
{{- $shouldRunMigrationHook := false }}
33
{{- $currentVersion := 0 }}
44
{{- $targetVersion := .Values.migrations.targetVersion | int }}
5+
{{- /*
6+
Resolve migrations.etcdAdoptSkipBackup to the exact string migration 50
7+
matches. Two hazards, both silent:
8+
- the key is absent on a Platform Package that predates it, so `dig` supplies
9+
the default rather than letting the whole platform render fail;
10+
- the script tests for "1", so a bare bool renders "true" and the hatch is
11+
accepted, displayed in the Job spec, and ignored. Normalise through a
12+
string so a bool, a quoted "true" out of the Package CR's Values JSON, or
13+
a numeric 1 all land on the same answer.
14+
Anything unrecognised (nil, garbage) resolves to "0": taking the hatch must
15+
require an affirmative spelling, never a typo.
16+
*/}}
17+
{{- $skipBackupRaw := dig "etcdAdoptSkipBackup" false .Values.migrations }}
18+
{{- $skipBackup := has (lower (printf "%v" $skipBackupRaw)) (list "1" "true" "yes") }}
519
{{- $configMap := lookup "v1" "ConfigMap" .Release.Namespace "cozystack-version" }}
620
{{- if $configMap }}
721
{{- $currentVersion = dig "data" "version" "0" $configMap | int }}
@@ -38,6 +52,11 @@ spec:
3852
value: {{ $currentVersion | quote }}
3953
- name: TARGET_VERSION
4054
value: {{ $targetVersion | quote }}
55+
# Always emitted, including the "0" default, so the Job spec states
56+
# the cluster's safety posture outright: an operator reading the Job
57+
# can see whether the snapshot was skipped without diffing values.
58+
- name: ETCD_ADOPT_SKIP_BACKUP
59+
value: {{ $skipBackup | ternary "1" "0" | quote }}
4160
restartPolicy: Never
4261
---
4362
apiVersion: rbac.authorization.k8s.io/v1
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
suite: migrations.etcdAdoptSkipBackup reaches the migration Job as the value the script matches
2+
templates:
3+
- templates/migration-hook.yaml
4+
release:
5+
name: cozystack
6+
namespace: cozy-system
7+
# The Job only renders when the cozystack-version ConfigMap reports a version
8+
# BELOW migrations.targetVersion. Under plain `helm template` (and therefore
9+
# `helm lint`/CI) lookup returns nil and the whole hook is skipped, so the env
10+
# block is unreachable without mocking the ConfigMap — which is precisely why
11+
# this template had no coverage and shipped an unreachable escape hatch.
12+
kubernetesProvider:
13+
scheme:
14+
"v1/ConfigMap":
15+
gvr:
16+
version: "v1"
17+
resource: "configmaps"
18+
namespaced: true
19+
objects:
20+
- kind: ConfigMap
21+
apiVersion: v1
22+
metadata:
23+
name: cozystack-version
24+
namespace: cozy-system
25+
data:
26+
version: "52"
27+
tests:
28+
# The trap this suite exists for: migration 50 tests the env var against the
29+
# literal string "1". A bare bool renders "true", which the script does not
30+
# match, so the hatch would appear in the Job spec and silently do nothing —
31+
# the worst outcome for a safety valve an operator reaches for when stuck.
32+
# Assert the RENDERED VALUE, never merely that the var exists.
33+
- it: renders ETCD_ADOPT_SKIP_BACKUP=1 (not "true") when the flag is enabled
34+
set:
35+
migrations:
36+
enabled: true
37+
etcdAdoptSkipBackup: true
38+
documentSelector:
39+
path: kind
40+
value: Job
41+
asserts:
42+
- equal:
43+
path: spec.template.spec.containers[0].env[3].name
44+
value: ETCD_ADOPT_SKIP_BACKUP
45+
- equal:
46+
path: spec.template.spec.containers[0].env[3].value
47+
value: "1"
48+
49+
- it: renders ETCD_ADOPT_SKIP_BACKUP=0 when the flag is explicitly disabled
50+
set:
51+
migrations:
52+
enabled: true
53+
etcdAdoptSkipBackup: false
54+
documentSelector:
55+
path: kind
56+
value: Job
57+
asserts:
58+
- equal:
59+
path: spec.template.spec.containers[0].env[3].value
60+
value: "0"
61+
62+
# A Platform Package predating this knob carries no such key. The chart's own
63+
# values.yaml supplies the default on merge, but the template must not depend
64+
# on that: a nil here would fail the render of the ENTIRE platform chart and
65+
# brick every upgrade — far worse than the hatch being unavailable.
66+
- it: renders cleanly and OFF when the key is absent entirely (pre-existing Package CR)
67+
set:
68+
migrations:
69+
enabled: true
70+
etcdAdoptSkipBackup: null
71+
documentSelector:
72+
path: kind
73+
value: Job
74+
asserts:
75+
- equal:
76+
path: spec.template.spec.containers[0].env[3].value
77+
value: "0"
78+
79+
# Values arrive as JSON from the Package CR, where a quoted "true" is an easy
80+
# mistake. It must take the hatch rather than silently render "0".
81+
- it: accepts a stringified true from the Package CR Values JSON
82+
set:
83+
migrations:
84+
enabled: true
85+
etcdAdoptSkipBackup: "true"
86+
documentSelector:
87+
path: kind
88+
value: Job
89+
asserts:
90+
- equal:
91+
path: spec.template.spec.containers[0].env[3].value
92+
value: "1"
93+
94+
# The dangerous direction must require an affirmative spelling: anything
95+
# unrecognised resolves to "0" rather than being treated as truthy.
96+
- it: treats an unrecognised value as OFF, never as an accidental skip
97+
set:
98+
migrations:
99+
enabled: true
100+
etcdAdoptSkipBackup: "banana"
101+
documentSelector:
102+
path: kind
103+
value: Job
104+
asserts:
105+
- equal:
106+
path: spec.template.spec.containers[0].env[3].value
107+
value: "0"
108+
109+
# The pre-existing env contract must survive: the script reads all four.
110+
- it: keeps NAMESPACE/CURRENT_VERSION/TARGET_VERSION alongside the new var
111+
set:
112+
migrations:
113+
enabled: true
114+
etcdAdoptSkipBackup: true
115+
documentSelector:
116+
path: kind
117+
value: Job
118+
asserts:
119+
- equal:
120+
path: spec.template.spec.containers[0].env[0].name
121+
value: NAMESPACE
122+
- equal:
123+
path: spec.template.spec.containers[0].env[1].name
124+
value: CURRENT_VERSION
125+
- equal:
126+
path: spec.template.spec.containers[0].env[1].value
127+
value: "52"
128+
- equal:
129+
path: spec.template.spec.containers[0].env[2].name
130+
value: TARGET_VERSION
131+
132+
- it: emits no Job at all when migrations are disabled
133+
set:
134+
migrations:
135+
enabled: false
136+
etcdAdoptSkipBackup: true
137+
asserts:
138+
- hasDocuments:
139+
count: 0

‎packages/core/platform/values.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,24 @@ migrations:
1515
# a stale pin fails loudly rather than silently skipping the etcd adoption.
1616
image: ghcr.io/cozystack/cozystack/platform-migrations:v1.5.0@sha256:8bf61f17e99eb4a3365394e3d97ac52a1d68cc84aa104894ecb09de25721dc1d
1717
targetVersion: 53
18+
# Adopt legacy etcd clusters onto the v1alpha2 operator WITHOUT the mandatory
19+
# pre-adoption safety snapshot (migration 50). Leave false unless migration 50
20+
# has actually refused to proceed and you accept the risk: adoption rewires
21+
# ownership of LIVE etcd storage, and a botched adoption on a tenant
22+
# control-plane etcd is UNRECOVERABLE with no snapshot.
23+
#
24+
# Migration 50 refuses when it cannot resolve, or cannot trust, a snapshot
25+
# destination — backups disabled, an external S3 whose credentials were never
26+
# staged, SeaweedFS not ready at upgrade time, or a bucket it cannot classify.
27+
# Those are individually-supported configurations, so rather than leave such a
28+
# cluster permanently unable to upgrade, this flag lets an operator proceed
29+
# deliberately. Prefer fixing the condition the migration names in its log:
30+
# the migration is idempotent, so re-running after the fix is the safe path,
31+
# and this flag is the last resort when there is nothing to fix.
32+
#
33+
# Reset it to false once the upgrade is through, or the next platform upgrade
34+
# that adds an etcd migration will skip its snapshot too.
35+
etcdAdoptSkipBackup: false
1836
# Bundle deployment configuration
1937
bundles:
2038
system:

0 commit comments

Comments
 (0)