Skip to content

Commit 8499b5f

Browse files
fix(seaweedfs): move the naming-migration guard into the render that actually renames
The class-S guard from #3282 lives in extra/seaweedfs, but that render is not in the path of a platform upgrade: the <name>-system HelmRelease pulls system/seaweedfs from a platform-managed ExternalArtifact, so bumping the platform upgrades the release directly from the new artifact and applies fullnameOverride: seaweedfs without extra/seaweedfs ever re-rendering. On a tenant installed fresh on 1.5.x that stood up an empty chart-named set beside the live release-named data, deleted the renamed s3 Service, and left S3 readable only because the empty set's pods cannot start. Worse, the damage flips the guard's own classification: creating the chart-named artifacts makes $legacyData true, so the guard then reports D-split and sends the operator to runbook Step 3 — which quiesces the renamed set, i.e. the one holding ALL the data. - Add the ENFORCING guard to system/seaweedfs (templates/naming-guard.yaml): the only render guaranteed to sit between a platform upgrade and the tenant's workloads. It also refuses to render if the fullnameOverride pin is ever lifted. - Tell S from D by PVC AGE when both generations exist: PVCs are never recreated in place (StatefulSets are, by the adoption hook), so the older volume-PVC generation is where the data was born. Renamed-older ⇒ class S (damaged variant, new Step 2a), never D-split. - Fail closed: the release namespace is a lookup canary; an upgrade that cannot see the cluster refuses instead of classifying blind. Client-side installs (CI lint/unittest) still render — they never touch live data. - Mirror the same classification in extra/seaweedfs so the operator-facing SeaweedFS application reports the correct class. - Tests now model the ExternalArtifact upgrade path in the chart it actually runs through (the old ones mocked lookups only in extra/seaweedfs, which is why this shipped green), including the damaged-S tenant observed live. - Runbook: add S-damaged audit classification (PVC age), Step 2a cleanup of the empty chart-named set, and the D-wedged anti-affinity stall workaround observed on 3-node clusters. helm-unittest 1.0.3 ignores release.isUpgrade, so the canary refusal is verified with 'helm template --is-upgrade' instead of a unit test. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 6338b0b commit 8499b5f

8 files changed

Lines changed: 696 additions & 150 deletions

File tree

‎docs/operations/seaweedfs-431-rename-recovery.md‎

Lines changed: 47 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Before 4.31 the chart named workloads after the chart (`seaweedfs-*`), ignoring
1111

1212
The fix pins `fullnameOverride: seaweedfs` in `system/seaweedfs` values, so workloads are always named after the chart, exactly as they were before 4.31. Upgrading past the bump therefore **adopts the running set and its volumes in place**.
1313

14-
One class cannot be adopted that way: a tenant installed **fresh on 1.5.x**, whose data was written under the release-based names and lives on `data1-seaweedfs-system-volume-*` PVCs. Pinning the chart name there would rename the workloads *away* from that data. Helm cannot move data between PVCs, so `extra/seaweedfs` refuses to render for such a tenant and points here. Re-bind its volumes (below) before upgrading.
14+
One class cannot be adopted that way: a tenant installed **fresh on 1.5.x**, whose data was written under the release-based names and lives on `data1-seaweedfs-system-volume-*` PVCs. Pinning the chart name there would rename the workloads *away* from that data. Helm cannot move data between PVCs, so the charts refuse to render for such a tenant and point here. The **enforcing** guard lives in `system/seaweedfs` (`templates/naming-guard.yaml`) — the `<name>-system` HelmRelease pulls that chart straight from a platform-managed ExternalArtifact, so a platform upgrade re-renders it directly and nothing else stands between the upgrade and the tenant's workloads; `extra/seaweedfs` carries a sibling copy so the refusal is also visible on the SeaweedFS application itself. Re-bind the tenant's volumes (below) before upgrading.
1515

1616
## Step 1 — Audit the fleet (read-only)
1717

@@ -30,15 +30,26 @@ nss=$(kubectl get pvc -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "
3030
| awk '$2 ~ /^data1-.*volume/ && $2 ~ /seaweedfs/ {print $1}' | sort -u)
3131
[ -z "${nss}" ] && { echo "(no SeaweedFS volume PVCs found)"; exit 0; }
3232
for ns in ${nss}; do
33-
pvcs=$(kubectl get pvc -n "$ns" -o name | sed 's|persistentvolumeclaim/||')
33+
# name<TAB>creationTimestamp — the AGE ordering below tells a genuine D tenant
34+
# from an S tenant that an unguarded upgrade already damaged.
35+
pvcs=$(kubectl get pvc -n "$ns" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}')
3436
legacy=$(echo "$pvcs" | grep -cE '^data1-seaweedfs-volume' || true)
3537
system=$(echo "$pvcs" | grep -E '^data1-.*volume' | grep -E 'seaweedfs' | grep -vE '^data1-seaweedfs-volume' | grep -c . || true)
38+
legacy_oldest=$(echo "$pvcs" | grep -E '^data1-seaweedfs-volume' | cut -f2 | sort | head -1)
39+
system_oldest=$(echo "$pvcs" | grep -E '^data1-.*volume' | grep -E 'seaweedfs' | grep -vE '^data1-seaweedfs-volume' | cut -f2 | sort | head -1)
3640
running_new=$(kubectl get pods -n "$ns" \
3741
-l app.kubernetes.io/name=seaweedfs,app.kubernetes.io/component=volume \
3842
--field-selector=status.phase=Running -o name 2>/dev/null \
3943
| grep -vc '/seaweedfs-volume-' || true)
4044
if [ "$legacy" -gt 0 ] && [ "$system" -gt 0 ]; then
41-
if [ "${running_new:-0}" -gt 0 ]; then
45+
# PVCs are never recreated in place, so the OLDER generation is where the
46+
# data was born. Renamed older ⇒ a fresh-1.5.x tenant that an unguarded
47+
# 1.6.0 upgrade already hit: the chart-named set is the NEWER, EMPTY one.
48+
# That tenant is S (recover via Step 2a + Step 2), NEVER D-split — the
49+
# D-split procedure would quiesce the set that holds all the data.
50+
if [ "$system_oldest" \< "$legacy_oldest" ]; then
51+
printf '%-28s %-10s %s\n' "$ns" "S-damaged" "renamed volumes are OLDER: data born there, chart-named set is empty (Step 2a, then Step 2)"
52+
elif [ "${running_new:-0}" -gt 0 ]; then
4253
printf '%-28s %-10s %s\n' "$ns" "D-split" "DANGER: duplicate volume servers Running — audit before upgrading"
4354
else
4455
printf '%-28s %-10s %s\n' "$ns" "D-wedged" "duplicate set never served; upgrade adopts the legacy set"
@@ -51,7 +62,16 @@ for ns in ${nss}; do
5162
done
5263
```
5364

54-
Act on the classes in this order: resolve every **D-split** and every **S** tenant first (both need an operator), then upgrade. **L** and **D-wedged** need nothing.
65+
Act on the classes in this order: resolve every **D-split**, **S** and **S-damaged** tenant first (all need an operator), then upgrade. **L** needs nothing.
66+
67+
**D-wedged** normally needs nothing — but on a cluster with no spare nodes (nodes ≤ replicas) the adoption rollout can stall: the wedged duplicate pods are still *scheduled*, carry the same labels as the adopted set (including `app.kubernetes.io/instance`), and their hard pod anti-affinity blocks the adopted set's rolled pods from landing anywhere (observed as `seaweedfs-filer-1`/`seaweedfs-master-2` stuck Pending on `didn't match pod anti-affinity rules`, wedging the `<name>-system` HelmRelease in upgrade/rollback loops). If that happens, scale the duplicate down first — it never served, so this is safe:
68+
69+
```sh
70+
ns=<tenant>
71+
kubectl -n "$ns" get sts -l app.kubernetes.io/name=seaweedfs -o name | sed 's|statefulset.apps/||' \
72+
| grep -vE '^seaweedfs-(master|filer|volume)($|-)' \
73+
| xargs -r -I{} kubectl -n "$ns" scale sts {} --replicas=0
74+
```
5575

5676
## Step 2 — `S` tenants: re-bind the volumes before upgrading
5777

@@ -127,6 +147,29 @@ Then upgrade. The filer metadata lives in the shared `seaweedfs-db` Postgres and
127147

128148
The loop handles pools and zones (their PVC names carry the pool/zone key) and both renamed shapes: `data1-seaweedfs-system-volume-*` for the default instance, and `data1-<name>-system-seaweedfs-volume-*` for an instance running under another name, because 4.31's name helper appends the chart name when the release name does not contain it.
129149

150+
## Step 2a — `S-damaged` tenants: remove the empty chart-named set first
151+
152+
An `S` tenant that an unguarded 1.6.0 upgrade already reached has an extra problem: the upgrade **created** chart-named workloads (`seaweedfs-master/-filer/-volume`, an s3 Deployment) and **empty** `data1-seaweedfs-volume-*` PVCs beside the live renamed set, and deleted the renamed `<fullname>-s3` Service (only `seaweedfs-s3` remains — its endpoints may still resolve to the renamed set's pods because both sets carry identical labels, which is luck, not design: if the chart-named pods ever become Ready, the Service splits reads across a live set and an empty one).
153+
154+
Verify the direction before touching anything — the chart-named PVCs must be the **newer** generation (compare `kubectl get pvc -o custom-columns=NAME:.metadata.name,CREATED:.metadata.creationTimestamp`), matching the `S-damaged` audit row. Then clear the empty chart-named set so Step 2 can re-bind onto those names:
155+
156+
```sh
157+
ns=<tenant>
158+
app=<seaweedfs-instance-name>
159+
kubectl -n "$ns" patch helmrelease "${app}-system" --type merge -p '{"spec":{"suspend":true}}'
160+
# The chart-named workloads were created by the aborted upgrade and never held
161+
# data. Delete them so the re-bind can take over their names.
162+
kubectl -n "$ns" delete sts seaweedfs-master seaweedfs-filer --ignore-not-found
163+
kubectl -n "$ns" get sts -o name | sed 's|statefulset.apps/||' | grep -E '^seaweedfs-volume($|-)' \
164+
| xargs -r -I{} kubectl -n "$ns" delete sts {}
165+
# The EMPTY chart-named claims block Step 2's re-bind (same names). Double-check
166+
# each is the newer, never-served generation before deleting.
167+
kubectl -n "$ns" get pvc -o name | sed 's|persistentvolumeclaim/||' | grep -E '^data1-seaweedfs-volume' \
168+
| xargs -r -I{} kubectl -n "$ns" delete pvc {}
169+
```
170+
171+
Leave the HelmRelease suspended and continue with Step 2 (skip its suspend line; the renamed StatefulSets it scales down are the ones holding the data, exactly as in a plain `S` tenant). Do NOT scale down or delete anything named `*-system-*` before its volumes are re-bound.
172+
130173
## Step 3 — `D-split` tenants: stop the split before upgrading
131174

132175
Do **not** upgrade first and do not delete the duplicate PVCs — they may hold objects written through the split endpoint.

‎packages/extra/seaweedfs/templates/seaweedfs.yaml‎

Lines changed: 51 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -131,36 +131,76 @@
131131
an unrelated `data1-*-volume-*` claim. StatefulSets do carry the labels, so they
132132
are matched on those, which also covers a tenant whose PVCs are not provisioned
133133
yet and one whose name the chart truncated past the chart name; the renamed
134-
volume StatefulSet is also where the D-split liveness signal is read. */}}
135-
{{- $legacyData := false }}
136-
{{- $systemData := false }}
134+
volume StatefulSet is also where the D-split liveness signal is read.
135+
136+
Telling S from D when BOTH PVC generations exist is done by AGE: PVCs are never
137+
recreated in place (StatefulSets are, by the adoption hook, so their timestamps
138+
are useless), so the OLDER volume-PVC generation is where the data was born.
139+
Release-named older → a fresh-1.5.x tenant that an unguarded upgrade already
140+
damaged by creating the empty chart-named set: still class S; reporting it as
141+
D-split would send the operator to quiesce the set that holds ALL the data.
142+
143+
This copy of the guard warns on the SeaweedFS application itself, where the
144+
operator looks first. The ENFORCING copy lives in system/seaweedfs
145+
(templates/naming-guard.yaml): the <name>-system HelmRelease pulls its chart
146+
from a platform-managed ExternalArtifact, so a platform upgrade re-renders
147+
THAT chart directly without ever re-rendering this one. Keep the two
148+
classifications in sync.
149+
150+
Fail-closed: lookup errors abort the render, but a client-side render returns
151+
nothing — the release namespace is used as a canary, and an upgrade that
152+
cannot see the cluster refuses instead of misclassifying. */}}
153+
{{- $canary := lookup "v1" "Namespace" "" .Release.Namespace }}
154+
{{- if and (not $canary) .Release.IsUpgrade }}
155+
{{- fail (printf "SeaweedFS naming-migration guard for %s in namespace %s: cannot see the cluster (lookup returned nothing for the release namespace itself). This render decides whether workloads are renamed away from live data, so refusing to upgrade blind. Renders applied by helm-controller always see the cluster; if templating by hand, render server-side." .Release.Name .Release.Namespace) }}
156+
{{- end }}
157+
{{- if $canary }}
158+
{{- $legacyPVC := false }}
159+
{{- $systemPVC := false }}
160+
{{- $legacyOldest := "" }}
161+
{{- $systemOldest := "" }}
162+
{{- $legacySTS := false }}
163+
{{- $systemSTS := false }}
137164
{{- $systemRunning := false }}
138165
{{- range (lookup "v1" "PersistentVolumeClaim" .Release.Namespace "").items | default list }}
139166
{{- if and (hasPrefix "data1-" .metadata.name) (contains "-volume" .metadata.name) (contains "seaweedfs" .metadata.name) }}
167+
{{- $ts := dig "metadata" "creationTimestamp" "" . }}
140168
{{- if hasPrefix "data1-seaweedfs-volume" .metadata.name }}
141-
{{- $legacyData = true }}
169+
{{- $legacyPVC = true }}
170+
{{- if and $ts (or (not $legacyOldest) (lt $ts $legacyOldest)) }}
171+
{{- $legacyOldest = $ts }}
172+
{{- end }}
142173
{{- else }}
143-
{{- $systemData = true }}
174+
{{- $systemPVC = true }}
175+
{{- if and $ts (or (not $systemOldest) (lt $ts $systemOldest)) }}
176+
{{- $systemOldest = $ts }}
177+
{{- end }}
144178
{{- end }}
145179
{{- end }}
146180
{{- end }}
147181
{{- range (lookup "apps/v1" "StatefulSet" .Release.Namespace "").items | default list }}
148182
{{- if and (eq (dig "app.kubernetes.io/name" "" (.metadata.labels | default dict)) "seaweedfs") (contains "volume" .metadata.name) }}
149183
{{- if hasPrefix "seaweedfs-volume" .metadata.name }}
150-
{{- $legacyData = true }}
184+
{{- $legacySTS = true }}
151185
{{- else }}
152-
{{- $systemData = true }}
186+
{{- $systemSTS = true }}
153187
{{- if gt (int (dig "status" "readyReplicas" 0 .)) 0 }}
154188
{{- $systemRunning = true }}
155189
{{- end }}
156190
{{- end }}
157191
{{- end }}
158192
{{- end }}
159-
{{- if and $legacyData $systemData $systemRunning }}
160-
{{- fail (printf "SeaweedFS %s in namespace %s is running two live SeaweedFS sets (D-split): the renamed set has ready volume servers alongside the adopted set, and both write to the shared seaweedfs-db metadata. Rendering would adopt the chart-named set and strand the objects the renamed set wrote. Stop the split and reconcile the two sets before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 3)." .Release.Name .Release.Namespace) }}
193+
{{- if or $systemPVC $systemSTS }}
194+
{{- if and $systemPVC (not $legacyPVC) }}
195+
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on volumes named after the Helm release (this tenant was installed on Cozystack 1.5.x). Workloads are named seaweedfs-*, so rendering would rename them away from that data and bring up an empty cluster. Re-bind the existing PVs onto the data1-seaweedfs-volume-* PVC names before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 2)." .Release.Name .Release.Namespace) }}
196+
{{- else if and $systemPVC $legacyPVC $systemOldest $legacyOldest (lt $systemOldest $legacyOldest) }}
197+
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on the release-named volume PVCs (created %s, OLDER than the chart-named data1-seaweedfs-volume-* claims created %s) — a fresh 1.5.x install that a previous unguarded upgrade already damaged by creating the empty chart-named set. Classify as S, never as D-split — do NOT quiesce the release-named set, it holds all the data. Remove the empty chart-named set, then re-bind the PVs — see docs/operations/seaweedfs-431-rename-recovery.md (Step 2a, then Step 2)." .Release.Name .Release.Namespace $systemOldest $legacyOldest) }}
198+
{{- else if and (not $systemPVC) (not $legacyPVC) (not $legacySTS) }}
199+
{{- fail (printf "SeaweedFS %s in namespace %s runs release-named workloads (fresh 1.5.x install; volume PVCs not matchable by name — likely a long instance name the chart truncated). Rendering the chart-based names would rename the workloads away from their data. Re-bind the volumes first — see docs/operations/seaweedfs-431-rename-recovery.md (Step 2)." .Release.Name .Release.Namespace) }}
200+
{{- else if $systemRunning }}
201+
{{- fail (printf "SeaweedFS %s in namespace %s is running two live SeaweedFS sets (D-split): the renamed set has ready volume servers alongside the adopted set, and both write to the shared seaweedfs-db metadata. Rendering would adopt the chart-named set and strand the objects the renamed set wrote. Stop the split and reconcile the two sets before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 3)." .Release.Name .Release.Namespace) }}
202+
{{- end }}
161203
{{- end }}
162-
{{- if and $systemData (not $legacyData) }}
163-
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on volumes named after the Helm release (this tenant was installed on Cozystack 1.5.x). Workloads are named seaweedfs-*, so rendering would rename them away from that data and bring up an empty cluster. Re-bind the existing PVs onto the data1-seaweedfs-volume-* PVC names before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md" .Release.Name .Release.Namespace) }}
164204
{{- end }}
165205
apiVersion: helm.toolkit.fluxcd.io/v2
166206
kind: HelmRelease

0 commit comments

Comments
 (0)