You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8499b5f
Browse filesBrowse the repository at this point in the historyBrowse files
fix(seaweedfs): move the naming-migration guard into the render that actually renames
The class-S guard from #3282 lives in extra/seaweedfs, but that render is not
in the path of a platform upgrade: the <name>-system HelmRelease pulls
system/seaweedfs from a platform-managed ExternalArtifact, so bumping the
platform upgrades the release directly from the new artifact and applies
fullnameOverride: seaweedfs without extra/seaweedfs ever re-rendering. On a
tenant installed fresh on 1.5.x that stood up an empty chart-named set beside
the live release-named data, deleted the renamed s3 Service, and left S3
readable only because the empty set's pods cannot start.
Worse, the damage flips the guard's own classification: creating the
chart-named artifacts makes $legacyData true, so the guard then reports
D-split and sends the operator to runbook Step 3 — which quiesces the renamed
set, i.e. the one holding ALL the data.
- Add the ENFORCING guard to system/seaweedfs (templates/naming-guard.yaml):
the only render guaranteed to sit between a platform upgrade and the
tenant's workloads. It also refuses to render if the fullnameOverride pin
is ever lifted.
- Tell S from D by PVC AGE when both generations exist: PVCs are never
recreated in place (StatefulSets are, by the adoption hook), so the older
volume-PVC generation is where the data was born. Renamed-older ⇒ class S
(damaged variant, new Step 2a), never D-split.
- Fail closed: the release namespace is a lookup canary; an upgrade that
cannot see the cluster refuses instead of classifying blind. Client-side
installs (CI lint/unittest) still render — they never touch live data.
- Mirror the same classification in extra/seaweedfs so the operator-facing
SeaweedFS application reports the correct class.
- Tests now model the ExternalArtifact upgrade path in the chart it actually
runs through (the old ones mocked lookups only in extra/seaweedfs, which is
why this shipped green), including the damaged-S tenant observed live.
- Runbook: add S-damaged audit classification (PVC age), Step 2a cleanup of
the empty chart-named set, and the D-wedged anti-affinity stall workaround
observed on 3-node clusters.
helm-unittest 1.0.3 ignores release.isUpgrade, so the canary refusal is
verified with 'helm template --is-upgrade' instead of a unit test.
Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
Copy file name to clipboardExpand all lines: docs/operations/seaweedfs-431-rename-recovery.md
+47-4Lines changed: 47 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ Before 4.31 the chart named workloads after the chart (`seaweedfs-*`), ignoring
11
11
12
12
The fix pins `fullnameOverride: seaweedfs` in `system/seaweedfs` values, so workloads are always named after the chart, exactly as they were before 4.31. Upgrading past the bump therefore **adopts the running set and its volumes in place**.
13
13
14
-
One class cannot be adopted that way: a tenant installed **fresh on 1.5.x**, whose data was written under the release-based names and lives on `data1-seaweedfs-system-volume-*` PVCs. Pinning the chart name there would rename the workloads *away* from that data. Helm cannot move data between PVCs, so `extra/seaweedfs` refuses to render for such a tenant and points here. Re-bind its volumes (below) before upgrading.
14
+
One class cannot be adopted that way: a tenant installed **fresh on 1.5.x**, whose data was written under the release-based names and lives on `data1-seaweedfs-system-volume-*` PVCs. Pinning the chart name there would rename the workloads *away* from that data. Helm cannot move data between PVCs, so the charts refuse to render for such a tenant and point here. The **enforcing** guard lives in `system/seaweedfs` (`templates/naming-guard.yaml`) — the `<name>-system` HelmRelease pulls that chart straight from a platform-managed ExternalArtifact, so a platform upgrade re-renders it directly and nothing else stands between the upgrade and the tenant's workloads; `extra/seaweedfs` carries a sibling copy so the refusal is also visible on the SeaweedFS application itself. Re-bind the tenant's volumes (below) before upgrading.
15
15
16
16
## Step 1 — Audit the fleet (read-only)
17
17
@@ -30,15 +30,26 @@ nss=$(kubectl get pvc -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "
printf'%-28s %-10s %s\n'"$ns""D-wedged""duplicate set never served; upgrade adopts the legacy set"
@@ -51,7 +62,16 @@ for ns in ${nss}; do
51
62
done
52
63
```
53
64
54
-
Act on the classes in this order: resolve every **D-split** and every **S** tenant first (both need an operator), then upgrade. **L** and **D-wedged** need nothing.
65
+
Act on the classes in this order: resolve every **D-split**, **S** and **S-damaged** tenant first (all need an operator), then upgrade. **L** needs nothing.
66
+
67
+
**D-wedged** normally needs nothing — but on a cluster with no spare nodes (nodes ≤ replicas) the adoption rollout can stall: the wedged duplicate pods are still *scheduled*, carry the same labels as the adopted set (including `app.kubernetes.io/instance`), and their hard pod anti-affinity blocks the adopted set's rolled pods from landing anywhere (observed as `seaweedfs-filer-1`/`seaweedfs-master-2` stuck Pending on `didn't match pod anti-affinity rules`, wedging the `<name>-system` HelmRelease in upgrade/rollback loops). If that happens, scale the duplicate down first — it never served, so this is safe:
68
+
69
+
```sh
70
+
ns=<tenant>
71
+
kubectl -n "$ns" get sts -l app.kubernetes.io/name=seaweedfs -o name | sed 's|statefulset.apps/||' \
## Step 2 — `S` tenants: re-bind the volumes before upgrading
57
77
@@ -127,6 +147,29 @@ Then upgrade. The filer metadata lives in the shared `seaweedfs-db` Postgres and
127
147
128
148
The loop handles pools and zones (their PVC names carry the pool/zone key) and both renamed shapes: `data1-seaweedfs-system-volume-*` for the default instance, and `data1-<name>-system-seaweedfs-volume-*` for an instance running under another name, because 4.31's name helper appends the chart name when the release name does not contain it.
129
149
150
+
## Step 2a — `S-damaged` tenants: remove the empty chart-named set first
151
+
152
+
An `S` tenant that an unguarded 1.6.0 upgrade already reached has an extra problem: the upgrade **created** chart-named workloads (`seaweedfs-master/-filer/-volume`, an s3 Deployment) and **empty**`data1-seaweedfs-volume-*` PVCs beside the live renamed set, and deleted the renamed `<fullname>-s3` Service (only `seaweedfs-s3` remains — its endpoints may still resolve to the renamed set's pods because both sets carry identical labels, which is luck, not design: if the chart-named pods ever become Ready, the Service splits reads across a live set and an empty one).
153
+
154
+
Verify the direction before touching anything — the chart-named PVCs must be the **newer** generation (compare `kubectl get pvc -o custom-columns=NAME:.metadata.name,CREATED:.metadata.creationTimestamp`), matching the `S-damaged` audit row. Then clear the empty chart-named set so Step 2 can re-bind onto those names:
# each is the newer, never-served generation before deleting.
167
+
kubectl -n "$ns" get pvc -o name | sed 's|persistentvolumeclaim/||'| grep -E '^data1-seaweedfs-volume' \
168
+
| xargs -r -I{} kubectl -n "$ns" delete pvc {}
169
+
```
170
+
171
+
Leave the HelmRelease suspended and continue with Step 2 (skip its suspend line; the renamed StatefulSets it scales down are the ones holding the data, exactly as in a plain `S` tenant). Do NOT scale down or delete anything named `*-system-*` before its volumes are re-bound.
172
+
130
173
## Step 3 — `D-split` tenants: stop the split before upgrading
131
174
132
175
Do **not** upgrade first and do not delete the duplicate PVCs — they may hold objects written through the split endpoint.
{{- fail (printf "SeaweedFS naming-migration guard for %s in namespace %s: cannot see the cluster (lookup returned nothing for the release namespace itself). This render decides whether workloads are renamed away from live data, so refusing to upgrade blind. Renders applied by helm-controller always see the cluster; if templating by hand, render server-side." .Release.Name .Release.Namespace) }}
156
+
{{- end }}
157
+
{{- if $canary }}
158
+
{{- $legacyPVC := false }}
159
+
{{- $systemPVC := false }}
160
+
{{- $legacyOldest := "" }}
161
+
{{- $systemOldest := "" }}
162
+
{{- $legacySTS := false }}
163
+
{{- $systemSTS := false }}
137
164
{{- $systemRunning := false }}
138
165
{{- range (lookup "v1" "PersistentVolumeClaim" .Release.Namespace "").items | default list }}
139
166
{{- if and (hasPrefix "data1-" .metadata.name) (contains "-volume" .metadata.name) (contains "seaweedfs" .metadata.name) }}
{{- if and $legacyData $systemData $systemRunning }}
160
-
{{- fail (printf "SeaweedFS %s in namespace %s is running two live SeaweedFS sets (D-split): the renamed set has ready volume servers alongside the adopted set, and both write to the shared seaweedfs-db metadata. Rendering would adopt the chart-named set and strand the objects the renamed set wrote. Stop the split and reconcile the two sets before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 3)." .Release.Name .Release.Namespace) }}
193
+
{{- if or $systemPVC $systemSTS }}
194
+
{{- if and $systemPVC (not $legacyPVC) }}
195
+
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on volumes named after the Helm release (this tenant was installed on Cozystack 1.5.x). Workloads are named seaweedfs-*, so rendering would rename them away from that data and bring up an empty cluster. Re-bind the existing PVs onto the data1-seaweedfs-volume-* PVC names before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 2)." .Release.Name .Release.Namespace) }}
196
+
{{- else if and $systemPVC $legacyPVC $systemOldest $legacyOldest (lt $systemOldest $legacyOldest) }}
197
+
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on the release-named volume PVCs (created %s, OLDER than the chart-named data1-seaweedfs-volume-* claims created %s) — a fresh 1.5.x install that a previous unguarded upgrade already damaged by creating the empty chart-named set. Classify as S, never as D-split — do NOT quiesce the release-named set, it holds all the data. Remove the empty chart-named set, then re-bind the PVs — see docs/operations/seaweedfs-431-rename-recovery.md (Step 2a, then Step 2)." .Release.Name .Release.Namespace $systemOldest $legacyOldest) }}
198
+
{{- else if and (not $systemPVC) (not $legacyPVC) (not $legacySTS) }}
199
+
{{- fail (printf "SeaweedFS %s in namespace %s runs release-named workloads (fresh 1.5.x install; volume PVCs not matchable by name — likely a long instance name the chart truncated). Rendering the chart-based names would rename the workloads away from their data. Re-bind the volumes first — see docs/operations/seaweedfs-431-rename-recovery.md (Step 2)." .Release.Name .Release.Namespace) }}
200
+
{{- else if $systemRunning }}
201
+
{{- fail (printf "SeaweedFS %s in namespace %s is running two live SeaweedFS sets (D-split): the renamed set has ready volume servers alongside the adopted set, and both write to the shared seaweedfs-db metadata. Rendering would adopt the chart-named set and strand the objects the renamed set wrote. Stop the split and reconcile the two sets before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 3)." .Release.Name .Release.Namespace) }}
202
+
{{- end }}
161
203
{{- end }}
162
-
{{- if and $systemData (not $legacyData) }}
163
-
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on volumes named after the Helm release (this tenant was installed on Cozystack 1.5.x). Workloads are named seaweedfs-*, so rendering would rename them away from that data and bring up an empty cluster. Re-bind the existing PVs onto the data1-seaweedfs-volume-* PVC names before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md" .Release.Name .Release.Namespace) }}
0 commit comments