Skip to content

Commit 052ce2e

Browse files
committed
fix(seaweedfs): refuse to rename a running SeaweedFS away from its data
With the chart-based names pinned back, an upgrade adopts the running workloads and their volumes in place. Before 4.31 the chart ignored the release name, so this holds for an instance running under any name: its data sits on the chart-named data1-seaweedfs-volume-* PVCs either way. Two states cannot be adopted and stop the render instead, each pointing the operator at the matching recovery step: - Fresh on 1.5.x: only the renamed volumes exist, so adopting the chart-based name would rename the workloads away from that data and bring up an empty cluster. Helm cannot move data between PVCs, so the operator is sent to the PV re-bind procedure. - D-split: both sets exist AND the renamed volume servers are live, so they may hold objects written through the split endpoint. Adopting would strand them, so the operator is sent to reconcile the split first. A duplicate that never served (zero ready replicas) is safe to adopt and renders through. The guard mirrors the chart's fullname helper to recognise what 4.31 named this instance (the helper appends the chart name when the release name does not contain it, then truncates), and matches the renamed volumes by shape plus the chart labels on the StatefulSet rather than reconstructing the exact name. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 7bcd8e0 commit 052ce2e

3 files changed

Lines changed: 553 additions & 0 deletions

File tree

‎packages/extra/seaweedfs/templates/seaweedfs.yaml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,70 @@
9898
{{- $solver := (index .Values._cluster "solver") | default "http01" }}
9999
{{- $clusterIssuer := (index .Values._cluster "issuer-name") | default "letsencrypt-prod" }}
100100
{{- $wildcardSecret := (index .Values._cluster "wildcard-secret-name") | default "" }}
101+
{{- /* Refuse to rename a running SeaweedFS away from its data.
102+
103+
Before 4.31 the chart named workloads after the CHART, ignoring the release
104+
name, so every instance ran as `seaweedfs-*` with its data on
105+
`data1-seaweedfs-volume-*`. 4.31 names them after the release instead, so the
106+
upgrade could not rename in place (StatefulSet names are immutable) and Helm
107+
stood up a second, empty set while the data stayed put. system/seaweedfs pins
108+
the chart-based name back, so those workloads and volumes are adopted in place.
109+
110+
Two states cannot be adopted and stop the render instead:
111+
112+
- FRESH on 1.5.x: only the renamed volumes exist, so the data was written under
113+
them. Adopting the chart-based name would rename the workloads AWAY from that
114+
data and bring up an empty cluster beside it. Helm cannot move data between
115+
PVCs, so the operator is sent to the PV re-bind runbook (Step 2).
116+
- D-split: both sets exist AND the renamed volume servers are live, so they may
117+
hold objects written through the split endpoint. Adopting the chart-based set
118+
would strand those. The operator is sent to reconcile the split first (Step 3).
119+
A duplicate that never served (D-wedged: zero ready replicas) is safe to adopt
120+
and renders through.
121+
122+
The renamed volumes are matched by SHAPE rather than by reconstructing the
123+
name: a volume PVC is always `data1-<fullname>-volume[-<pool|zone>]-N`, and
124+
`<fullname>` is `seaweedfs` only for the adopted naming. Reconstructing it
125+
instead would have to replay the chart's fullname helper AND its 56-character
126+
component truncation, and would silently miss a tenant whose name is long or
127+
does not contain the chart name.
128+
129+
PVCs hold the data and outlive any workload, but carry no chart labels, so they
130+
are matched on the name shape and must mention the chart to avoid tripping over
131+
an unrelated `data1-*-volume-*` claim. StatefulSets do carry the labels, so they
132+
are matched on those, which also covers a tenant whose PVCs are not provisioned
133+
yet and one whose name the chart truncated past the chart name; the renamed
134+
volume StatefulSet is also where the D-split liveness signal is read. */}}
135+
{{- $legacyData := false }}
136+
{{- $systemData := false }}
137+
{{- $systemRunning := false }}
138+
{{- range (lookup "v1" "PersistentVolumeClaim" .Release.Namespace "").items | default list }}
139+
{{- if and (hasPrefix "data1-" .metadata.name) (contains "-volume" .metadata.name) (contains "seaweedfs" .metadata.name) }}
140+
{{- if hasPrefix "data1-seaweedfs-volume" .metadata.name }}
141+
{{- $legacyData = true }}
142+
{{- else }}
143+
{{- $systemData = true }}
144+
{{- end }}
145+
{{- end }}
146+
{{- end }}
147+
{{- range (lookup "apps/v1" "StatefulSet" .Release.Namespace "").items | default list }}
148+
{{- if and (eq (dig "app.kubernetes.io/name" "" (.metadata.labels | default dict)) "seaweedfs") (contains "volume" .metadata.name) }}
149+
{{- if hasPrefix "seaweedfs-volume" .metadata.name }}
150+
{{- $legacyData = true }}
151+
{{- else }}
152+
{{- $systemData = true }}
153+
{{- if gt (int (dig "status" "readyReplicas" 0 .)) 0 }}
154+
{{- $systemRunning = true }}
155+
{{- end }}
156+
{{- end }}
157+
{{- end }}
158+
{{- end }}
159+
{{- if and $legacyData $systemData $systemRunning }}
160+
{{- fail (printf "SeaweedFS %s in namespace %s is running two live SeaweedFS sets (D-split): the renamed set has ready volume servers alongside the adopted set, and both write to the shared seaweedfs-db metadata. Rendering would adopt the chart-named set and strand the objects the renamed set wrote. Stop the split and reconcile the two sets before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md (Step 3)." .Release.Name .Release.Namespace) }}
161+
{{- end }}
162+
{{- if and $systemData (not $legacyData) }}
163+
{{- fail (printf "SeaweedFS %s in namespace %s keeps its data on volumes named after the Helm release (this tenant was installed on Cozystack 1.5.x). Workloads are named seaweedfs-*, so rendering would rename them away from that data and bring up an empty cluster. Re-bind the existing PVs onto the data1-seaweedfs-volume-* PVC names before upgrading — see docs/operations/seaweedfs-431-rename-recovery.md" .Release.Name .Release.Namespace) }}
164+
{{- end }}
101165
apiVersion: helm.toolkit.fluxcd.io/v2
102166
kind: HelmRelease
103167
metadata:
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
suite: seaweedfs naming guard for a non-default instance name
2+
3+
# SeaweedFS is a user-creatable kind, so an instance can run under a name other
4+
# than `seaweedfs`. Two things follow, and the guard has to get both right:
5+
# - Before 4.31 the chart ignored the release name, so such an instance ALSO
6+
# stored its data on the chart-named data1-seaweedfs-volume-* PVCs.
7+
# - On 4.31 the fullname helper appends the chart name when the release name
8+
# does not contain it, so a fresh 1.5.x `archive` wrote its data to
9+
# data1-archive-system-seaweedfs-volume-*, NOT data1-archive-system-volume-*.
10+
11+
templates:
12+
- templates/seaweedfs.yaml
13+
14+
release:
15+
name: archive
16+
namespace: tenant-root
17+
18+
set: &values
19+
topology: Simple
20+
replicationFactor: 2
21+
_namespace:
22+
host: example.org
23+
ingress: tenant-root
24+
_cluster:
25+
issuer-name: letsencrypt-prod
26+
solver: http01
27+
28+
tests:
29+
- it: adopts a pre-4.31 instance whose data is on the chart-named PVCs
30+
set: *values
31+
kubernetesProvider:
32+
scheme:
33+
"v1/PersistentVolumeClaim":
34+
gvr:
35+
group: ""
36+
version: "v1"
37+
resource: "persistentvolumeclaims"
38+
namespaced: true
39+
"apps/v1/StatefulSet":
40+
gvr:
41+
group: "apps"
42+
version: "v1"
43+
resource: "statefulsets"
44+
namespaced: true
45+
objects:
46+
- kind: PersistentVolumeClaim
47+
apiVersion: v1
48+
metadata:
49+
name: data1-seaweedfs-volume-0
50+
namespace: tenant-root
51+
- kind: StatefulSet
52+
apiVersion: apps/v1
53+
metadata:
54+
name: unrelated-app
55+
namespace: tenant-root
56+
documentSelector:
57+
path: kind
58+
value: HelmRelease
59+
asserts:
60+
- equal:
61+
path: metadata.name
62+
value: archive-system
63+
64+
- it: refuses to render when this release's data is on its renamed volumes
65+
set: *values
66+
kubernetesProvider:
67+
scheme:
68+
"v1/PersistentVolumeClaim":
69+
gvr:
70+
group: ""
71+
version: "v1"
72+
resource: "persistentvolumeclaims"
73+
namespaced: true
74+
"apps/v1/StatefulSet":
75+
gvr:
76+
group: "apps"
77+
version: "v1"
78+
resource: "statefulsets"
79+
namespaced: true
80+
objects:
81+
- kind: PersistentVolumeClaim
82+
apiVersion: v1
83+
metadata:
84+
name: data1-archive-system-seaweedfs-volume-0
85+
namespace: tenant-root
86+
- kind: StatefulSet
87+
apiVersion: apps/v1
88+
metadata:
89+
name: unrelated-app
90+
namespace: tenant-root
91+
asserts:
92+
- failedTemplate:
93+
errorPattern: "SeaweedFS archive in namespace tenant-root keeps its data on volumes named after the Helm release"
94+
95+
- it: catches a long instance name whose renamed workloads the chart truncated
96+
set: *values
97+
kubernetesProvider:
98+
scheme:
99+
"v1/PersistentVolumeClaim":
100+
gvr:
101+
group: ""
102+
version: "v1"
103+
resource: "persistentvolumeclaims"
104+
namespaced: true
105+
"apps/v1/StatefulSet":
106+
gvr:
107+
group: "apps"
108+
version: "v1"
109+
resource: "statefulsets"
110+
namespaced: true
111+
objects:
112+
# 4.31 truncates <fullname> to 56 chars before appending -volume, so a long
113+
# name loses the chart name from its tail and the PVC name alone no longer
114+
# identifies it. The StatefulSet still carries the chart labels, so the
115+
# guard sees it there.
116+
- kind: StatefulSet
117+
apiVersion: apps/v1
118+
metadata:
119+
name: archive-system-a-very-long-instance-name-that-gets-trunca-volume
120+
namespace: tenant-root
121+
labels:
122+
app.kubernetes.io/name: seaweedfs
123+
- kind: PersistentVolumeClaim
124+
apiVersion: v1
125+
metadata:
126+
name: unrelated-app-data-0
127+
namespace: tenant-root
128+
asserts:
129+
- failedTemplate:
130+
errorPattern: "keeps its data on volumes named after the Helm release"
131+
132+
- it: ignores an unrelated data1-*-volume-* PVC belonging to another app
133+
set: *values
134+
kubernetesProvider:
135+
scheme:
136+
"v1/PersistentVolumeClaim":
137+
gvr:
138+
group: ""
139+
version: "v1"
140+
resource: "persistentvolumeclaims"
141+
namespaced: true
142+
"apps/v1/StatefulSet":
143+
gvr:
144+
group: "apps"
145+
version: "v1"
146+
resource: "statefulsets"
147+
namespaced: true
148+
objects:
149+
- kind: PersistentVolumeClaim
150+
apiVersion: v1
151+
metadata:
152+
name: data1-clickhouse-volume-0
153+
namespace: tenant-root
154+
- kind: StatefulSet
155+
apiVersion: apps/v1
156+
metadata:
157+
name: clickhouse-volume
158+
namespace: tenant-root
159+
labels:
160+
app.kubernetes.io/name: clickhouse
161+
documentSelector:
162+
path: kind
163+
value: HelmRelease
164+
asserts:
165+
- equal:
166+
path: metadata.name
167+
value: archive-system

0 commit comments

Comments
 (0)