Skip to content

Commit 393592a

Browse files
lexfreilexfrei
committed
test(e2e): lower the seaweedfs volume size limit on a live instance
Every existing case installs SeaweedFS at one limit and stays there, so nothing walks the transition an operator makes when lowering it: each volume at or above the new value leaves the writable set and the bucket writes again only after a grow. The failure is a PUT that times out while every release stays Ready, so the suite ends on a write into a bucket filled between the two limits. The suite runs its own instance, created directly so the limit can move without racing the tenant module, and asserts on the live master that the value set on the resource arrives with the growth counts intact. seaweedfs changes now select the suite, which the selector's mapping and its tests are updated for. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 0c8a266 commit 393592a

6 files changed

Lines changed: 386 additions & 14 deletions

File tree

‎hack/e2e-chainsaw/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ hack/e2e-chainsaw/
1818
└── ...
1919
```
2020

21-
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`.
21+
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`.
2222

2323
## What Chainsaw buys over the BATS suite
2424

@@ -54,7 +54,7 @@ Declarative suites assert on `status.conditions` and concrete fields. Inherently
5454

5555
## Running locally
5656

57-
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` additionally needs `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`harbor` need `jq`.
57+
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`harbor` need `jq`.
5858

5959
```bash
6060
# install chainsaw

‎hack/e2e-chainsaw/seaweedfs/bucket.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Raw COSI objects rather than a Bucket application: a Bucket goes to the
2+
# instance its tenant names, which for the test namespace is tenant-root's.
3+
# The class names are the instance's namespace, set by the app chart's
4+
# cosi.bucketClassName.
5+
apiVersion: objectstorage.k8s.io/v1alpha1
6+
kind: BucketClaim
7+
metadata:
8+
name: seaweedfs-resize
9+
spec:
10+
bucketClassName: tenant-test
11+
protocols:
12+
- s3
13+
---
14+
apiVersion: objectstorage.k8s.io/v1alpha1
15+
kind: BucketAccess
16+
metadata:
17+
name: seaweedfs-resize
18+
spec:
19+
bucketAccessClassName: tenant-test
20+
bucketClaimName: seaweedfs-resize
21+
credentialsSecretName: seaweedfs-resize
22+
protocol: s3

‎hack/e2e-chainsaw/seaweedfs/chainsaw-test.yaml‎

Lines changed: 320 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,320 @@
1+
# Lowering master.volumeSizeLimitMB on a live instance. Every volume at or
2+
# above the new limit leaves the master's writable set, so a bucket whose
3+
# volumes have all crossed it takes its next write only after a grow. Readiness
4+
# stays green through all of it; the failure is a PUT that times out while the
5+
# disk still has room, so the test ends on a write.
6+
#
7+
# 200 -> 20 stands in for the shipped 1000 -> 400: what matters is that a
8+
# bucket can be filled into the band between them, and at these numbers that
9+
# costs about a hundred megabytes rather than gigabytes.
10+
#
11+
# The same run checks the hand-off between the two SeaweedFS charts on the live
12+
# merge: the value set on the SeaweedFS resource must start the master, before
13+
# and after the change, with the growth counts the data-plane chart carries
14+
# still in place. hack/seaweedfs-chart-seam.bats checks the same hand-off by
15+
# rendering, on every pull request.
16+
apiVersion: chainsaw.kyverno.io/v1alpha1
17+
kind: Test
18+
metadata:
19+
name: seaweedfs-resize
20+
spec:
21+
# Uninstalling the instance runs the chart's post-delete Job, which removes
22+
# the volume claims and the CNPG cluster behind the filer, so the delete
23+
# waits on that Job as well as on the workloads.
24+
timeouts:
25+
cleanup: 10m
26+
catch:
27+
- events: {}
28+
- describe:
29+
apiVersion: helm.toolkit.fluxcd.io/v2
30+
kind: HelmRelease
31+
name: seaweedfs
32+
- describe:
33+
apiVersion: helm.toolkit.fluxcd.io/v2
34+
kind: HelmRelease
35+
name: seaweedfs-system
36+
- describe:
37+
apiVersion: apps/v1
38+
kind: StatefulSet
39+
name: seaweedfs-master
40+
- podLogs:
41+
selector: app.kubernetes.io/component=master
42+
tail: 200
43+
- description: master volume map
44+
script:
45+
timeout: 2m
46+
content: |
47+
set -u
48+
echo "» master /vol/status (per server: Id, Collection, Size, ReadOnly):"
49+
timeout 60 kubectl -n "$NAMESPACE" exec statefulset/seaweedfs-master -c seaweedfs -- \
50+
curl -s --max-time 20 http://localhost:9333/vol/status \
51+
|| echo "» /vol/status read failed"
52+
echo
53+
exit 0
54+
steps:
55+
- name: create-instance
56+
try:
57+
- apply:
58+
file: seaweedfs.yaml
59+
60+
- name: wait-instance-ready
61+
try:
62+
# 11m: the generated release carries the 10m install timeout every
63+
# ApplicationDefinition gets unless it overrides it, and seaweedfs-rd does
64+
# not; the extra minute is the start margin. The parent's install waits on
65+
# the seaweedfs-system and seaweedfs-db releases it renders.
66+
- assert:
67+
timeout: 11m
68+
resource:
69+
apiVersion: helm.toolkit.fluxcd.io/v2
70+
kind: HelmRelease
71+
metadata:
72+
name: seaweedfs
73+
status:
74+
(conditions[?type == 'Ready']):
75+
- status: "True"
76+
- assert:
77+
resource:
78+
apiVersion: apps/v1
79+
kind: StatefulSet
80+
metadata:
81+
name: seaweedfs-master
82+
spec:
83+
template:
84+
spec:
85+
(containers[?name == 'seaweedfs']):
86+
- (contains(command[2], '-volumeSizeLimitMB=200 ')): true
87+
(env[?starts_with(name, 'WEED_MASTER_VOLUME_GROWTH_')]):
88+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_1
89+
value: "7"
90+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_2
91+
value: "3"
92+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_3
93+
value: "3"
94+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_OTHER
95+
value: "1"
96+
97+
- name: create-bucket
98+
try:
99+
- apply:
100+
file: bucket.yaml
101+
- assert:
102+
timeout: 2m
103+
resource:
104+
apiVersion: objectstorage.k8s.io/v1alpha1
105+
kind: BucketClaim
106+
metadata:
107+
name: seaweedfs-resize
108+
status:
109+
bucketReady: true
110+
- assert:
111+
resource:
112+
apiVersion: objectstorage.k8s.io/v1alpha1
113+
kind: BucketAccess
114+
metadata:
115+
name: seaweedfs-resize
116+
status:
117+
accessGranted: true
118+
119+
# Write until every volume of the bucket's collection sits between the two
120+
# limits. Each write goes to a volume picked at random, so a fixed amount
121+
# leaves one short volume often enough, and that one volume keeps the bucket
122+
# writable after the change and nothing is tested. Below 180 MiB, rather than
123+
# 200, keeps the master from growing more volumes at the old limit, which it
124+
# does once no writable volume is under nine tenths of it.
125+
- name: fill-bucket-between-the-limits
126+
try:
127+
- script:
128+
timeout: 5m
129+
content: |
130+
set -eu
131+
# The op timeout kills only this shell, and an mc it leaves behind
132+
# holds the op open past it (#4268). Every call below carries its own
133+
# cap, and no write starts after this deadline: the last one, a size
134+
# read and the pause after it still end inside the 5m.
135+
deadline=$(( $(date +%s) + 190 ))
136+
workdir=$(mktemp -d)
137+
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
138+
139+
k() { timeout 30 kubectl "$@"; }
140+
k -n "$NAMESPACE" get secret seaweedfs-resize -o jsonpath='{.data.BucketInfo}' > "$workdir/info.b64"
141+
base64 -d "$workdir/info.b64" > "$workdir/info.json"
142+
ACCESS_KEY=$(jq -r '.spec.secretS3.accessKeyID // ""' "$workdir/info.json")
143+
SECRET_KEY=$(jq -r '.spec.secretS3.accessSecretKey // ""' "$workdir/info.json")
144+
BUCKET=$(jq -r '.spec.bucketName // ""' "$workdir/info.json")
145+
if [ -z "$ACCESS_KEY" ] || [ -z "$SECRET_KEY" ] || [ -z "$BUCKET" ]; then
146+
echo "premise broken: BucketInfo lacks the access key, secret key or bucket name" >&2
147+
exit 1
148+
fi
149+
150+
# Volume sizes as the master last heard them, one line per replica:
151+
# "<id> <bytes>". The filer stores a bucket in the collection named
152+
# after it.
153+
sizes() {
154+
k -n "$NAMESPACE" exec statefulset/seaweedfs-master -c seaweedfs -- \
155+
curl -sf --max-time 20 http://localhost:9333/vol/status > "$workdir/vol.json"
156+
jq -r --arg c "$BUCKET" \
157+
'.Volumes.DataCenters[][][] | .[]? | select(.Collection == $c) | "\(.Id) \(.Size)"' \
158+
"$workdir/vol.json"
159+
}
160+
161+
kubectl port-forward service/seaweedfs-s3 -n "$NAMESPACE" 8334:8333 >/dev/null 2>&1 &
162+
pf_pid=$!
163+
timeout 30 sh -ec 'until nc -z localhost 8334; do sleep 1; done'
164+
timeout 15 mc alias set swfs https://localhost:8334 "$ACCESS_KEY" "$SECRET_KEY" --insecure
165+
166+
# Incompressible, so the bytes written are the bytes stored.
167+
head -c 4194304 /dev/urandom > "$workdir/chunk.bin"
168+
low=$((24 * 1048576))
169+
high=$((180 * 1048576))
170+
n=0
171+
while :; do
172+
for _ in 1 2 3 4 5; do
173+
if [ "$(date +%s)" -ge "$deadline" ]; then
174+
echo "premise broken: out of time after $n writes with a volume still under 24 MiB" >&2
175+
exit 1
176+
fi
177+
n=$((n + 1))
178+
# 45s: the first write into a new bucket waits for its grow.
179+
timeout 45 mc cp --insecure "$workdir/chunk.bin" "swfs/$BUCKET/fill-$n.bin" >/dev/null
180+
done
181+
# The master learns sizes from heartbeats, a few seconds apart.
182+
# TODO(e2e-replace-fixed-timeouts): nothing reports a heartbeat as
183+
# received, so the read waits out one interval.
184+
sleep 6
185+
sizes > "$workdir/sizes"
186+
echo "» after $n writes of 4 MiB:"
187+
cat "$workdir/sizes"
188+
if [ ! -s "$workdir/sizes" ]; then
189+
echo "premise broken: collection $BUCKET has no volume after $n writes; collections present:" >&2
190+
jq -r '[.Volumes.DataCenters[][][] | .[]? | .Collection] | unique | .[]' "$workdir/vol.json" >&2
191+
exit 1
192+
fi
193+
if awk -v h="$high" '$2 >= h { found = 1 } END { exit !found }' "$workdir/sizes"; then
194+
echo "premise broken: a volume reached 180 MiB before every volume passed 24 MiB" >&2
195+
exit 1
196+
fi
197+
if awk -v l="$low" '$2 < l { found = 1 } END { exit found }' "$workdir/sizes"; then
198+
echo "» every volume of $BUCKET is between the limits"
199+
exit 0
200+
fi
201+
done
202+
check:
203+
($error == null): true
204+
205+
- name: lower-the-limit
206+
try:
207+
- patch:
208+
resource:
209+
apiVersion: apps.cozystack.io/v1alpha1
210+
kind: SeaweedFS
211+
metadata:
212+
name: seaweedfs
213+
spec:
214+
master:
215+
volumeSizeLimitMB: 20
216+
# The change reaches the master only through two Helm upgrades, the
217+
# parent's and seaweedfs-system's, and then a rollout of the master.
218+
- assert:
219+
timeout: 6m
220+
resource:
221+
apiVersion: apps/v1
222+
kind: StatefulSet
223+
metadata:
224+
name: seaweedfs-master
225+
spec:
226+
template:
227+
spec:
228+
(containers[?name == 'seaweedfs']):
229+
- (contains(command[2], '-volumeSizeLimitMB=20 ')): true
230+
(env[?starts_with(name, 'WEED_MASTER_VOLUME_GROWTH_')]):
231+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_1
232+
value: "7"
233+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_2
234+
value: "3"
235+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_3
236+
value: "3"
237+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_OTHER
238+
value: "1"
239+
# Without the generation, the status of the previous template matches
240+
# before the controller has seen the new one.
241+
(status.observedGeneration == metadata.generation): true
242+
status:
243+
readyReplicas: 1
244+
updatedReplicas: 1
245+
(currentRevision == updateRevision): true
246+
# Both upgrades have to have finished, not only the rollout they started,
247+
# or cleanup lands on a release still upgrading. 11m clears the 10m install
248+
# and upgrade timeouts of either release by the start margin.
249+
- assert:
250+
timeout: 11m
251+
resource:
252+
apiVersion: helm.toolkit.fluxcd.io/v2
253+
kind: HelmRelease
254+
metadata:
255+
name: seaweedfs
256+
(status.observedGeneration == metadata.generation): true
257+
status:
258+
(conditions[?type == 'Ready']):
259+
- status: "True"
260+
- assert:
261+
timeout: 11m
262+
resource:
263+
apiVersion: helm.toolkit.fluxcd.io/v2
264+
kind: HelmRelease
265+
metadata:
266+
name: seaweedfs-system
267+
(status.observedGeneration == metadata.generation): true
268+
status:
269+
(conditions[?type == 'Ready']):
270+
- status: "True"
271+
272+
# Every volume of the bucket is now over the limit, so this write lands only
273+
# in a volume grown after the change. A volume under 20 MiB in the
274+
# collection is that grow: none was under 24 MiB before it.
275+
- name: write-after-lowering
276+
try:
277+
- script:
278+
timeout: 5m
279+
content: |
280+
set -eu
281+
workdir=$(mktemp -d)
282+
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
283+
284+
k() { timeout 30 kubectl "$@"; }
285+
k -n "$NAMESPACE" get secret seaweedfs-resize -o jsonpath='{.data.BucketInfo}' > "$workdir/info.b64"
286+
base64 -d "$workdir/info.b64" > "$workdir/info.json"
287+
ACCESS_KEY=$(jq -r '.spec.secretS3.accessKeyID // ""' "$workdir/info.json")
288+
SECRET_KEY=$(jq -r '.spec.secretS3.accessSecretKey // ""' "$workdir/info.json")
289+
BUCKET=$(jq -r '.spec.bucketName // ""' "$workdir/info.json")
290+
if [ -z "$ACCESS_KEY" ] || [ -z "$SECRET_KEY" ] || [ -z "$BUCKET" ]; then
291+
echo "premise broken: BucketInfo lacks the access key, secret key or bucket name" >&2
292+
exit 1
293+
fi
294+
295+
kubectl port-forward service/seaweedfs-s3 -n "$NAMESPACE" 8334:8333 >/dev/null 2>&1 &
296+
pf_pid=$!
297+
timeout 30 sh -ec 'until nc -z localhost 8334; do sleep 1; done'
298+
timeout 15 mc alias set swfs https://localhost:8334 "$ACCESS_KEY" "$SECRET_KEY" --insecure
299+
300+
# 90s: the master was just replaced, so the volume servers have to
301+
# register with it before the grow this write needs can be placed.
302+
echo "after lowering" > "$workdir/probe.txt"
303+
timeout 90 mc cp --insecure "$workdir/probe.txt" "swfs/$BUCKET/probe.txt"
304+
timeout 15 mc cat --insecure "swfs/$BUCKET/probe.txt" | grep -qx "after lowering"
305+
306+
# TODO(e2e-replace-fixed-timeouts): the new volume reaches /vol/status
307+
# with the next heartbeat, and nothing reports that.
308+
sleep 6
309+
k -n "$NAMESPACE" exec statefulset/seaweedfs-master -c seaweedfs -- \
310+
curl -sf --max-time 20 http://localhost:9333/vol/status > "$workdir/vol.json"
311+
jq -r --arg c "$BUCKET" \
312+
'.Volumes.DataCenters[][][] | .[]? | select(.Collection == $c) | "\(.Id) \(.Size)"' \
313+
"$workdir/vol.json" > "$workdir/sizes"
314+
cat "$workdir/sizes"
315+
if ! awk -v l=$((20 * 1048576)) '$2 < l { found = 1 } END { exit !found }' "$workdir/sizes"; then
316+
echo "FAIL: the write landed but $BUCKET grew no volume, so the master is not applying the lowered limit" >&2
317+
exit 1
318+
fi
319+
check:
320+
($error == null): true

‎hack/e2e-chainsaw/seaweedfs/seaweedfs.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# A second instance, in the test namespace rather than beside tenant-root's:
2+
# the data-plane chart pins its workload names to `seaweedfs`, so two instances
3+
# cannot share a namespace. It is created directly rather than through the
4+
# tenant module, whose HelmRelease takes no values of its own, so the size
5+
# limit can be moved here without racing a parent.
6+
#
7+
# Sized down from the defaults to what the test needs. replicationFactor stays
8+
# at 2 on two volume servers, because that is the shape whose growth count
9+
# (COPY_2) the platform ships.
10+
apiVersion: apps.cozystack.io/v1alpha1
11+
kind: SeaweedFS
12+
metadata:
13+
name: seaweedfs
14+
spec:
15+
replicationFactor: 2
16+
db:
17+
replicas: 1
18+
size: 1Gi
19+
master:
20+
replicas: 1
21+
volumeSizeLimitMB: 200
22+
filer:
23+
replicas: 1
24+
volume:
25+
replicas: 2
26+
size: 2Gi
27+
s3:
28+
replicas: 1

0 commit comments

Comments
 (0)