Skip to content

Commit 9c9a8c2

Browse files
authored
test(seaweedfs): lower the volume size limit on a live instance and test the chart hand-off (#4707)
## What this PR does This adds an e2e case that lowers the SeaweedFS volume size limit on a live instance and checks that the bucket still takes writes. It also adds a unit test for the hand-off between the SeaweedFS app chart and the data-plane chart. The new `seaweedfs` suite creates its own instance in `tenant-test`, directly through the API, so the limit can change without racing the tenant module. It installs at 200 MB and writes into one bucket until every volume of the bucket sits between 20 and 200 MB. Then it patches the resource to 20 and waits for both releases and the master rollout. After that a write to the same bucket has to land, and the bucket has to have a new volume. If the master never picks up the new limit, the write goes to an old volume and the second check fails. 200 and 20 stand in for the shipped 1000 and 400, because filling a bucket into that band at real sizes would cost gigabytes in the sandbox. The suite also checks the live master StatefulSet. The limit from the resource has to be in the master command before and after the change, and all four `WEED_MASTER_VOLUME_GROWTH_COPY_*` counts have to stay. The same hand-off is checked on every PR by `hack/seaweedfs-chart-seam.bats`. It renders the app chart, takes `spec.values` from the `seaweedfs-system` HelmRelease and renders the data-plane chart with them. The expected default comes from the app chart's values, so this test does not pin 400 a second time. I checked in a scratch copy that it goes red when the app chart's `volumeSizeLimitMB` key or its `master:` block is renamed. A new suite directory broke two cases in `hack/select-e2e_test.bats`, so `seaweedfs-application` in `hack/select-e2e.sh` now maps to the new suite too. That is one line plus its comment. About the suite order question in #4302: this suite runs on its own instance, so it takes nothing from tenant-root's capacity and does not move the margin of `bucket-1-capacity`. Left for later: - A generic helper for the two-chart shape (render a chart, take `spec.values` from a HelmRelease, render the target). This PR covers only the SeaweedFS pair. - The `status.history` remediation guard on the upgrade the suite triggers. A failed upgrade that is retried and then succeeds would pass here without notice. The suite has not run on a cluster yet. Locally `chainsaw lint` passes. I also ran the StatefulSet and HelmRelease asserts with `chainsaw assert` against rendered objects. They pass on the expected state and fail on a wrong limit, a missing `COPY_2`, a rollout in progress and a stale generation, both on the StatefulSet and on the HelmRelease. Closes #4302 Closes #4285 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): add a seaweedfs suite that lowers the volume size limit on a live instance and checks that the bucket still takes writes, and a unit test for the hand-off between the SeaweedFS app and data-plane charts. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added end-to-end coverage for SeaweedFS bucket resizing, including verifying that writes succeed after the volume-size limit changes. - Added SeaweedFS to the available and automatically selected end-to-end test suites. - **Documentation** - Updated local test prerequisites to note the required tools for SeaweedFS tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 10a6cf2 + 393592a commit 9c9a8c2

7 files changed

Lines changed: 464 additions & 14 deletions

File tree

‎hack/e2e-chainsaw/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ hack/e2e-chainsaw/
1818
└── ...
1919
```
2020

21-
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`.
21+
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`.
2222

2323
## What Chainsaw buys over the BATS suite
2424

@@ -54,7 +54,7 @@ Declarative suites assert on `status.conditions` and concrete fields. Inherently
5454

5555
## Running locally
5656

57-
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` additionally needs `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`harbor` need `jq`.
57+
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`harbor` need `jq`.
5858

5959
```bash
6060
# install chainsaw
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Raw COSI objects rather than a Bucket application: a Bucket goes to the
2+
# instance its tenant names, which for the test namespace is tenant-root's.
3+
# The class names are the instance's namespace, set by the app chart's
4+
# cosi.bucketClassName.
5+
apiVersion: objectstorage.k8s.io/v1alpha1
6+
kind: BucketClaim
7+
metadata:
8+
name: seaweedfs-resize
9+
spec:
10+
bucketClassName: tenant-test
11+
protocols:
12+
- s3
13+
---
14+
apiVersion: objectstorage.k8s.io/v1alpha1
15+
kind: BucketAccess
16+
metadata:
17+
name: seaweedfs-resize
18+
spec:
19+
bucketAccessClassName: tenant-test
20+
bucketClaimName: seaweedfs-resize
21+
credentialsSecretName: seaweedfs-resize
22+
protocol: s3
Lines changed: 320 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,320 @@
1+
# Lowering master.volumeSizeLimitMB on a live instance. Every volume at or
2+
# above the new limit leaves the master's writable set, so a bucket whose
3+
# volumes have all crossed it takes its next write only after a grow. Readiness
4+
# stays green through all of it; the failure is a PUT that times out while the
5+
# disk still has room, so the test ends on a write.
6+
#
7+
# 200 -> 20 stands in for the shipped 1000 -> 400: what matters is that a
8+
# bucket can be filled into the band between them, and at these numbers that
9+
# costs about a hundred megabytes rather than gigabytes.
10+
#
11+
# The same run checks the hand-off between the two SeaweedFS charts on the live
12+
# merge: the value set on the SeaweedFS resource must start the master, before
13+
# and after the change, with the growth counts the data-plane chart carries
14+
# still in place. hack/seaweedfs-chart-seam.bats checks the same hand-off by
15+
# rendering, on every pull request.
16+
apiVersion: chainsaw.kyverno.io/v1alpha1
17+
kind: Test
18+
metadata:
19+
name: seaweedfs-resize
20+
spec:
21+
# Uninstalling the instance runs the chart's post-delete Job, which removes
22+
# the volume claims and the CNPG cluster behind the filer, so the delete
23+
# waits on that Job as well as on the workloads.
24+
timeouts:
25+
cleanup: 10m
26+
catch:
27+
- events: {}
28+
- describe:
29+
apiVersion: helm.toolkit.fluxcd.io/v2
30+
kind: HelmRelease
31+
name: seaweedfs
32+
- describe:
33+
apiVersion: helm.toolkit.fluxcd.io/v2
34+
kind: HelmRelease
35+
name: seaweedfs-system
36+
- describe:
37+
apiVersion: apps/v1
38+
kind: StatefulSet
39+
name: seaweedfs-master
40+
- podLogs:
41+
selector: app.kubernetes.io/component=master
42+
tail: 200
43+
- description: master volume map
44+
script:
45+
timeout: 2m
46+
content: |
47+
set -u
48+
echo "» master /vol/status (per server: Id, Collection, Size, ReadOnly):"
49+
timeout 60 kubectl -n "$NAMESPACE" exec statefulset/seaweedfs-master -c seaweedfs -- \
50+
curl -s --max-time 20 http://localhost:9333/vol/status \
51+
|| echo "» /vol/status read failed"
52+
echo
53+
exit 0
54+
steps:
55+
- name: create-instance
56+
try:
57+
- apply:
58+
file: seaweedfs.yaml
59+
60+
- name: wait-instance-ready
61+
try:
62+
# 11m: the generated release carries the 10m install timeout every
63+
# ApplicationDefinition gets unless it overrides it, and seaweedfs-rd does
64+
# not; the extra minute is the start margin. The parent's install waits on
65+
# the seaweedfs-system and seaweedfs-db releases it renders.
66+
- assert:
67+
timeout: 11m
68+
resource:
69+
apiVersion: helm.toolkit.fluxcd.io/v2
70+
kind: HelmRelease
71+
metadata:
72+
name: seaweedfs
73+
status:
74+
(conditions[?type == 'Ready']):
75+
- status: "True"
76+
- assert:
77+
resource:
78+
apiVersion: apps/v1
79+
kind: StatefulSet
80+
metadata:
81+
name: seaweedfs-master
82+
spec:
83+
template:
84+
spec:
85+
(containers[?name == 'seaweedfs']):
86+
- (contains(command[2], '-volumeSizeLimitMB=200 ')): true
87+
(env[?starts_with(name, 'WEED_MASTER_VOLUME_GROWTH_')]):
88+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_1
89+
value: "7"
90+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_2
91+
value: "3"
92+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_3
93+
value: "3"
94+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_OTHER
95+
value: "1"
96+
97+
- name: create-bucket
98+
try:
99+
- apply:
100+
file: bucket.yaml
101+
- assert:
102+
timeout: 2m
103+
resource:
104+
apiVersion: objectstorage.k8s.io/v1alpha1
105+
kind: BucketClaim
106+
metadata:
107+
name: seaweedfs-resize
108+
status:
109+
bucketReady: true
110+
- assert:
111+
resource:
112+
apiVersion: objectstorage.k8s.io/v1alpha1
113+
kind: BucketAccess
114+
metadata:
115+
name: seaweedfs-resize
116+
status:
117+
accessGranted: true
118+
119+
# Write until every volume of the bucket's collection sits between the two
120+
# limits. Each write goes to a volume picked at random, so a fixed amount
121+
# leaves one short volume often enough, and that one volume keeps the bucket
122+
# writable after the change and nothing is tested. Below 180 MiB, rather than
123+
# 200, keeps the master from growing more volumes at the old limit, which it
124+
# does once no writable volume is under nine tenths of it.
125+
- name: fill-bucket-between-the-limits
126+
try:
127+
- script:
128+
timeout: 5m
129+
content: |
130+
set -eu
131+
# The op timeout kills only this shell, and an mc it leaves behind
132+
# holds the op open past it (#4268). Every call below carries its own
133+
# cap, and no write starts after this deadline: the last one, a size
134+
# read and the pause after it still end inside the 5m.
135+
deadline=$(( $(date +%s) + 190 ))
136+
workdir=$(mktemp -d)
137+
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
138+
139+
k() { timeout 30 kubectl "$@"; }
140+
k -n "$NAMESPACE" get secret seaweedfs-resize -o jsonpath='{.data.BucketInfo}' > "$workdir/info.b64"
141+
base64 -d "$workdir/info.b64" > "$workdir/info.json"
142+
ACCESS_KEY=$(jq -r '.spec.secretS3.accessKeyID // ""' "$workdir/info.json")
143+
SECRET_KEY=$(jq -r '.spec.secretS3.accessSecretKey // ""' "$workdir/info.json")
144+
BUCKET=$(jq -r '.spec.bucketName // ""' "$workdir/info.json")
145+
if [ -z "$ACCESS_KEY" ] || [ -z "$SECRET_KEY" ] || [ -z "$BUCKET" ]; then
146+
echo "premise broken: BucketInfo lacks the access key, secret key or bucket name" >&2
147+
exit 1
148+
fi
149+
150+
# Volume sizes as the master last heard them, one line per replica:
151+
# "<id> <bytes>". The filer stores a bucket in the collection named
152+
# after it.
153+
sizes() {
154+
k -n "$NAMESPACE" exec statefulset/seaweedfs-master -c seaweedfs -- \
155+
curl -sf --max-time 20 http://localhost:9333/vol/status > "$workdir/vol.json"
156+
jq -r --arg c "$BUCKET" \
157+
'.Volumes.DataCenters[][][] | .[]? | select(.Collection == $c) | "\(.Id) \(.Size)"' \
158+
"$workdir/vol.json"
159+
}
160+
161+
kubectl port-forward service/seaweedfs-s3 -n "$NAMESPACE" 8334:8333 >/dev/null 2>&1 &
162+
pf_pid=$!
163+
timeout 30 sh -ec 'until nc -z localhost 8334; do sleep 1; done'
164+
timeout 15 mc alias set swfs https://localhost:8334 "$ACCESS_KEY" "$SECRET_KEY" --insecure
165+
166+
# Incompressible, so the bytes written are the bytes stored.
167+
head -c 4194304 /dev/urandom > "$workdir/chunk.bin"
168+
low=$((24 * 1048576))
169+
high=$((180 * 1048576))
170+
n=0
171+
while :; do
172+
for _ in 1 2 3 4 5; do
173+
if [ "$(date +%s)" -ge "$deadline" ]; then
174+
echo "premise broken: out of time after $n writes with a volume still under 24 MiB" >&2
175+
exit 1
176+
fi
177+
n=$((n + 1))
178+
# 45s: the first write into a new bucket waits for its grow.
179+
timeout 45 mc cp --insecure "$workdir/chunk.bin" "swfs/$BUCKET/fill-$n.bin" >/dev/null
180+
done
181+
# The master learns sizes from heartbeats, a few seconds apart.
182+
# TODO(e2e-replace-fixed-timeouts): nothing reports a heartbeat as
183+
# received, so the read waits out one interval.
184+
sleep 6
185+
sizes > "$workdir/sizes"
186+
echo "» after $n writes of 4 MiB:"
187+
cat "$workdir/sizes"
188+
if [ ! -s "$workdir/sizes" ]; then
189+
echo "premise broken: collection $BUCKET has no volume after $n writes; collections present:" >&2
190+
jq -r '[.Volumes.DataCenters[][][] | .[]? | .Collection] | unique | .[]' "$workdir/vol.json" >&2
191+
exit 1
192+
fi
193+
if awk -v h="$high" '$2 >= h { found = 1 } END { exit !found }' "$workdir/sizes"; then
194+
echo "premise broken: a volume reached 180 MiB before every volume passed 24 MiB" >&2
195+
exit 1
196+
fi
197+
if awk -v l="$low" '$2 < l { found = 1 } END { exit found }' "$workdir/sizes"; then
198+
echo "» every volume of $BUCKET is between the limits"
199+
exit 0
200+
fi
201+
done
202+
check:
203+
($error == null): true
204+
205+
- name: lower-the-limit
206+
try:
207+
- patch:
208+
resource:
209+
apiVersion: apps.cozystack.io/v1alpha1
210+
kind: SeaweedFS
211+
metadata:
212+
name: seaweedfs
213+
spec:
214+
master:
215+
volumeSizeLimitMB: 20
216+
# The change reaches the master only through two Helm upgrades, the
217+
# parent's and seaweedfs-system's, and then a rollout of the master.
218+
- assert:
219+
timeout: 6m
220+
resource:
221+
apiVersion: apps/v1
222+
kind: StatefulSet
223+
metadata:
224+
name: seaweedfs-master
225+
spec:
226+
template:
227+
spec:
228+
(containers[?name == 'seaweedfs']):
229+
- (contains(command[2], '-volumeSizeLimitMB=20 ')): true
230+
(env[?starts_with(name, 'WEED_MASTER_VOLUME_GROWTH_')]):
231+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_1
232+
value: "7"
233+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_2
234+
value: "3"
235+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_3
236+
value: "3"
237+
- name: WEED_MASTER_VOLUME_GROWTH_COPY_OTHER
238+
value: "1"
239+
# Without the generation, the status of the previous template matches
240+
# before the controller has seen the new one.
241+
(status.observedGeneration == metadata.generation): true
242+
status:
243+
readyReplicas: 1
244+
updatedReplicas: 1
245+
(currentRevision == updateRevision): true
246+
# Both upgrades have to have finished, not only the rollout they started,
247+
# or cleanup lands on a release still upgrading. 11m clears the 10m install
248+
# and upgrade timeouts of either release by the start margin.
249+
- assert:
250+
timeout: 11m
251+
resource:
252+
apiVersion: helm.toolkit.fluxcd.io/v2
253+
kind: HelmRelease
254+
metadata:
255+
name: seaweedfs
256+
(status.observedGeneration == metadata.generation): true
257+
status:
258+
(conditions[?type == 'Ready']):
259+
- status: "True"
260+
- assert:
261+
timeout: 11m
262+
resource:
263+
apiVersion: helm.toolkit.fluxcd.io/v2
264+
kind: HelmRelease
265+
metadata:
266+
name: seaweedfs-system
267+
(status.observedGeneration == metadata.generation): true
268+
status:
269+
(conditions[?type == 'Ready']):
270+
- status: "True"
271+
272+
# Every volume of the bucket is now over the limit, so this write lands only
273+
# in a volume grown after the change. A volume under 20 MiB in the
274+
# collection is that grow: none was under 24 MiB before it.
275+
- name: write-after-lowering
276+
try:
277+
- script:
278+
timeout: 5m
279+
content: |
280+
set -eu
281+
workdir=$(mktemp -d)
282+
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
283+
284+
k() { timeout 30 kubectl "$@"; }
285+
k -n "$NAMESPACE" get secret seaweedfs-resize -o jsonpath='{.data.BucketInfo}' > "$workdir/info.b64"
286+
base64 -d "$workdir/info.b64" > "$workdir/info.json"
287+
ACCESS_KEY=$(jq -r '.spec.secretS3.accessKeyID // ""' "$workdir/info.json")
288+
SECRET_KEY=$(jq -r '.spec.secretS3.accessSecretKey // ""' "$workdir/info.json")
289+
BUCKET=$(jq -r '.spec.bucketName // ""' "$workdir/info.json")
290+
if [ -z "$ACCESS_KEY" ] || [ -z "$SECRET_KEY" ] || [ -z "$BUCKET" ]; then
291+
echo "premise broken: BucketInfo lacks the access key, secret key or bucket name" >&2
292+
exit 1
293+
fi
294+
295+
kubectl port-forward service/seaweedfs-s3 -n "$NAMESPACE" 8334:8333 >/dev/null 2>&1 &
296+
pf_pid=$!
297+
timeout 30 sh -ec 'until nc -z localhost 8334; do sleep 1; done'
298+
timeout 15 mc alias set swfs https://localhost:8334 "$ACCESS_KEY" "$SECRET_KEY" --insecure
299+
300+
# 90s: the master was just replaced, so the volume servers have to
301+
# register with it before the grow this write needs can be placed.
302+
echo "after lowering" > "$workdir/probe.txt"
303+
timeout 90 mc cp --insecure "$workdir/probe.txt" "swfs/$BUCKET/probe.txt"
304+
timeout 15 mc cat --insecure "swfs/$BUCKET/probe.txt" | grep -qx "after lowering"
305+
306+
# TODO(e2e-replace-fixed-timeouts): the new volume reaches /vol/status
307+
# with the next heartbeat, and nothing reports that.
308+
sleep 6
309+
k -n "$NAMESPACE" exec statefulset/seaweedfs-master -c seaweedfs -- \
310+
curl -sf --max-time 20 http://localhost:9333/vol/status > "$workdir/vol.json"
311+
jq -r --arg c "$BUCKET" \
312+
'.Volumes.DataCenters[][][] | .[]? | select(.Collection == $c) | "\(.Id) \(.Size)"' \
313+
"$workdir/vol.json" > "$workdir/sizes"
314+
cat "$workdir/sizes"
315+
if ! awk -v l=$((20 * 1048576)) '$2 < l { found = 1 } END { exit !found }' "$workdir/sizes"; then
316+
echo "FAIL: the write landed but $BUCKET grew no volume, so the master is not applying the lowered limit" >&2
317+
exit 1
318+
fi
319+
check:
320+
($error == null): true
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# A second instance, in the test namespace rather than beside tenant-root's:
2+
# the data-plane chart pins its workload names to `seaweedfs`, so two instances
3+
# cannot share a namespace. It is created directly rather than through the
4+
# tenant module, whose HelmRelease takes no values of its own, so the size
5+
# limit can be moved here without racing a parent.
6+
#
7+
# Sized down from the defaults to what the test needs. replicationFactor stays
8+
# at 2 on two volume servers, because that is the shape whose growth count
9+
# (COPY_2) the platform ships.
10+
apiVersion: apps.cozystack.io/v1alpha1
11+
kind: SeaweedFS
12+
metadata:
13+
name: seaweedfs
14+
spec:
15+
replicationFactor: 2
16+
db:
17+
replicas: 1
18+
size: 1Gi
19+
master:
20+
replicas: 1
21+
volumeSizeLimitMB: 200
22+
filer:
23+
replicas: 1
24+
volume:
25+
replicas: 2
26+
size: 2Gi
27+
s3:
28+
replicas: 1

0 commit comments

Comments
 (0)