Skip to content

Commit 9399d8f

Browse files
lexfreilexfrei
committed
fix(seaweedfs): raise the bucket ceiling by sizing volumes smaller
A volume server started with -max 0 derives its slot count from free disk, minus the headroom its existing volumes still claim, divided by the master's volume size limit. An unfilled volume reserves its whole limit, so a volume counts against the budget for as long as it stays writable. Replication 001 puts a copy on each of the two servers, so a logical volume costs a slot on both, and the master asks for six logical volumes per collection, keeping whatever fit when the slots run out. On a 10 GiB volume PVC at 1000 MB the unnamed default collection takes six slots at first write. Those six reserve 6000 MB of the roughly 9900 available, leaving 3900 unclaimed and worth three more slots, so the server reports nine and only three are ever reachable by a bucket. One bucket can use them; the next is answered with "Not enough data nodes found" while the disk is still almost entirely free. At 250 MB the same PVC yields about thirty-nine slots, so five more full collections fit beside the default one. The price is four times as many volume files, each a .dat and an .idx: a 10 TiB volume server holds roughly 84k of them at this limit against 21k at 1000 MB, enough to cross a 65536 nofile limit where one is set. That count scales with data actually stored, and on a much larger PVC it is the reason to raise this value rather than lower it further. Existing volumes larger than the new limit keep working. The volume server takes the new limit from its heartbeat and never marks a volume read-only for exceeding it, and the master only drops such a volume from its writable set, so it stays readable while new writes go to freshly grown volumes. Unit tests pin both rendered flags. The limit is only half the arithmetic: the volume server's -max 0 is what makes the slot count auto-derived in the first place, and the template branch rendering it exists only because 0 is falsy in Go templates, so it reads as a redundant special case. Dropping it in a later simplification would pin the count to 7, leaving the master's flag rendering 250 while the slot arithmetic silently stopped applying. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 79602e9 commit 9399d8f

2 files changed

Lines changed: 62 additions & 10 deletions

File tree

‎packages/system/seaweedfs/tests/volume_slot_ceiling_test.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
suite: seaweedfs volume slot ceiling
2+
3+
# Two rendered flags decide how many volume slots a server reports, and the
4+
# slot count is what caps how many bucket collections an instance can hold:
5+
# -max 0 makes the count auto-derived from free disk, and -volumeSizeLimitMB
6+
# is the divisor. Both are asserted on the rendered output rather than on the
7+
# values key, because a chart bump changes the rendering, not the overlay.
8+
#
9+
# The zero branch in the volume template is a workaround for Go template
10+
# truthiness: 0 is falsy, so `not $dir.maxVolumes` is true for zero and the
11+
# next branch would render 7. It therefore looks like a redundant special
12+
# case for a value the following branch already covers. `make update` wipes
13+
# charts/ and refetches it, so the edit to fear is upstream reordering or
14+
# removing that branch and this chart picking it up on the next bump, which
15+
# would leave the master flag intact while the slot arithmetic stopped
16+
# applying.
17+
18+
templates:
19+
- charts/seaweedfs/templates/master/master-statefulset.yaml
20+
- charts/seaweedfs/templates/volume/volume-statefulset.yaml
21+
22+
release:
23+
name: seaweedfs
24+
namespace: cozy-seaweedfs
25+
26+
tests:
27+
- it: starts the master with the pinned volume size limit
28+
template: charts/seaweedfs/templates/master/master-statefulset.yaml
29+
asserts:
30+
- matchRegex:
31+
path: spec.template.spec.containers[0].command[2]
32+
pattern: '-volumeSizeLimitMB=250\s'
33+
34+
- it: leaves the volume server slot count auto-derived
35+
template: charts/seaweedfs/templates/volume/volume-statefulset.yaml
36+
asserts:
37+
- matchRegex:
38+
path: spec.template.spec.containers[0].command[2]
39+
pattern: '-max 0\s'

‎packages/system/seaweedfs/values.yaml‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -26,16 +26,29 @@ seaweedfs:
2626
# The guard also refuses to render at all if this pin is ever lifted.
2727
fullnameOverride: seaweedfs
2828
master:
29-
# 30 GB per volume × 10 GiB PVC = 0 → rounded to 1 maxVolume per
30-
# volume server. With defaultReplication=001 (one replica on a
31-
# different node) two volume servers can host exactly one logical
32-
# volume between them. The platform now ships a default
33-
# bucket-cozy-backups Bucket that grabs that single slot, so any
34-
# second collection (the bucket E2E, a tenant app bucket, etc.)
35-
# blocks on `assign volume: No writable volumes` until the test
36-
# times out. Smaller volumes = more parallel collections on a
37-
# 10 GiB E2E sandbox PVC. The upstream chart default is 1000.
38-
volumeSizeLimitMB: 1000
29+
# A volume server runs with -max 0, so its slot count is free disk minus
30+
# the headroom existing volumes still claim, divided by this limit. An
31+
# unfilled volume reserves its whole limit, so a volume counts against the
32+
# budget for as long as it stays writable. Replication 001 puts a copy on
33+
# each of the two servers, so a logical volume costs a slot on both, and
34+
# the master asks for six logical volumes per collection (Copy2Count, for
35+
# two copies), keeping whatever fit when the slots run out.
36+
#
37+
# On a 10 GiB PVC at 1000 MB the unnamed default collection takes six at
38+
# first write; those six reserve 6000 MB of the ~9900, leaving 3900
39+
# unclaimed for three more slots. Nine total, three of them ever reachable
40+
# by a bucket. At 250 MB the same PVC gives about thirty-nine, so five
41+
# more full collections fit beside the default one.
42+
#
43+
# The price is four times the volume files, a .dat and an .idx each: a
44+
# 10 TiB server holds roughly 84k of them here against 21k at 1000 MB,
45+
# enough to cross a 65536 nofile limit where one is set. Deriving this per
46+
# instance from volume.size was rejected because pools and zones each
47+
# carry their own PVC size while this setting is master-wide, so it would
48+
# have to track the smallest pool. Lowering master.volume_growth.copy_2
49+
# through master.config would shrink the six instead, at an unmeasured
50+
# cost to write spread within a collection.
51+
volumeSizeLimitMB: 250
3952
replicas: 3
4053
# replication type is XYZ:
4154
# X number of replica in other data centers

0 commit comments

Comments
 (0)