Skip to content

Commit 9b6ed66

Browse files
myasnikovdaniillexfrei
authored andcommitted
fix(linstor): exempt the satellite from a memory limit, like the controller
The satellite sizing rested on a false premise. The comment claimed the satellite is "a Go binary with a bounded working set" and sized a 2Gi limit on that basis, but it is the same Java linstor-server distribution the controller runs: images/piraeus-server/Dockerfile installs default-jre-headless and this container starts with `startSatellite`, against the controller's `startController`. That matters in the direction the sizing went. A memory.max is not only an RSS ceiling on a JVM, it is what container-aware heap sizing reads, so a 2Gi limit yields a heap of roughly a quarter of that while the 368Mi peak behind the request is already whole-process RSS. On a storage node with a large DRBD resource count that trades a rare pressure-driven kill for an OutOfMemoryError or GC thrash mid-operation, which is exactly the failure linstor-controller is exempted to avoid. So the satellite now carries requests and no limit, the same call cluster_test.yaml already pins for the controller. The cost is explicit rather than implicit: with this container limit-free the pod cgroup gets no memory.max, so the pod stays in the Talos OOM handler's victim set and the plunger sidecar limits bound a runaway sidecar and nothing more. Closing that needs a peak measured on a high-resource-count cluster plus an explicit heap share, not a guessed ceiling. satellites_test.yaml pins the exemption with notExists on the limit, so re-adding a ceiling fails the suite rather than passing quietly. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent d2e3eb4 commit 9b6ed66

3 files changed

Lines changed: 31 additions & 19 deletions

File tree

‎packages/system/linstor/templates/satellites-cozy.yaml‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,21 @@ spec:
2121
# workload thrashing against its own limit would get the satellite
2222
# killed, taking DRBD connections down with it.
2323
#
24-
# Note this is the opposite call from linstor-controller, which is
25-
# deliberately left limit-free in values.yaml (a JVM that must not be
26-
# OOM-killed mid-operation). The satellite is a Go binary with a bounded
27-
# working set, so a generous limit costs nothing and buys immunity.
24+
# Requests only, no limit: the same call linstor-controller makes in
25+
# values.yaml, and for the same reason, because this is the same JVM.
26+
# images/piraeus-server/Dockerfile installs default-jre-headless and
27+
# starts this container with `startSatellite`, the distribution the
28+
# controller runs with `startController`. A memory.max is what a
29+
# container-aware JVM reads to size its heap, so a limit here shrinks the
30+
# heap rather than only capping RSS. The cost is that the pod cgroup gets
31+
# no memory.max, so this pod stays in the OOM handler's victim set; closing
32+
# that needs a peak measured on a high-resource-count cluster plus an
33+
# explicit heap share, not a guessed ceiling.
2834
# 14d peak 368Mi on a 3-node dev cluster; grows with resource count.
2935
resources:
3036
requests:
3137
cpu: 100m
3238
memory: 128Mi
33-
limits:
34-
memory: 2Gi
3539
startupProbe:
3640
tcpSocket:
3741
port: linstor

‎packages/system/linstor/templates/satellites-plunger.yaml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,11 @@ spec:
1212
containers:
1313
- name: plunger
1414
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
15-
# Sidecars in the satellite pod. Every container needs a memory limit
16-
# for the pod cgroup to get memory.max, which is what keeps the pod out
17-
# of the Talos userspace OOM handler's victim set — one limit-free
18-
# sidecar is enough to put the whole satellite pod back in it.
15+
# Sidecars in the satellite pod. These limits bound a runaway sidecar,
16+
# they do not buy the pod immunity: the satellite container beside them
17+
# is deliberately limit-free because it is a JVM, so the pod cgroup gets
18+
# no memory.max either way and the pod stays in the Talos userspace OOM
19+
# handler's victim set. See satellites-cozy.yaml.
1920
# 14d peaks on a 3-node dev cluster: plunger 18Mi, drbd-logger 5Mi.
2021
resources:
2122
requests:

‎packages/system/linstor/tests/satellites_test.yaml‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,28 +6,35 @@ suite: satellite pod resource limits
66
# pressure. A tenant workload thrashing against its own limit would get the
77
# satellite killed, taking DRBD connections on that node down with it.
88
#
9-
# Every container in the pod needs a limit for the pod cgroup to get memory.max,
10-
# so the plunger sidecars are pinned alongside the satellite itself — one
11-
# limit-free sidecar is enough to put the whole pod back in the victim set.
9+
# The satellite is the exception and this suite pins it as one: it is the same JVM
10+
# as linstor-controller, which cluster_test.yaml pins as deliberately limit-free,
11+
# so it carries a request and no limit. A memory.max is what a container-aware JVM
12+
# reads to size its heap, so a limit shrinks the heap rather than only capping RSS,
13+
# and the 368Mi peak behind the request grows with DRBD resource count and was
14+
# measured on three nodes.
1215
#
13-
# Note this is the opposite call from linstor-controller, which cluster_test.yaml
14-
# pins as deliberately limit-free (a JVM that must not be OOM-killed mid-operation).
16+
# The consequence is pinned rather than left implicit: with the satellite container
17+
# limit-free the pod cgroup gets no memory.max, so the plunger limits below bound a
18+
# runaway sidecar and nothing more. Reverting the exemption means measuring on a
19+
# high-resource-count cluster first.
1520

1621
templates:
1722
- templates/satellites-cozy.yaml
1823
- templates/satellites-no-drbd.yaml
1924
- templates/satellites-plunger.yaml
2025

2126
tests:
22-
- it: satellite declares a memory limit and request
27+
- it: satellite declares a memory request and deliberately no limit
2328
template: templates/satellites-cozy.yaml
2429
asserts:
25-
- equal:
26-
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].resources.limits.memory
27-
value: 2Gi
2830
- equal:
2931
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].resources.requests.memory
3032
value: 128Mi
33+
- equal:
34+
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].resources.requests.cpu
35+
value: 100m
36+
- notExists:
37+
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].resources.limits
3138

3239
- it: plunger sidecars declare memory limits
3340
template: templates/satellites-plunger.yaml

0 commit comments

Comments
 (0)