Skip to content

Commit b8c86f2

Browse files
committed
fix(linstor): add opt-in graceful satellite shutdown on Talos
Signed-off-by: Yan Bondarenko <[email protected]>
1 parent b99db14 commit b8c86f2

7 files changed

Lines changed: 444 additions & 0 deletions

File tree

‎packages/system/linstor/Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ LINSTOR_CSI_VERSION ?= v1.11.2
99

1010
test:
1111
helm unittest .
12+
python3 -m unittest discover -s tests -p 'test_*.py' -v
1213

1314
image: image-piraeus-server image-linstor-csi
1415

‎packages/system/linstor/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Talos satellite graceful shutdown
2+
3+
Cozystack's Talos configuration removes the systemd `drbd-shutdown-guard`. The optional satellite preStop hook releases unused Secondary DRBD resources so their backing devices, including ZFS zvols, can be released during normal shutdown.
4+
5+
| Parameter | Default | Effect |
6+
| --- | --- | --- |
7+
| `talos.gracefulShutdown.enabled` | `false` | Add the satellite preStop hook when both `talos.enabled` and `drbd.enabled` are also `true`. |
8+
9+
The default satellite lifecycle is unchanged. To enable the hook, set these values for the `linstor` component of the `cozystack.linstor` package:
10+
11+
```yaml
12+
talos:
13+
gracefulShutdown:
14+
enabled: true
15+
```
16+
17+
Keep the setting in the package's declarative configuration at `spec.components.linstor.values.talos.gracefulShutdown.enabled`. The LINSTOR Helm chart embeds `hack/satellite-pre-stop.py` in `LinstorSatelliteConfiguration/cozystack-talos`; Piraeus converts its `podTemplate` to a strategic merge patch for each selected satellite DaemonSet. Package reconciliation and upgrades therefore retain the hook while the value remains enabled. Editing a generated Pod or DaemonSet directly does not persist through reconciliation. The hook uses Python and `drbdsetup` already supplied by the Cozystack satellite image.
18+
19+
The hook runs on every graceful satellite container stop, including ordinary pod restarts. It skips all Primary resources and any resource with an open device, rechecks each candidate, then calls `drbdsetup down` without force. Unknown or malformed status, command failure and timeout stop further actions. It does not demote resources, call the Kubernetes API, or change access permissions.
20+
21+
Enable it only when satellite and node restarts are performed sequentially, with replication and quorum checked between nodes. Piraeus creates a separate DaemonSet for each node; this chart does not serialize their rollouts. A shared configuration or image update can therefore stop unused Secondary replicas on several nodes at once and pause I/O until they return. Changing or disabling the setting can trigger a pod rollout, and the old pod still runs its existing hook.
22+
23+
Drain workloads before a planned node shutdown. The hook has a 60-second work budget, with each status command limited to 4 seconds and each down command to 10 seconds, and requests a 90-second pod termination grace period. The kubelet's node-shutdown budget can terminate the hook earlier; the pod grace period does not override that budget. Forced reboot, failed kubelet shutdown, still-open devices and unfinished replication are not fixed by this hook.
24+
25+
Related upstream discussion: [Piraeus orderly shutdown on Talos](https://github.com/piraeusdatastore/piraeus-operator/issues/860).
26+
27+
Run `make test` in this directory with Helm, helm-unittest and Python 3.11 or later. The tests cover rendering with each feature flag, resource selection and rechecks, bounded errors and timeouts, dry-run behavior, and logging in the shared pod PID namespace.
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
import json
2+
import os
3+
import re
4+
import subprocess
5+
import sys
6+
import time
7+
8+
9+
def connect_container_stdout():
10+
# Piraeus shares the pod PID namespace, so /proc/1 belongs to the sandbox.
11+
with open("/proc/self/cgroup") as stream:
12+
cgroup = stream.read()
13+
candidates = []
14+
for pid in os.listdir("/proc"):
15+
if not pid.isdigit():
16+
continue
17+
try:
18+
with open("/proc/" + pid + "/cgroup") as stream:
19+
same_container = stream.read() == cgroup
20+
with open("/proc/" + pid + "/comm") as stream:
21+
is_java = stream.read().strip() == "java"
22+
if same_container and is_java:
23+
candidates.append(pid)
24+
except OSError:
25+
continue
26+
if len(candidates) != 1:
27+
raise RuntimeError("unique-satellite-java-required")
28+
fd = os.open("/proc/" + candidates[0] + "/fd/1", os.O_WRONLY | os.O_APPEND)
29+
try:
30+
os.dup2(fd, 1)
31+
os.dup2(fd, 2)
32+
finally:
33+
os.close(fd)
34+
35+
36+
def log(event, **fields):
37+
print(json.dumps(dict(event=event, **fields), sort_keys=True), flush=True)
38+
39+
40+
def parse_status(text):
41+
data = json.loads(text)
42+
if type(data) is not list:
43+
raise ValueError("status-list-required")
44+
result = {}
45+
for resource in data:
46+
if type(resource) is not dict:
47+
raise ValueError("resource-object-required")
48+
name = resource.get("name")
49+
if type(name) is not str or not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,254}", name):
50+
raise ValueError("invalid-resource-name")
51+
if name in result or resource.get("role") not in ("Primary", "Secondary"):
52+
raise ValueError("duplicate-name-or-unknown-role")
53+
devices = resource.get("devices")
54+
if type(devices) is not list or not devices:
55+
raise ValueError("devices-required")
56+
volumes = set()
57+
for device in devices:
58+
if type(device) is not dict or type(device.get("open")) is not bool:
59+
raise ValueError("explicit-open-boolean-required")
60+
volume = device.get("volume")
61+
if type(volume) is not int or volume < 0 or volume in volumes:
62+
raise ValueError("invalid-volume")
63+
volumes.add(volume)
64+
result[name] = resource
65+
return result
66+
67+
68+
def eligible(resource):
69+
# A pod restart must leave mounted volumes and their Primary role intact.
70+
return resource["role"] == "Secondary" and all(device["open"] is False for device in resource["devices"])
71+
72+
73+
def read_status(name=None, timeout=4):
74+
args = ["/usr/sbin/drbdsetup", "status"]
75+
if name is not None:
76+
args.append(name)
77+
args.append("--json")
78+
result = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
79+
if result.returncode != 0:
80+
raise RuntimeError("status-command-failed")
81+
return parse_status(result.stdout)
82+
83+
84+
def main():
85+
if sys.argv[1:] not in ([], ["--execute"]):
86+
raise ValueError("unsupported-arguments")
87+
execute = sys.argv[1:] == ["--execute"]
88+
deadline = time.monotonic() + 60
89+
initial = read_status()
90+
for name, resource in initial.items():
91+
if not eligible(resource):
92+
log("skip-in-use", resource=name, role=resource["role"])
93+
continue
94+
remaining = deadline - time.monotonic()
95+
if remaining < 1:
96+
raise RuntimeError("hook-deadline")
97+
current = read_status(name, timeout=min(4, remaining))
98+
if set(current) != {name}:
99+
raise ValueError("resource-read-mismatch")
100+
if not eligible(current[name]):
101+
log("skip-state-changed", resource=name)
102+
continue
103+
if not execute:
104+
log("would-down", resource=name)
105+
continue
106+
remaining = deadline - time.monotonic()
107+
if remaining < 1:
108+
raise RuntimeError("hook-deadline")
109+
log("down-start", resource=name)
110+
result = subprocess.run(
111+
["/usr/sbin/drbdsetup", "down", name],
112+
stdout=subprocess.DEVNULL,
113+
stderr=subprocess.DEVNULL,
114+
timeout=min(10, remaining),
115+
)
116+
if result.returncode != 0:
117+
raise RuntimeError("down-command-failed")
118+
log("down-finished", resource=name)
119+
log("finished", execute=execute, resources=len(initial))
120+
121+
122+
if __name__ == "__main__":
123+
try:
124+
connect_container_stdout()
125+
main()
126+
except Exception as error:
127+
log("abort-no-further-actions", error_type=type(error).__name__)
128+
sys.exit(1)

‎packages/system/linstor/templates/satellites-talos.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,31 @@
11
{{- if .Values.talos.enabled }}
2+
{{- $shutdown := .Values.talos.gracefulShutdown | default dict }}
3+
{{- $enabled := dig "enabled" false $shutdown }}
4+
{{- if kindIs "invalid" $enabled }}{{- $enabled = false }}{{- end }}
5+
{{- if not (kindIs "bool" $enabled) }}
6+
{{- fail "linstor: talos.gracefulShutdown.enabled must be a boolean" }}
7+
{{- end }}
28
apiVersion: piraeus.io/v1
39
kind: LinstorSatelliteConfiguration
410
metadata:
511
name: cozystack-talos
612
spec:
13+
{{- if and .Values.drbd.enabled $enabled }}
14+
podTemplate:
15+
spec:
16+
terminationGracePeriodSeconds: 90
17+
containers:
18+
- name: linstor-satellite
19+
lifecycle:
20+
preStop:
21+
exec:
22+
command:
23+
- /usr/bin/python3
24+
- -c
25+
- |
26+
{{ required "satellite pre-stop script is missing" (.Files.Get "hack/satellite-pre-stop.py") | indent 16 }}
27+
- --execute
28+
{{- end }}
729
patches:
830
- target:
931
group: apps
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
suite: Talos satellite graceful shutdown
2+
templates:
3+
- templates/satellites-talos.yaml
4+
5+
tests:
6+
- it: preserves the existing satellite lifecycle by default
7+
asserts:
8+
- isKind:
9+
of: LinstorSatelliteConfiguration
10+
- equal:
11+
path: metadata.name
12+
value: cozystack-talos
13+
- notExists:
14+
path: spec.podTemplate
15+
16+
- it: runs the bounded hook only inside the satellite container
17+
set:
18+
talos.gracefulShutdown.enabled: true
19+
asserts:
20+
- equal:
21+
path: metadata.name
22+
value: cozystack-talos
23+
- equal:
24+
path: spec.podTemplate.spec.terminationGracePeriodSeconds
25+
value: 90
26+
- lengthEqual:
27+
path: spec.podTemplate.spec.containers
28+
count: 1
29+
- equal:
30+
path: spec.podTemplate.spec.containers[0].name
31+
value: linstor-satellite
32+
- equal:
33+
path: spec.podTemplate.spec.containers[0].lifecycle.preStop.exec.command[0]
34+
value: /usr/bin/python3
35+
- equal:
36+
path: spec.podTemplate.spec.containers[0].lifecycle.preStop.exec.command[1]
37+
value: -c
38+
- matchRegex:
39+
path: spec.podTemplate.spec.containers[0].lifecycle.preStop.exec.command[2]
40+
pattern: 'def main\(\):'
41+
- equal:
42+
path: spec.podTemplate.spec.containers[0].lifecycle.preStop.exec.command[3]
43+
value: --execute
44+
45+
- it: does not turn a quoted false into an enabled shutdown hook
46+
set:
47+
talos.gracefulShutdown.enabled: "false"
48+
asserts:
49+
- failedTemplate:
50+
errorMessage: 'linstor: talos.gracefulShutdown.enabled must be a boolean'
51+
52+
- it: clearing the opt-in value restores the default
53+
set:
54+
talos.gracefulShutdown.enabled: null
55+
asserts:
56+
- isKind:
57+
of: LinstorSatelliteConfiguration
58+
- notExists:
59+
path: spec.podTemplate
60+
61+
- it: leaves the systemd shutdown guard to Piraeus on generic Linux
62+
set:
63+
talos.enabled: false
64+
talos.gracefulShutdown.enabled: true
65+
asserts:
66+
- hasDocuments:
67+
count: 0
68+
69+
- it: omits the hook on Talos substrates without DRBD
70+
set:
71+
drbd.enabled: false
72+
talos.gracefulShutdown.enabled: true
73+
asserts:
74+
- isKind:
75+
of: LinstorSatelliteConfiguration
76+
- notExists:
77+
path: spec.podTemplate
78+
- matchRegex:
79+
path: spec.patches[0].patch
80+
pattern: 'name: drbd-shutdown-guard'

0 commit comments

Comments
 (0)