Skip to content

Commit 6f8928d

Browse files
authored
docs(talm): document the ZFS export-on-shutdown knob (#721)
The talm page gets a section on the new cozystack preset setting that stops the ZFS extension from exporting the pool on shutdown. Without it, a node with LINSTOR on ZFS hangs in `rebooting` on every reboot, upgrade and reset while DRBD holds its zvols (cozystack/cozystack#4597). The talm side is cozystack/talm#251. The section says which disk layouts need the export back and that the fix only works on Talos v1.13.4 or later. Most of the text is the one-time reboot for clusters that already run. The running `zfs-service` keeps its old environment, so the first reboot after the change can still hang. The step that closes a dm-crypt mapping left by the LINSTOR LUKS layer has not been run on a live cluster yet. The satellite commands use `ds/linstor-satellite.<node>`. Piraeus runs one DaemonSet per node, so the `pod/linstor-satellite.<node>` form used on other install pages does not resolve. I left those pages alone in this PR. The heading says Talm v0.36+, the next minor after v0.35.0, so this should merge once that release is out. Applied to `next` and `v1.6`.
2 parents 8017c1c + 5ca6853 commit 6f8928d

2 files changed

Lines changed: 122 additions & 2 deletions

File tree

  • content/en/docs

‎content/en/docs/next/install/kubernetes/talm.md‎

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,15 +238,75 @@ extraMachineFiles:
238238

239239
The `generic` preset ships no defaults under any of these sections — each block emits only when the matching `extra*` key is non-empty.
240240

241-
Beyond the `extra*` extension points, the `cozystack` preset exposes two opinionated tunables you can change without forking the chart:
241+
Beyond the `extra*` extension points, the `cozystack` preset exposes three opinionated tunables you can change without forking the chart:
242242

243243
| Key | Default | Effect |
244244
| --- | --- | --- |
245245
| `tcpKeepaliveTuning` | `false` | When `true`, adds `net.ipv4.tcp_keepalive_time=600` / `intvl=10` / `probes=6` to `machine.sysctls`, reaping a dead idle socket in ~660s instead of the kernel default ~2h. These sysctls are kernel-wide — they change failure detection for every long-lived idle TCP connection on the node, not just DRBD — so they are opt-in. DRBD already detects dead peers in seconds via its own protocol-level ping, so leave this off unless you specifically want faster node-wide dead-socket detection. |
246246
| `etcd.quotaBackendBytes` | `"8589934592"` (8 GiB) | etcd backend DB size ceiling, emitted as `cluster.etcd.extraArgs.quota-backend-bytes` on controlplane nodes only. Raises etcd's own 2 GiB default so a LINSTOR-heavy control plane holding many DRBD-resource CRDs in aggregate does not trip the NOSPACE alarm. It is a ceiling, not a reservation: a small DB stays small and costs no extra RAM/disk. Set it to `""` to fall back to etcd's built-in default. This governs total DB size, not single-object size — per-object writes stay bounded by kube-apiserver's fixed 3 MiB request-body limit, which has no configuration knob. |
247+
| `zfs.exportOnShutdown` | `false` | When `false`, emits an `ExtensionServiceConfig` for `zfs-service` with `ZFS_EXPORT_TIMEOUT=0`, so the ZFS extension does not export the pool on shutdown and re-imports it with `zpool import -fal` on the next boot. The section below explains when to set it to `true`. |
247248
248249
The five always-on DRBD/LINSTOR sysctls listed in the `extraSysctls` row above ship unconditionally on the `cozystack` preset — they address TCP-port exhaustion observed under DRBD reconnect storms and have no equivalent on the `generic` preset.
249250
251+
#### ZFS pool export on shutdown (Talm v0.36+)
252+
253+
The ZFS extension runs `zpool export -a` when the node shuts down. While DRBD holds the zvols, the export blocks in the kernel and cannot be killed, so the node stays in `rebooting` on every reboot, upgrade and reset until it is power-cycled. The `cozystack` preset turns the export off by default.
254+
255+
The default fits a pool built on raw disks or partitions, which is the standard Cozystack layout. LUKS on top of zvols (the LINSTOR LUKS layer) and native ZFS encryption also keep raw disks under the pool, so they work with the default too.
256+
257+
Set `zfs.exportOnShutdown: true` in two cases:
258+
259+
- The pool is built on dm-crypt/LUKS devices. Without the export, `cryptsetup close` cannot release the device at shutdown. With this layout, bring DRBD down on the node before every reboot instead, as described below.
260+
- Several hosts can import the pool, for example over shared SAN or multipath storage. The export protects the pool from being imported twice.
261+
262+
The setting takes effect on Talos v1.13.4 and later, where the ZFS extension reads `ZFS_EXPORT_TIMEOUT`. Talos v1.13.0 to v1.13.3 always export, so on those versions the document does nothing: move to v1.13.4 or later, or bring DRBD down before every reboot as described below. Earlier versions do not export at all.
263+
264+
To keep the export with your own timeout, set `zfs.exportOnShutdown: true` and add your own `zfs-service` document. A `ZFS_EXPORT_TIMEOUT` placed in a node file is appended next to the preset's entry instead of replacing it.
265+
266+
Node files written by `talm template` carry the rendered `zfs-service` document, and `talm apply` merges it back over the chart. After switching to `true`, remove that document from every node file, or regenerate the files, otherwise the export stays disabled.
267+
268+
##### One-time reboot on a running cluster
269+
270+
An existing project gets the document only after its preset templates are updated: run `talm init --update --preset cozystack` after upgrading the talm binary, review the template diff, then `talm apply` each node. No `values.yaml` change is needed, since a missing `zfs` key means the default.
271+
272+
The running `zfs-service` keeps the environment it was started with. On a node currently running Talos v1.13.0 or later, the first reboot after the change, including the one inside `talm upgrade`, still runs the export and can hang. Nodes on older Talos do not export, so their first reboot is safe. For that one reboot, handle each affected node in turn:
273+
274+
1. Drain the node:
275+
276+
```bash
277+
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
278+
```
279+
280+
2. Bring DRBD down on the node:
281+
282+
```bash
283+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- drbdadm down all
284+
```
285+
286+
3. Check that no zvol has a holder left. The command must print nothing, and any holder it lists blocks the reboot:
287+
288+
```bash
289+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- sh -c 'find /sys/block/zd*/holders -mindepth 1 2>/dev/null'
290+
```
291+
292+
A `drbd*` holder means a resource is still up, usually one still Primary on this node. Find it with `drbdsetup status` in the same pod and repeat step 2 once nothing uses it.
293+
294+
A `dm-*` holder is usually the dm-crypt mapping of a resource with the LINSTOR LUKS layer, which stays open after DRBD goes down. Close it by name, with `dm-N` replaced by the device the check printed:
295+
296+
```bash
297+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- sh -c 'cryptsetup close "$(cat /sys/block/dm-N/dm/name)"'
298+
```
299+
300+
4. Reboot the node right away, or run `talm upgrade` for it.
301+
302+
5. Once the node is back and `Ready`, return it to scheduling:
303+
304+
```bash
305+
kubectl uncordon <node>
306+
```
307+
308+
From the next boot on, the setting is in effect and reboots need no manual steps. Clusters installed with the document already in their config are not affected.
309+
250310
#### Describing the node's network and registries from values (Talm v0.34+)
251311

252312
The `extra*` keys above add to the preset's curated defaults. A second group of keys describes parts of the config the preset used to leave to the template — so a topology that once required forking `templates/_helpers.tpl` is now expressible in `values.yaml`. Both presets accept all of them.

‎content/en/docs/v1.6/install/kubernetes/talm.md‎

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,15 +238,75 @@ extraMachineFiles:
238238

239239
The `generic` preset ships no defaults under any of these sections — each block emits only when the matching `extra*` key is non-empty.
240240

241-
Beyond the `extra*` extension points, the `cozystack` preset exposes two opinionated tunables you can change without forking the chart:
241+
Beyond the `extra*` extension points, the `cozystack` preset exposes three opinionated tunables you can change without forking the chart:
242242

243243
| Key | Default | Effect |
244244
| --- | --- | --- |
245245
| `tcpKeepaliveTuning` | `false` | When `true`, adds `net.ipv4.tcp_keepalive_time=600` / `intvl=10` / `probes=6` to `machine.sysctls`, reaping a dead idle socket in ~660s instead of the kernel default ~2h. These sysctls are kernel-wide — they change failure detection for every long-lived idle TCP connection on the node, not just DRBD — so they are opt-in. DRBD already detects dead peers in seconds via its own protocol-level ping, so leave this off unless you specifically want faster node-wide dead-socket detection. |
246246
| `etcd.quotaBackendBytes` | `"8589934592"` (8 GiB) | etcd backend DB size ceiling, emitted as `cluster.etcd.extraArgs.quota-backend-bytes` on controlplane nodes only. Raises etcd's own 2 GiB default so a LINSTOR-heavy control plane holding many DRBD-resource CRDs in aggregate does not trip the NOSPACE alarm. It is a ceiling, not a reservation: a small DB stays small and costs no extra RAM/disk. Set it to `""` to fall back to etcd's built-in default. This governs total DB size, not single-object size — per-object writes stay bounded by kube-apiserver's fixed 3 MiB request-body limit, which has no configuration knob. |
247+
| `zfs.exportOnShutdown` | `false` | When `false`, emits an `ExtensionServiceConfig` for `zfs-service` with `ZFS_EXPORT_TIMEOUT=0`, so the ZFS extension does not export the pool on shutdown and re-imports it with `zpool import -fal` on the next boot. The section below explains when to set it to `true`. |
247248
248249
The five always-on DRBD/LINSTOR sysctls listed in the `extraSysctls` row above ship unconditionally on the `cozystack` preset — they address TCP-port exhaustion observed under DRBD reconnect storms and have no equivalent on the `generic` preset.
249250
251+
#### ZFS pool export on shutdown (Talm v0.36+)
252+
253+
The ZFS extension runs `zpool export -a` when the node shuts down. While DRBD holds the zvols, the export blocks in the kernel and cannot be killed, so the node stays in `rebooting` on every reboot, upgrade and reset until it is power-cycled. The `cozystack` preset turns the export off by default.
254+
255+
The default fits a pool built on raw disks or partitions, which is the standard Cozystack layout. LUKS on top of zvols (the LINSTOR LUKS layer) and native ZFS encryption also keep raw disks under the pool, so they work with the default too.
256+
257+
Set `zfs.exportOnShutdown: true` in two cases:
258+
259+
- The pool is built on dm-crypt/LUKS devices. Without the export, `cryptsetup close` cannot release the device at shutdown. With this layout, bring DRBD down on the node before every reboot instead, as described below.
260+
- Several hosts can import the pool, for example over shared SAN or multipath storage. The export protects the pool from being imported twice.
261+
262+
The setting takes effect on Talos v1.13.4 and later, where the ZFS extension reads `ZFS_EXPORT_TIMEOUT`. Talos v1.13.0 to v1.13.3 always export, so on those versions the document does nothing: move to v1.13.4 or later, or bring DRBD down before every reboot as described below. Earlier versions do not export at all.
263+
264+
To keep the export with your own timeout, set `zfs.exportOnShutdown: true` and add your own `zfs-service` document. A `ZFS_EXPORT_TIMEOUT` placed in a node file is appended next to the preset's entry instead of replacing it.
265+
266+
Node files written by `talm template` carry the rendered `zfs-service` document, and `talm apply` merges it back over the chart. After switching to `true`, remove that document from every node file, or regenerate the files, otherwise the export stays disabled.
267+
268+
##### One-time reboot on a running cluster
269+
270+
An existing project gets the document only after its preset templates are updated: run `talm init --update --preset cozystack` after upgrading the talm binary, review the template diff, then `talm apply` each node. No `values.yaml` change is needed, since a missing `zfs` key means the default.
271+
272+
The running `zfs-service` keeps the environment it was started with. On a node currently running Talos v1.13.0 or later, the first reboot after the change, including the one inside `talm upgrade`, still runs the export and can hang. Nodes on older Talos do not export, so their first reboot is safe. For that one reboot, handle each affected node in turn:
273+
274+
1. Drain the node:
275+
276+
```bash
277+
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
278+
```
279+
280+
2. Bring DRBD down on the node:
281+
282+
```bash
283+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- drbdadm down all
284+
```
285+
286+
3. Check that no zvol has a holder left. The command must print nothing, and any holder it lists blocks the reboot:
287+
288+
```bash
289+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- sh -c 'find /sys/block/zd*/holders -mindepth 1 2>/dev/null'
290+
```
291+
292+
A `drbd*` holder means a resource is still up, usually one still Primary on this node. Find it with `drbdsetup status` in the same pod and repeat step 2 once nothing uses it.
293+
294+
A `dm-*` holder is usually the dm-crypt mapping of a resource with the LINSTOR LUKS layer, which stays open after DRBD goes down. Close it by name, with `dm-N` replaced by the device the check printed:
295+
296+
```bash
297+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- sh -c 'cryptsetup close "$(cat /sys/block/dm-N/dm/name)"'
298+
```
299+
300+
4. Reboot the node right away, or run `talm upgrade` for it.
301+
302+
5. Once the node is back and `Ready`, return it to scheduling:
303+
304+
```bash
305+
kubectl uncordon <node>
306+
```
307+
308+
From the next boot on, the setting is in effect and reboots need no manual steps. Clusters installed with the document already in their config are not affected.
309+
250310
#### Describing the node's network and registries from values (Talm v0.34+)
251311

252312
The `extra*` keys above add to the preset's curated defaults. A second group of keys describes parts of the config the preset used to leave to the template — so a topology that once required forking `templates/_helpers.tpl` is now expressible in `values.yaml`. Both presets accept all of them.

0 commit comments

Comments
 (0)