You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6f8928d
Browse filesBrowse the repository at this point in the historyBrowse files
docs(talm): document the ZFS export-on-shutdown knob (#721)
The talm page gets a section on the new cozystack preset setting that
stops the ZFS extension from exporting the pool on shutdown. Without it,
a node with LINSTOR on ZFS hangs in `rebooting` on every reboot, upgrade
and reset while DRBD holds its zvols (cozystack/cozystack#4597). The
talm side is cozystack/talm#251.
The section says which disk layouts need the export back and that the
fix only works on Talos v1.13.4 or later. Most of the text is the
one-time reboot for clusters that already run. The running `zfs-service`
keeps its old environment, so the first reboot after the change can
still hang. The step that closes a dm-crypt mapping left by the LINSTOR
LUKS layer has not been run on a live cluster yet.
The satellite commands use `ds/linstor-satellite.<node>`. Piraeus runs
one DaemonSet per node, so the `pod/linstor-satellite.<node>` form used
on other install pages does not resolve. I left those pages alone in
this PR.
The heading says Talm v0.36+, the next minor after v0.35.0, so this
should merge once that release is out. Applied to `next` and `v1.6`.
Copy file name to clipboardExpand all lines: content/en/docs/next/install/kubernetes/talm.md
+61-1Lines changed: 61 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -238,15 +238,75 @@ extraMachineFiles:
238
238
239
239
The `generic` preset ships no defaults under any of these sections — each block emits only when the matching `extra*` key is non-empty.
240
240
241
-
Beyond the `extra*` extension points, the `cozystack` preset exposes two opinionated tunables you can change without forking the chart:
241
+
Beyond the `extra*` extension points, the `cozystack` preset exposes three opinionated tunables you can change without forking the chart:
242
242
243
243
| Key | Default | Effect |
244
244
| --- | --- | --- |
245
245
|`tcpKeepaliveTuning`|`false`| When `true`, adds `net.ipv4.tcp_keepalive_time=600` / `intvl=10` / `probes=6` to `machine.sysctls`, reaping a dead idle socket in~660s instead of the kernel default ~2h. These sysctls are kernel-wide — they change failure detection forevery long-lived idle TCP connection on the node, not just DRBD — so they are opt-in. DRBD already detects dead peersin seconds via its own protocol-level ping, so leave this off unless you specifically want faster node-wide dead-socket detection. |
246
246
|`etcd.quotaBackendBytes`|`"8589934592"` (8 GiB) | etcd backend DB size ceiling, emitted as `cluster.etcd.extraArgs.quota-backend-bytes` on controlplane nodes only. Raises etcd's own 2 GiB default so a LINSTOR-heavy control plane holding many DRBD-resource CRDs in aggregate does not trip the NOSPACE alarm. It is a ceiling, not a reservation: a small DB stays small and costs no extra RAM/disk. Set it to `""` to fall back to etcd's built-in default. This governs total DB size, not single-object size — per-object writes stay bounded by kube-apiserver's fixed 3 MiB request-body limit, which has no configuration knob. |
247
+
| `zfs.exportOnShutdown` | `false` | When `false`, emits an `ExtensionServiceConfig` for `zfs-service` with `ZFS_EXPORT_TIMEOUT=0`, so the ZFS extension does not export the pool on shutdown and re-imports it with `zpool import -fal` on the next boot. The section below explains when to set it to `true`. |
247
248
248
249
The five always-on DRBD/LINSTOR sysctls listed in the `extraSysctls` row above ship unconditionally on the `cozystack` preset — they address TCP-port exhaustion observed under DRBD reconnect storms and have no equivalent on the `generic` preset.
249
250
251
+
#### ZFS pool export on shutdown (Talm v0.36+)
252
+
253
+
The ZFS extension runs `zpool export -a` when the node shuts down. While DRBD holds the zvols, the export blocks in the kernel and cannot be killed, so the node stays in `rebooting` on every reboot, upgrade and reset until it is power-cycled. The `cozystack` preset turns the export off by default.
254
+
255
+
The default fits a pool built on raw disks or partitions, which is the standard Cozystack layout. LUKS on top of zvols (the LINSTOR LUKS layer) and native ZFS encryption also keep raw disks under the pool, so they work with the default too.
256
+
257
+
Set `zfs.exportOnShutdown: true` in two cases:
258
+
259
+
- The pool is built on dm-crypt/LUKS devices. Without the export, `cryptsetup close` cannot release the device at shutdown. With this layout, bring DRBD down on the node before every reboot instead, as described below.
260
+
- Several hosts can import the pool, for example over shared SAN or multipath storage. The export protects the pool from being imported twice.
261
+
262
+
The setting takes effect on Talos v1.13.4 and later, where the ZFS extension reads `ZFS_EXPORT_TIMEOUT`. Talos v1.13.0 to v1.13.3 always export, so on those versions the document does nothing: move to v1.13.4 or later, or bring DRBD down before every reboot as described below. Earlier versions do not export at all.
263
+
264
+
To keep the export with your own timeout, set `zfs.exportOnShutdown: true` and add your own `zfs-service` document. A `ZFS_EXPORT_TIMEOUT` placed in a node file is appended next to the preset's entry instead of replacing it.
265
+
266
+
Node files written by `talm template` carry the rendered `zfs-service` document, and `talm apply` merges it back over the chart. After switching to `true`, remove that document from every node file, or regenerate the files, otherwise the export stays disabled.
267
+
268
+
##### One-time reboot on a running cluster
269
+
270
+
An existing project gets the document only after its preset templates are updated: run `talm init --update --preset cozystack` after upgrading the talm binary, review the template diff, then`talm apply` each node. No `values.yaml` change is needed, since a missing `zfs` key means the default.
271
+
272
+
The running `zfs-service` keeps the environment it was started with. On a node currently running Talos v1.13.0 or later, the first reboot after the change, including the one inside `talm upgrade`, still runs the export and can hang. Nodes on older Talos do not export, so their first reboot is safe. For that one reboot, handle each affected node in turn:
A `drbd*` holder means a resource is still up, usually one still Primary on this node. Find it with `drbdsetup status`in the same pod and repeat step 2 once nothing uses it.
293
+
294
+
A `dm-*` holder is usually the dm-crypt mapping of a resource with the LINSTOR LUKS layer, which stays open after DRBD goes down. Close it by name, with `dm-N` replaced by the device the check printed:
295
+
296
+
```bash
297
+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- sh -c 'cryptsetup close "$(cat /sys/block/dm-N/dm/name)"'
298
+
```
299
+
300
+
4. Reboot the node right away, or run `talm upgrade`for it.
301
+
302
+
5. Once the node is back and `Ready`, return it to scheduling:
303
+
304
+
```bash
305
+
kubectl uncordon <node>
306
+
```
307
+
308
+
From the next boot on, the setting is in effect and reboots need no manual steps. Clusters installed with the document already in their config are not affected.
309
+
250
310
#### Describing the node's network and registries from values (Talm v0.34+)
251
311
252
312
The `extra*` keys above add to the preset's curated defaults. A second group of keys describes parts of the config the preset used to leave to the template — so a topology that once required forking `templates/_helpers.tpl` is now expressible in `values.yaml`. Both presets accept all of them.
Copy file name to clipboardExpand all lines: content/en/docs/v1.6/install/kubernetes/talm.md
+61-1Lines changed: 61 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -238,15 +238,75 @@ extraMachineFiles:
238
238
239
239
The `generic` preset ships no defaults under any of these sections — each block emits only when the matching `extra*` key is non-empty.
240
240
241
-
Beyond the `extra*` extension points, the `cozystack` preset exposes two opinionated tunables you can change without forking the chart:
241
+
Beyond the `extra*` extension points, the `cozystack` preset exposes three opinionated tunables you can change without forking the chart:
242
242
243
243
| Key | Default | Effect |
244
244
| --- | --- | --- |
245
245
|`tcpKeepaliveTuning`|`false`| When `true`, adds `net.ipv4.tcp_keepalive_time=600` / `intvl=10` / `probes=6` to `machine.sysctls`, reaping a dead idle socket in~660s instead of the kernel default ~2h. These sysctls are kernel-wide — they change failure detection forevery long-lived idle TCP connection on the node, not just DRBD — so they are opt-in. DRBD already detects dead peersin seconds via its own protocol-level ping, so leave this off unless you specifically want faster node-wide dead-socket detection. |
246
246
|`etcd.quotaBackendBytes`|`"8589934592"` (8 GiB) | etcd backend DB size ceiling, emitted as `cluster.etcd.extraArgs.quota-backend-bytes` on controlplane nodes only. Raises etcd's own 2 GiB default so a LINSTOR-heavy control plane holding many DRBD-resource CRDs in aggregate does not trip the NOSPACE alarm. It is a ceiling, not a reservation: a small DB stays small and costs no extra RAM/disk. Set it to `""` to fall back to etcd's built-in default. This governs total DB size, not single-object size — per-object writes stay bounded by kube-apiserver's fixed 3 MiB request-body limit, which has no configuration knob. |
247
+
| `zfs.exportOnShutdown` | `false` | When `false`, emits an `ExtensionServiceConfig` for `zfs-service` with `ZFS_EXPORT_TIMEOUT=0`, so the ZFS extension does not export the pool on shutdown and re-imports it with `zpool import -fal` on the next boot. The section below explains when to set it to `true`. |
247
248
248
249
The five always-on DRBD/LINSTOR sysctls listed in the `extraSysctls` row above ship unconditionally on the `cozystack` preset — they address TCP-port exhaustion observed under DRBD reconnect storms and have no equivalent on the `generic` preset.
249
250
251
+
#### ZFS pool export on shutdown (Talm v0.36+)
252
+
253
+
The ZFS extension runs `zpool export -a` when the node shuts down. While DRBD holds the zvols, the export blocks in the kernel and cannot be killed, so the node stays in `rebooting` on every reboot, upgrade and reset until it is power-cycled. The `cozystack` preset turns the export off by default.
254
+
255
+
The default fits a pool built on raw disks or partitions, which is the standard Cozystack layout. LUKS on top of zvols (the LINSTOR LUKS layer) and native ZFS encryption also keep raw disks under the pool, so they work with the default too.
256
+
257
+
Set `zfs.exportOnShutdown: true` in two cases:
258
+
259
+
- The pool is built on dm-crypt/LUKS devices. Without the export, `cryptsetup close` cannot release the device at shutdown. With this layout, bring DRBD down on the node before every reboot instead, as described below.
260
+
- Several hosts can import the pool, for example over shared SAN or multipath storage. The export protects the pool from being imported twice.
261
+
262
+
The setting takes effect on Talos v1.13.4 and later, where the ZFS extension reads `ZFS_EXPORT_TIMEOUT`. Talos v1.13.0 to v1.13.3 always export, so on those versions the document does nothing: move to v1.13.4 or later, or bring DRBD down before every reboot as described below. Earlier versions do not export at all.
263
+
264
+
To keep the export with your own timeout, set `zfs.exportOnShutdown: true` and add your own `zfs-service` document. A `ZFS_EXPORT_TIMEOUT` placed in a node file is appended next to the preset's entry instead of replacing it.
265
+
266
+
Node files written by `talm template` carry the rendered `zfs-service` document, and `talm apply` merges it back over the chart. After switching to `true`, remove that document from every node file, or regenerate the files, otherwise the export stays disabled.
267
+
268
+
##### One-time reboot on a running cluster
269
+
270
+
An existing project gets the document only after its preset templates are updated: run `talm init --update --preset cozystack` after upgrading the talm binary, review the template diff, then`talm apply` each node. No `values.yaml` change is needed, since a missing `zfs` key means the default.
271
+
272
+
The running `zfs-service` keeps the environment it was started with. On a node currently running Talos v1.13.0 or later, the first reboot after the change, including the one inside `talm upgrade`, still runs the export and can hang. Nodes on older Talos do not export, so their first reboot is safe. For that one reboot, handle each affected node in turn:
A `drbd*` holder means a resource is still up, usually one still Primary on this node. Find it with `drbdsetup status`in the same pod and repeat step 2 once nothing uses it.
293
+
294
+
A `dm-*` holder is usually the dm-crypt mapping of a resource with the LINSTOR LUKS layer, which stays open after DRBD goes down. Close it by name, with `dm-N` replaced by the device the check printed:
295
+
296
+
```bash
297
+
kubectl exec -n cozy-linstor ds/linstor-satellite.<node> -- sh -c 'cryptsetup close "$(cat /sys/block/dm-N/dm/name)"'
298
+
```
299
+
300
+
4. Reboot the node right away, or run `talm upgrade`for it.
301
+
302
+
5. Once the node is back and `Ready`, return it to scheduling:
303
+
304
+
```bash
305
+
kubectl uncordon <node>
306
+
```
307
+
308
+
From the next boot on, the setting is in effect and reboots need no manual steps. Clusters installed with the document already in their config are not affected.
309
+
250
310
#### Describing the node's network and registries from values (Talm v0.34+)
251
311
252
312
The `extra*` keys above add to the preset's curated defaults. A second group of keys describes parts of the config the preset used to leave to the template — so a topology that once required forking `templates/_helpers.tpl` is now expressible in `values.yaml`. Both presets accept all of them.
0 commit comments