Skip to content

Commit 95c63fc

Browse files
committed
fix(cozystack): skip ZFS pool export on shutdown by default
zfs-service runs `zpool export -a` when it stops. While DRBD holds the zvols the export blocks in the kernel in D-state, the extension's own timeout cannot kill it, and Talos waits for the service with no timeout, so a LINSTOR-on-ZFS node hangs in `rebooting` on every reboot, upgrade and reset until it is power-cycled. The preset now emits an ExtensionServiceConfig setting ZFS_EXPORT_TIMEOUT=0, which makes zfs-service skip the export; the pool is re-imported with `zpool import -f` on the next boot. The export only matters when the pool sits on dm-crypt devices or can be imported by several hosts, so zfs.exportOnShutdown: true drops the document for those setups. The chart has no values schema, so a non-boolean exportOnShutdown or a non-mapping zfs fails the render: template truthiness would read a quoted "false" as true and silently drop the document. The document is emitted on the legacy schema as well: Talos loaded ExtensionServiceConfig next to v1alpha1 long before v1.12. The legacy single-document contract is narrowed accordingly to what it guards: the machine config stays in one v1alpha1 document. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent dac1d2c commit 95c63fc

10 files changed

Lines changed: 269 additions & 14 deletions

‎charts/cozystack/templates/_helpers.tpl‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,12 +443,35 @@ cluster:
443443
{{- end }}
444444
{{- end }}
445445

446+
{{- define "talos.config.zfs" }}
447+
{{- $zfs := .Values.zfs }}
448+
{{- if kindIs "invalid" $zfs }}
449+
{{- $zfs = dict }}
450+
{{- end }}
451+
{{- if not (kindIs "map" $zfs) }}
452+
{{- fail (printf "talm: zfs must be a mapping (got %s). Example: zfs: { exportOnShutdown: false }." (kindOf $zfs)) }}
453+
{{- end }}
454+
{{- $export := $zfs.exportOnShutdown }}
455+
{{- if and (not (kindIs "invalid" $export)) (not (kindIs "bool" $export)) }}
456+
{{- fail (printf "talm: zfs.exportOnShutdown must be true or false (got %s %q). This field decides whether a reboot can hang on the pool export, so an ambiguous YAML scalar such as a quoted \"false\" is refused." (kindOf $export) (printf "%v" $export)) }}
457+
{{- end }}
458+
{{- if not $export }}
459+
---
460+
apiVersion: v1alpha1
461+
kind: ExtensionServiceConfig
462+
name: zfs-service
463+
environment:
464+
- ZFS_EXPORT_TIMEOUT=0
465+
{{- end }}
466+
{{- end }}
467+
446468
{{- define "talos.config.legacy" }}
447469
{{- include "talos.config.machine.common" . }}
448470
{{- include "talm.config.registries.legacy" . }}
449471
{{- include "talos.config.network.legacy" . }}
450472

451473
{{- include "talos.config.cluster" . }}
474+
{{- include "talos.config.zfs" . }}
452475
{{- end }}
453476

454477
{{- define "talos.config.multidoc" }}
@@ -457,4 +480,5 @@ cluster:
457480
{{- include "talos.config.cluster" . }}
458481
{{- include "talm.config.registries.multidoc" . }}
459482
{{- include "talos.config.network.multidoc" . }}
483+
{{- include "talos.config.zfs" . }}
460484
{{- end }}

‎charts/cozystack/values.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,32 @@ allocateNodeCIDRs: true
115115
# detection is worth the shorter idle timeout.
116116
tcpKeepaliveTuning: false
117117

118+
zfs:
119+
# Whether zfs-service runs `zpool export -a` on shutdown. Off by default:
120+
# while DRBD holds the zvols the export blocks in the kernel where SIGKILL
121+
# cannot end it, the service never stops, and the node hangs in `rebooting`
122+
# on every reboot, upgrade and reset. With the export off the chart emits an
123+
# ExtensionServiceConfig setting ZFS_EXPORT_TIMEOUT=0, and pools are
124+
# re-imported with `zpool import -f` on the next boot. Set true only when
125+
# the pool sits on dm-crypt/LUKS devices (the mapping cannot close while
126+
# the pool holds it) or when several hosts can import the pool. LUKS on top
127+
# of zvols and native ZFS encryption keep raw disks under the pool, so they
128+
# work with the default.
129+
#
130+
# Talos v1.13.0 added the export and v1.13.4 the variable, so on
131+
# v1.13.0-v1.13.3 the document has no effect and the hang remains.
132+
#
133+
# A running zfs-service keeps the environment it started with: the first
134+
# reboot after the document is applied, including the one inside
135+
# `talm upgrade`, still exports. For that one reboot take DRBD down by hand
136+
# (drain, `drbdadm down all` in the node's linstor-satellite, check
137+
# /sys/block/zd*/holders is empty, reboot); later reboots need nothing.
138+
#
139+
# To pick a non-zero timeout, set this to true and ship your own
140+
# zfs-service document: a ZFS_EXPORT_TIMEOUT in the node body is appended
141+
# next to the chart's entry instead of replacing it.
142+
exportOnShutdown: false
143+
118144
# Control-plane etcd tuning. Emitted only on controlplane nodes.
119145
etcd:
120146
# Max etcd backend DB size before etcd raises a NOSPACE alarm and

‎docs/configuration/talos-versions.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ A project created before those pins existed carries empty values for this key an
3939

4040
Selected via `Chart.yaml` (`templateOptions.talosVersion`) or `--talos-version`:
4141

42-
- **Talos < v1.12** — single YAML document with `machine.network` and `machine.registries` sections (the shape shown in [Node files](node-files.md)).
42+
- **Talos < v1.12** — the machine config is a single v1alpha1 document with `machine.network` and `machine.registries` sections (the shape shown in [Node files](node-files.md)). Standalone documents that older Talos already accepts, such as the cozystack preset's `ExtensionServiceConfig`, follow it.
4343
- **Talos >= v1.12** — multi-document format with separate typed documents instead of the deprecated monolithic network and registry fields.
4444
- **Talos >= v1.14** — a second round of the same move, this time for the Kubernetes settings: `machine.kubelet`, `machine.nodeLabels` and the control-plane components each get a document of their own, and a config carrying both shapes is rejected. The shipped charts still write the v1alpha1 fields, which is why the presets pin below this.
4545

@@ -54,6 +54,7 @@ For v1.12+ multi-doc output, one document is emitted per configurable link on th
5454
- `BridgeConfig` — bridges, symmetric to `BondConfig` for bonds. Ports discovered via `spec.slaveKind == "bridge"` + `spec.masterIndex`; STP / VLAN-filtering settings reach the output when the bridge controller reports them on `spec.bridgeMaster`.
5555
- `Layer2VIPConfig` — one per VIP. The `floatingIP` shorthand emits it on controlplane nodes; each `vips` entry emits one on any node, so a storage VIP works on a worker.
5656
- `RegistryMirrorConfig` and `RegistryTLSConfig` — from the `registryMirrors` and `registryTLS` values, available on both charts.
57+
- `ExtensionServiceConfig` for `zfs-service` — cozystack chart only, emitted on the legacy format too. It sets `ZFS_EXPORT_TIMEOUT=0` so the ZFS extension skips the pool export on shutdown, and is dropped by `zfs.exportOnShutdown: true`.
5758

5859
### Per-link emission rules
5960

‎docs/manual-test-plan.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -506,6 +506,22 @@ Collision check: with `tcpKeepaliveTuning: true`, set `extraSysctls: { "net.ipv4
506506

507507
Regression anchor: `TestContract_Machine_Sysctls_DRBDTuning_Cozystack`, `TestContract_Machine_Sysctls_TCPKeepalive_*`, and `TestContract_Cluster_Etcd_QuotaBackendBytes_*` pin every branch above; `TestContract_Machine_Sysctls_DRBDTuning_AbsentOnGeneric` / `TestContract_Cluster_Etcd_QuotaBackendBytes_AbsentOnGeneric` pin that the generic preset stays free of these opinions.
508508

509+
### B9a. ZFS pool export on shutdown (cozystack)
510+
511+
Render the cozystack preset at defaults, once with a `talosVersion` below v1.12 and once at v1.12 or later:
512+
513+
```bash
514+
talm template -f nodes/controlplane-0.yaml | yq 'select(.kind == "ExtensionServiceConfig")'
515+
```
516+
517+
Expected on both schemas: one document with `name: zfs-service` and `environment: [ZFS_EXPORT_TIMEOUT=0]`.
518+
519+
Set `zfs: { exportOnShutdown: true }` in `values.yaml` and re-render. Expected: the same query prints nothing.
520+
521+
On a live node with the document applied, run one reboot first: a running `ext-zfs-service` keeps the environment it started with, so this reboot still exports and hangs while DRBD holds the zvols. Before it, take DRBD down as the `zfs` comment in `charts/cozystack/values.yaml` describes: drain the node, run `drbdadm down all` in the node's linstor-satellite, and check that `/sys/block/zd*/holders` is empty. After that reboot, `talosctl read /proc/<ext-zfs-service pid>/environ` contains `ZFS_EXPORT_TIMEOUT=0`. With DRBD holding the zvols (`/sys/block/zd*/holders` lists `drbd*`), drain the node and run `talosctl reboot`. Expected: the node comes back within minutes instead of staying in `rebooting`, the pool is imported, and `linstor resource list` shows the node's resources `UpToDate`.
522+
523+
Regression anchor: `TestContract_Machine_ZFSExport_*` pins the default, the opt-out, the refusal of a non-boolean `exportOnShutdown` or a non-mapping `zfs`, the generic preset staying free of the document, and a render that loads in machinery after a node body restates the document.
524+
509525
### B10. Values knobs for richer instance description
510526

511527
These knobs let a node's specifics live in values instead of a forked or hand-edited template. All default empty, so a stock render is unchanged — the golden snapshots (`TestGoldenRender`) stay byte-identical and act as the backward-compat guard.

‎pkg/engine/contract_machine_test.go‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,12 @@ package engine
2929

3030
import (
3131
"encoding/base64"
32+
"os"
33+
"path/filepath"
3234
"strings"
3335
"testing"
3436

37+
"github.com/siderolabs/talos/pkg/machinery/config/configloader"
3538
"gopkg.in/yaml.v3"
3639
)
3740

@@ -1340,3 +1343,150 @@ func TestContract_Machine_RegistryTLS_PlainHostStaysBare_Multidoc_Cozystack(t *t
13401343
})
13411344
assertContains(t, out, "name: mirror.example.invalid:5000")
13421345
}
1346+
1347+
// Contract: cozystack turns off the zfs-service shutdown export by
1348+
// default. `zpool export` blocks in D-state while DRBD holds the zvols;
1349+
// SIGKILL cannot end it, the process keeps the service's PID namespace
1350+
// alive, and the node hangs in `rebooting` on every reboot, upgrade and
1351+
// reset.
1352+
func TestContract_Machine_ZFSExport_DisabledByDefault_Cozystack(t *testing.T) {
1353+
for _, cell := range cozystackCells() {
1354+
t.Run(cell.name, func(t *testing.T) {
1355+
out := renderChartTemplate(t, cell.chartPath, cell.templateFile, cell.talosVersion)
1356+
assertContains(t, out, "kind: ExtensionServiceConfig")
1357+
assertContains(t, out, "name: zfs-service")
1358+
assertContains(t, out, "ZFS_EXPORT_TIMEOUT=0")
1359+
})
1360+
}
1361+
}
1362+
1363+
// Contract: zfs.exportOnShutdown: true drops the document, restoring the
1364+
// extension's own export for pools that sit on dm-crypt devices.
1365+
func TestContract_Machine_ZFSExport_DocumentAbsentWhenEnabled_Cozystack(t *testing.T) {
1366+
out := renderCozystackWith(t, helmEngineEmptyLookup, map[string]any{
1367+
"advertisedSubnets": []any{testAdvertisedSubnet},
1368+
"zfs": map[string]any{"exportOnShutdown": true},
1369+
})
1370+
assertNotContains(t, out, "ExtensionServiceConfig")
1371+
assertNotContains(t, out, "ZFS_EXPORT_TIMEOUT")
1372+
}
1373+
1374+
func TestContract_Machine_ZFSExport_AbsentOnGeneric(t *testing.T) {
1375+
for _, cell := range genericCells() {
1376+
t.Run(cell.name, func(t *testing.T) {
1377+
out := renderChartTemplate(t, cell.chartPath, cell.templateFile, cell.talosVersion)
1378+
assertNotContains(t, out, "ExtensionServiceConfig")
1379+
})
1380+
}
1381+
}
1382+
1383+
// Contract: the document survives the apply pipeline on both schemas and
1384+
// the result loads in machinery. A node body that restates the document,
1385+
// as one written by `talm template -I` does, must not produce a second
1386+
// one: Talos rejects duplicate identities.
1387+
func TestContract_Machine_ZFSExport_LoadsThroughApplyPath_Cozystack(t *testing.T) {
1388+
const nodeBody = `apiVersion: v1alpha1
1389+
kind: ExtensionServiceConfig
1390+
name: zfs-service
1391+
environment:
1392+
- ZFS_EXPORT_TIMEOUT=0
1393+
`
1394+
1395+
for _, cell := range cozystackControlplaneCells() {
1396+
contract := cell.talosVersion
1397+
if contract == "" {
1398+
contract = "v1.11"
1399+
}
1400+
1401+
t.Run(cell.name, func(t *testing.T) {
1402+
chart := renderChartTemplate(t, cell.chartPath, cell.templateFile, cell.talosVersion)
1403+
1404+
rendered, err := applyPatchesAndRenderConfig(
1405+
Options{KubernetesVersion: "v1.34.3", TalosVersion: contract, Full: true},
1406+
[]string{chart})
1407+
if err != nil {
1408+
t.Fatalf("applyPatchesAndRenderConfig: %v", err)
1409+
}
1410+
1411+
nonFull, err := applyPatchesAndRenderConfig(
1412+
Options{KubernetesVersion: "v1.34.3", TalosVersion: contract},
1413+
[]string{chart})
1414+
if err != nil {
1415+
t.Fatalf("applyPatchesAndRenderConfig (non-full): %v", err)
1416+
}
1417+
1418+
bodyFile := filepath.Join(t.TempDir(), "node.yaml")
1419+
if err := os.WriteFile(bodyFile, []byte(nodeBody), 0o600); err != nil {
1420+
t.Fatal(err)
1421+
}
1422+
1423+
merged, err := MergeFileAsPatch(rendered, bodyFile)
1424+
if err != nil {
1425+
t.Fatalf("MergeFileAsPatch: %v", err)
1426+
}
1427+
1428+
for name, out := range map[string][]byte{"rendered": rendered, "non-full": nonFull, "merged": merged} {
1429+
if got := strings.Count(string(out), "kind: ExtensionServiceConfig"); got != 1 {
1430+
t.Errorf("%s config carries %d ExtensionServiceConfig documents, want 1\n%s", name, got, out)
1431+
}
1432+
}
1433+
1434+
for name, out := range map[string][]byte{"rendered": rendered, "merged": merged} {
1435+
if _, err := configloader.NewFromBytes(out); err != nil {
1436+
t.Errorf("Talos refuses the %s config: %v", name, err)
1437+
}
1438+
}
1439+
})
1440+
}
1441+
}
1442+
1443+
// Contract: the chart has no values schema, so a non-boolean
1444+
// exportOnShutdown is refused instead of read by template truthiness,
1445+
// where a quoted "false" is truthy and would drop the document.
1446+
func TestContract_Machine_ZFSExport_NonBoolFails_Cozystack(t *testing.T) {
1447+
for name, v := range map[string]any{"quoted false": "false", "quoted true": "true", "number": 0} {
1448+
t.Run(name, func(t *testing.T) {
1449+
err := renderCozystackExpectError(t, helmEngineEmptyLookup, map[string]any{
1450+
"advertisedSubnets": []any{testAdvertisedSubnet},
1451+
"zfs": map[string]any{"exportOnShutdown": v},
1452+
})
1453+
if err == nil {
1454+
t.Fatalf("expected a fail-fast for exportOnShutdown %#v", v)
1455+
}
1456+
if !strings.Contains(err.Error(), "zfs.exportOnShutdown must be true or false") {
1457+
t.Errorf("error should name the field and the accepted values, got %v", err)
1458+
}
1459+
})
1460+
}
1461+
}
1462+
1463+
func TestContract_Machine_ZFSExport_NonMapZFSFails_Cozystack(t *testing.T) {
1464+
for name, v := range map[string]any{"string": "foo", "false": false, "list": []any{}} {
1465+
t.Run(name, func(t *testing.T) {
1466+
err := renderCozystackExpectError(t, helmEngineEmptyLookup, map[string]any{
1467+
"advertisedSubnets": []any{testAdvertisedSubnet},
1468+
"zfs": v,
1469+
})
1470+
if err == nil {
1471+
t.Fatalf("expected a fail-fast for zfs %#v", v)
1472+
}
1473+
if !strings.Contains(err.Error(), "zfs must be a mapping") {
1474+
t.Errorf("error should explain zfs must be a mapping, got %v", err)
1475+
}
1476+
})
1477+
}
1478+
}
1479+
1480+
// Contract: an unset knob keeps the default, whether zfs is absent,
1481+
// empty, or left as `zfs:` with nothing after it.
1482+
func TestContract_Machine_ZFSExport_UnsetKeepsDefault_Cozystack(t *testing.T) {
1483+
for name, v := range map[string]any{"nil": nil, "empty map": map[string]any{}, "nil knob": map[string]any{"exportOnShutdown": nil}} {
1484+
t.Run(name, func(t *testing.T) {
1485+
out := renderCozystackWith(t, helmEngineEmptyLookup, map[string]any{
1486+
"advertisedSubnets": []any{testAdvertisedSubnet},
1487+
"zfs": v,
1488+
})
1489+
assertContains(t, out, "ZFS_EXPORT_TIMEOUT=0")
1490+
})
1491+
}
1492+
}

‎pkg/engine/contract_schema_test.go‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -133,23 +133,37 @@ func TestContract_Schema_Version112PreReleasesRenderMultidoc(t *testing.T) {
133133
}
134134
}
135135

136-
// Contract: legacy schema produces a single YAML document — no `---`
137-
// separators in the body. Talos's legacy parser expects exactly one
138-
// document. (Helm always prepends one leading newline; we check that
139-
// no internal separator appears.)
140-
func TestContract_Schema_LegacyIsSingleDocument(t *testing.T) {
136+
// Contract: the legacy schema keeps the machine config in one v1alpha1
137+
// document. The only documents allowed after it are standalone kinds
138+
// that pre-v1.12 Talos already loads next to v1alpha1.
139+
func TestContract_Schema_LegacyKeepsSingleV1alpha1Document(t *testing.T) {
140+
legacyStandaloneKinds := map[string]bool{"ExtensionServiceConfig": true}
141+
141142
for _, chartPath := range []string{cozystackChartPath, genericChartPath} {
142143
t.Run(chartPath, func(t *testing.T) {
143144
out := renderChartTemplate(t, chartPath, controlplaneTpl)
144-
// Match a `---` token surrounded by ANY newline form
145-
// (\n or \r\n) — Windows-rendered output uses CRLF and
146-
// pinning `\n---\n` would falsely pass on Windows.
147-
// Scan line-by-line: a single line that is exactly `---`
148-
// means an internal document separator.
145+
146+
// Split line by line on an exact `---`: CRLF output from a
147+
// Windows render would slip past a `\n---\n` match.
148+
docs := [][]string{nil}
149149
for line := range strings.SplitSeq(out, "\n") {
150-
if strings.TrimRight(line, "\r") == "---" {
151-
t.Errorf("legacy render must not contain `---` separator:\n%s", out)
152-
break
150+
line = strings.TrimRight(line, "\r")
151+
if line == "---" {
152+
docs = append(docs, nil)
153+
continue
154+
}
155+
docs[len(docs)-1] = append(docs[len(docs)-1], line)
156+
}
157+
158+
for i, doc := range docs[1:] {
159+
kind := ""
160+
for _, line := range doc {
161+
if k, ok := strings.CutPrefix(line, "kind: "); ok {
162+
kind = k
163+
}
164+
}
165+
if !legacyStandaloneKinds[kind] {
166+
t.Errorf("legacy render document %d has kind %q; only the v1alpha1 document and %v may appear:\n%s", i+2, kind, legacyStandaloneKinds, out)
153167
}
154168
}
155169
})

‎pkg/engine/testdata/golden/cozystack-controlplane-legacy.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/engine/testdata/golden/cozystack-controlplane-multidoc.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/engine/testdata/golden/cozystack-worker-legacy.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/engine/testdata/golden/cozystack-worker-multidoc.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)