Skip to content

Commit 8e117a5

Browse files
committed
fix(cozystack): skip ZFS pool export on shutdown by default
zfs-service runs `zpool export -a` when it stops. While DRBD holds the zvols the export blocks in the kernel in D-state, the extension's own timeout cannot kill it, and Talos waits for the service with no timeout, so a LINSTOR-on-ZFS node hangs in `rebooting` on every reboot, upgrade and reset until it is power-cycled. The preset now emits an ExtensionServiceConfig setting ZFS_EXPORT_TIMEOUT=0, which makes zfs-service skip the export; the pool is re-imported with `zpool import -f` on the next boot. The export only matters when the pool sits on dm-crypt devices or can be imported by several hosts, so zfs.exportOnShutdown: true drops the document for those setups. The document is emitted on the legacy schema as well: Talos loaded ExtensionServiceConfig next to v1alpha1 long before v1.12. The legacy single-document contract is narrowed accordingly to what it guards: the machine config stays in one v1alpha1 document. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent dac1d2c commit 8e117a5

9 files changed

Lines changed: 205 additions & 13 deletions

‎charts/cozystack/templates/_helpers.tpl‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,12 +443,24 @@ cluster:
443443
{{- end }}
444444
{{- end }}
445445

446+
{{- define "talos.config.zfs" }}
447+
{{- if not (dig "zfs" "exportOnShutdown" false .Values) }}
448+
---
449+
apiVersion: v1alpha1
450+
kind: ExtensionServiceConfig
451+
name: zfs-service
452+
environment:
453+
- ZFS_EXPORT_TIMEOUT=0
454+
{{- end }}
455+
{{- end }}
456+
446457
{{- define "talos.config.legacy" }}
447458
{{- include "talos.config.machine.common" . }}
448459
{{- include "talm.config.registries.legacy" . }}
449460
{{- include "talos.config.network.legacy" . }}
450461

451462
{{- include "talos.config.cluster" . }}
463+
{{- include "talos.config.zfs" . }}
452464
{{- end }}
453465

454466
{{- define "talos.config.multidoc" }}
@@ -457,4 +469,5 @@ cluster:
457469
{{- include "talos.config.cluster" . }}
458470
{{- include "talm.config.registries.multidoc" . }}
459471
{{- include "talos.config.network.multidoc" . }}
472+
{{- include "talos.config.zfs" . }}
460473
{{- end }}

‎charts/cozystack/values.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,32 @@ allocateNodeCIDRs: true
115115
# detection is worth the shorter idle timeout.
116116
tcpKeepaliveTuning: false
117117

118+
zfs:
119+
# Whether zfs-service runs `zpool export -a` on shutdown. Off by default:
120+
# while DRBD holds the zvols the export blocks in the kernel where SIGKILL
121+
# cannot end it, the service never stops, and the node hangs in `rebooting`
122+
# on every reboot, upgrade and reset. With the export off the chart emits an
123+
# ExtensionServiceConfig setting ZFS_EXPORT_TIMEOUT=0, and pools are
124+
# re-imported with `zpool import -f` on the next boot. Set true only when
125+
# the pool sits on dm-crypt/LUKS devices (the mapping cannot close while
126+
# the pool holds it) or when several hosts can import the pool. LUKS on top
127+
# of zvols and native ZFS encryption keep raw disks under the pool, so they
128+
# work with the default.
129+
#
130+
# Talos v1.13.0 added the export and v1.13.4 the variable, so on
131+
# v1.13.0-v1.13.3 the document has no effect and the hang remains.
132+
#
133+
# A running zfs-service keeps the environment it started with: the first
134+
# reboot after the document is applied, including the one inside
135+
# `talm upgrade`, still exports. For that one reboot take DRBD down by hand
136+
# (drain, stop linstor-satellite on the node, `drbdsetup down all`, check
137+
# /sys/block/zd*/holders is empty); later reboots need nothing.
138+
#
139+
# To pick a non-zero timeout, set this to true and ship your own
140+
# zfs-service document: a ZFS_EXPORT_TIMEOUT in the node body is appended
141+
# next to the chart's entry instead of replacing it.
142+
exportOnShutdown: false
143+
118144
# Control-plane etcd tuning. Emitted only on controlplane nodes.
119145
etcd:
120146
# Max etcd backend DB size before etcd raises a NOSPACE alarm and

‎docs/manual-test-plan.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -506,6 +506,22 @@ Collision check: with `tcpKeepaliveTuning: true`, set `extraSysctls: { "net.ipv4
506506

507507
Regression anchor: `TestContract_Machine_Sysctls_DRBDTuning_Cozystack`, `TestContract_Machine_Sysctls_TCPKeepalive_*`, and `TestContract_Cluster_Etcd_QuotaBackendBytes_*` pin every branch above; `TestContract_Machine_Sysctls_DRBDTuning_AbsentOnGeneric` / `TestContract_Cluster_Etcd_QuotaBackendBytes_AbsentOnGeneric` pin that the generic preset stays free of these opinions.
508508

509+
### B9a. ZFS pool export on shutdown (cozystack)
510+
511+
Render the cozystack preset at defaults, once with a `talosVersion` below v1.12 and once at v1.12 or later:
512+
513+
```bash
514+
talm template -f nodes/controlplane-0.yaml | yq 'select(.kind == "ExtensionServiceConfig")'
515+
```
516+
517+
Expected on both schemas: one document with `name: zfs-service` and `environment: [ZFS_EXPORT_TIMEOUT=0]`.
518+
519+
Set `zfs: { exportOnShutdown: true }` in `values.yaml` and re-render. Expected: the same query prints nothing.
520+
521+
On a live node with the document applied, run one reboot first: a running `ext-zfs-service` keeps the environment it started with. After that reboot, `talosctl read /proc/<ext-zfs-service pid>/environ` contains `ZFS_EXPORT_TIMEOUT=0`. With DRBD holding the zvols (`/sys/block/zd*/holders` lists `drbd*`), drain the node and run `talosctl reboot`. Expected: the node comes back within minutes instead of staying in `rebooting`, the pool is imported, and `linstor resource list` shows the node's resources `UpToDate`.
522+
523+
Regression anchor: `TestContract_Machine_ZFSExport_*` pins the default, the opt-out, the generic preset staying free of the document, and a render that loads in machinery after a node body restates the document.
524+
509525
### B10. Values knobs for richer instance description
510526

511527
These knobs let a node's specifics live in values instead of a forked or hand-edited template. All default empty, so a stock render is unchanged — the golden snapshots (`TestGoldenRender`) stay byte-identical and act as the backward-compat guard.

‎pkg/engine/contract_machine_test.go‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,12 @@ package engine
2929

3030
import (
3131
"encoding/base64"
32+
"os"
33+
"path/filepath"
3234
"strings"
3335
"testing"
3436

37+
"github.com/siderolabs/talos/pkg/machinery/config/configloader"
3538
"gopkg.in/yaml.v3"
3639
)
3740

@@ -1340,3 +1343,99 @@ func TestContract_Machine_RegistryTLS_PlainHostStaysBare_Multidoc_Cozystack(t *t
13401343
})
13411344
assertContains(t, out, "name: mirror.example.invalid:5000")
13421345
}
1346+
1347+
// Contract: cozystack turns off the zfs-service shutdown export by
1348+
// default. `zpool export` blocks in D-state while DRBD holds the zvols;
1349+
// SIGKILL cannot end it, the process keeps the service's PID namespace
1350+
// alive, and the node hangs in `rebooting` on every reboot, upgrade and
1351+
// reset.
1352+
func TestContract_Machine_ZFSExport_DisabledByDefault_Cozystack(t *testing.T) {
1353+
for _, cell := range cozystackCells() {
1354+
t.Run(cell.name, func(t *testing.T) {
1355+
out := renderChartTemplate(t, cell.chartPath, cell.templateFile, cell.talosVersion)
1356+
assertContains(t, out, "kind: ExtensionServiceConfig")
1357+
assertContains(t, out, "name: zfs-service")
1358+
assertContains(t, out, "ZFS_EXPORT_TIMEOUT=0")
1359+
})
1360+
}
1361+
}
1362+
1363+
// Contract: zfs.exportOnShutdown: true drops the document, restoring the
1364+
// extension's own export for pools that sit on dm-crypt devices.
1365+
func TestContract_Machine_ZFSExport_DocumentAbsentWhenEnabled_Cozystack(t *testing.T) {
1366+
out := renderCozystackWith(t, helmEngineEmptyLookup, map[string]any{
1367+
"advertisedSubnets": []any{testAdvertisedSubnet},
1368+
"zfs": map[string]any{"exportOnShutdown": true},
1369+
})
1370+
assertNotContains(t, out, "ExtensionServiceConfig")
1371+
assertNotContains(t, out, "ZFS_EXPORT_TIMEOUT")
1372+
}
1373+
1374+
func TestContract_Machine_ZFSExport_AbsentOnGeneric(t *testing.T) {
1375+
for _, cell := range genericCells() {
1376+
t.Run(cell.name, func(t *testing.T) {
1377+
out := renderChartTemplate(t, cell.chartPath, cell.templateFile, cell.talosVersion)
1378+
assertNotContains(t, out, "ExtensionServiceConfig")
1379+
})
1380+
}
1381+
}
1382+
1383+
// Contract: the document survives the apply pipeline on both schemas and
1384+
// the result loads in machinery. A node body that restates the document,
1385+
// as one written by `talm template -I` does, must not produce a second
1386+
// one: Talos rejects duplicate identities.
1387+
func TestContract_Machine_ZFSExport_LoadsThroughApplyPath_Cozystack(t *testing.T) {
1388+
const nodeBody = `apiVersion: v1alpha1
1389+
kind: ExtensionServiceConfig
1390+
name: zfs-service
1391+
environment:
1392+
- ZFS_EXPORT_TIMEOUT=0
1393+
`
1394+
1395+
for _, cell := range cozystackControlplaneCells() {
1396+
contract := cell.talosVersion
1397+
if contract == "" {
1398+
contract = "v1.11"
1399+
}
1400+
1401+
t.Run(cell.name, func(t *testing.T) {
1402+
chart := renderChartTemplate(t, cell.chartPath, cell.templateFile, cell.talosVersion)
1403+
1404+
rendered, err := applyPatchesAndRenderConfig(
1405+
Options{KubernetesVersion: "v1.34.3", TalosVersion: contract, Full: true},
1406+
[]string{chart})
1407+
if err != nil {
1408+
t.Fatalf("applyPatchesAndRenderConfig: %v", err)
1409+
}
1410+
1411+
nonFull, err := applyPatchesAndRenderConfig(
1412+
Options{KubernetesVersion: "v1.34.3", TalosVersion: contract},
1413+
[]string{chart})
1414+
if err != nil {
1415+
t.Fatalf("applyPatchesAndRenderConfig (non-full): %v", err)
1416+
}
1417+
1418+
bodyFile := filepath.Join(t.TempDir(), "node.yaml")
1419+
if err := os.WriteFile(bodyFile, []byte(nodeBody), 0o600); err != nil {
1420+
t.Fatal(err)
1421+
}
1422+
1423+
merged, err := MergeFileAsPatch(rendered, bodyFile)
1424+
if err != nil {
1425+
t.Fatalf("MergeFileAsPatch: %v", err)
1426+
}
1427+
1428+
for name, out := range map[string][]byte{"rendered": rendered, "non-full": nonFull, "merged": merged} {
1429+
if got := strings.Count(string(out), "kind: ExtensionServiceConfig"); got != 1 {
1430+
t.Errorf("%s config carries %d ExtensionServiceConfig documents, want 1\n%s", name, got, out)
1431+
}
1432+
}
1433+
1434+
for name, out := range map[string][]byte{"rendered": rendered, "merged": merged} {
1435+
if _, err := configloader.NewFromBytes(out); err != nil {
1436+
t.Errorf("Talos refuses the %s config: %v", name, err)
1437+
}
1438+
}
1439+
})
1440+
}
1441+
}

‎pkg/engine/contract_schema_test.go‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -133,23 +133,37 @@ func TestContract_Schema_Version112PreReleasesRenderMultidoc(t *testing.T) {
133133
}
134134
}
135135

136-
// Contract: legacy schema produces a single YAML document — no `---`
137-
// separators in the body. Talos's legacy parser expects exactly one
138-
// document. (Helm always prepends one leading newline; we check that
139-
// no internal separator appears.)
140-
func TestContract_Schema_LegacyIsSingleDocument(t *testing.T) {
136+
// Contract: the legacy schema keeps the machine config in one v1alpha1
137+
// document. The only documents allowed after it are standalone kinds
138+
// that pre-v1.12 Talos already loads next to v1alpha1.
139+
func TestContract_Schema_LegacyKeepsSingleV1alpha1Document(t *testing.T) {
140+
legacyStandaloneKinds := map[string]bool{"ExtensionServiceConfig": true}
141+
141142
for _, chartPath := range []string{cozystackChartPath, genericChartPath} {
142143
t.Run(chartPath, func(t *testing.T) {
143144
out := renderChartTemplate(t, chartPath, controlplaneTpl)
144-
// Match a `---` token surrounded by ANY newline form
145-
// (\n or \r\n) — Windows-rendered output uses CRLF and
146-
// pinning `\n---\n` would falsely pass on Windows.
147-
// Scan line-by-line: a single line that is exactly `---`
148-
// means an internal document separator.
145+
146+
// Split line by line on an exact `---`: CRLF output from a
147+
// Windows render would slip past a `\n---\n` match.
148+
docs := [][]string{nil}
149149
for line := range strings.SplitSeq(out, "\n") {
150-
if strings.TrimRight(line, "\r") == "---" {
151-
t.Errorf("legacy render must not contain `---` separator:\n%s", out)
152-
break
150+
line = strings.TrimRight(line, "\r")
151+
if line == "---" {
152+
docs = append(docs, nil)
153+
continue
154+
}
155+
docs[len(docs)-1] = append(docs[len(docs)-1], line)
156+
}
157+
158+
for i, doc := range docs[1:] {
159+
kind := ""
160+
for _, line := range doc {
161+
if k, ok := strings.CutPrefix(line, "kind: "); ok {
162+
kind = k
163+
}
164+
}
165+
if !legacyStandaloneKinds[kind] {
166+
t.Errorf("legacy render document %d has kind %q; only the v1alpha1 document and %v may appear:\n%s", i+2, kind, legacyStandaloneKinds, out)
153167
}
154168
}
155169
})

‎pkg/engine/testdata/golden/cozystack-controlplane-legacy.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/engine/testdata/golden/cozystack-controlplane-multidoc.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/engine/testdata/golden/cozystack-worker-legacy.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/engine/testdata/golden/cozystack-worker-multidoc.golden.yaml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)