Skip to content

Commit 35c1fad

Browse files
committed
fix(engine): report the version keys a render can no longer guess
Two keys used to have a working empty value and no longer do, and both failed quietly rather than loudly. An unset talosVersion means the Talos version talm was built against, which is still true. What changed under it is Talos: from v1.14 the Kubernetes settings the charts write as v1alpha1 fields (machine.kubelet, machine.nodeLabels) live in documents of their own, and machinery rejects a config carrying both shapes. An unpinned project therefore rendered a config the node refuses, with nothing said until the apply failed. The render now stops and names the fields and the way out. An unset kubernetesVersion used to mean "the node decides": machinery emitted no image field at all. v1.14 refuses to generate without a version, so the fallback that satisfies it would write this binary's Kubernetes version into the config instead — a version jump nobody asked for, invisible in the node file because the render's diff drops fields equal to the bundle default. On contracts where the node can still choose, the generated images are stripped again; on contracts where the typed documents require an image, the missing pin is reported. The generic preset gains the pins it now needs — cozystack already had them — and both errors carry a hint naming the key. A project created before this renders again once its Chart.yaml pins templateOptions.talosVersion and templateOptions.kubernetesVersion. Add the contract test that would have caught the first one: nothing in the suite loaded a rendered config back through machinery, so a render could stop being applicable while the whole suite stayed green. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-by: LLM
1 parent aaab350 commit 35c1fad

28 files changed

Lines changed: 1186 additions & 92 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ Full documentation lives at **[talm.cozystack.io](https://talm.cozystack.io/)**.
5050
- [Initializing a project](https://talm.cozystack.io/getting-started/init/) — `--image`, `--cluster-endpoint`, `--root`.
5151
- [Node files](https://talm.cozystack.io/configuration/node-files/) — the modeline, discovery comments, per-node overrides.
5252
- [Endpoints and VIPs](https://talm.cozystack.io/configuration/endpoints-and-vips/) — `endpoint`, `floatingIP`, `vipLink`.
53-
- [Talos versions and output format](https://talm.cozystack.io/configuration/talos-versions/) — the v1.12 multi-document split.
53+
- [Talos versions and output format](https://talm.cozystack.io/configuration/talos-versions/) — which schema a render targets, and the version keys to pin.
5454
- [Templates and values](https://talm.cozystack.io/configuration/templates/) — `lookup`, `--set` vs `--set-string`.
5555
- [Encryption](https://talm.cozystack.io/configuration/encryption/) — age-encrypted secrets and user values.
5656
- [Applying with side-patches](https://talm.cozystack.io/operations/side-patches/) — the `-f` chain.

‎charts/generic/Chart.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,14 @@ templateOptions:
1212
fileValues: []
1313
jsonValues: []
1414
literalValues: []
15-
talosVersion: ""
15+
# Pin both: an empty key either renders what the node picks (with a warning)
16+
# or stops the render on newer contracts. The Kubernetes version is the
17+
# default of the pinned Talos line, so a fresh project lands on what that
18+
# line ships and a re-sync does not move the control plane. Replace it with
19+
# the version your cluster runs.
20+
talosVersion: "v1.13"
1621
withSecrets: "secrets.yaml"
17-
kubernetesVersion: ""
22+
kubernetesVersion: "v1.36.2"
1823
full: false
1924
applyOptions:
2025
preserve: false

‎docs/configuration/talos-versions.md‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,46 @@
22

33
`templateOptions.talosVersion` in `Chart.yaml` (or `--talos-version` on the command line) selects which Talos config schema talm renders. It changes the shape of every node file.
44

5+
## What an unset version means
6+
7+
Leaving `templateOptions.talosVersion` empty means the version of Talos the `talm` binary was built against. It is not frozen at a release: upgrade talm, and an unpinned project follows it to whatever Talos that build carries.
8+
9+
That is why the shipped presets pin it. The charts write Kubernetes settings as v1alpha1 fields (`machine.kubelet`, `machine.nodeLabels`), and from v1.14 Talos keeps each of those in a document of its own and rejects a config that carries both shapes. Rendering an unpinned project against v1.14 therefore produces a config the node refuses, with `kubelet config is already set in v1alpha1 config (.machine.kubelet)` and siblings. Network settings made the same move earlier, in v1.12.
10+
11+
Rather than emit that config, the render stops. With both keys empty the Kubernetes version is reported first, since the bundle cannot be serialized without one; pin it and the contract conflict is reported next:
12+
13+
```text
14+
rendered config mixes v1alpha1 fields with the documents that superseded them: ...
15+
hint: templateOptions.talosVersion is unset, so the render targets the Talos version talm was built from.
16+
Pin it to v1.13 or lower in Chart.yaml; above that contract Talos keeps these settings in documents of their own,
17+
which the charts do not emit.
18+
```
19+
20+
A chart can also fail earlier than either guard: the cozystack preset deletes a default node label with `$patch: delete`, and on a contract where `machine.nodeLabels` has moved into `KubeNodeConfig` that path no longer exists, so patching stops with `failed to delete path ...: lookup failed` before any version check runs. The preset happens to pin `v1.12`, which predates that move and sidesteps it.
21+
22+
Pin `templateOptions.talosVersion` in `Chart.yaml`. The rule below still holds — the contract must not be newer than the Talos running on the node — and until the charts emit the typed documents there is a second bound on top of it: keep the pin at or below `v1.13`. Nodes running v1.14 are served by a v1.13 contract, which is the supported direction; the reverse is not. Projects created from the shipped presets already carry a pin.
23+
24+
## The Kubernetes version
25+
26+
`templateOptions.kubernetesVersion` picks the Kubernetes release the generated config targets: the kubelet image and, on contracts that keep them in v1alpha1, the control-plane component images.
27+
28+
Leaving it empty means the node keeps deciding, on contracts where it still can: the render emits no image fields at all, and each component uses the default of the Talos release it runs. The render says so on stderr, because that is a real choice and not a no-op — a node on a newer Talos release moves kubelet and the control plane with it, and the node file shows nothing either way. This is new behaviour, not a restored default: earlier talm substituted its own built-in Kubernetes version whenever the key was empty, so an unpinned project silently followed the binary. It no longer does.
29+
30+
From v1.14 the contract takes that choice away. The Kubernetes settings live in documents of their own and those documents require an image, so an unset version is reported rather than rendered — pin the key.
31+
32+
Pin it in `Chart.yaml` when you want to choose the version yourself, which is what both shipped presets do. A pinned version reaches every component, kubelet included, so they never split across releases.
33+
34+
The presets do not agree on a value. Generic pins `v1.36.2`, the default of the Talos line it targets, so a fresh project lands on what that line ships and a re-sync does not move the control plane. Cozystack pins `v1.34.3` against an older Talos line. Either way the pin is a starting point: set the key to the version your own cluster runs rather than inheriting the preset's.
35+
36+
A project created before those pins existed carries empty values for this key and for `talosVersion`. On the first render after upgrading talm it stops with an error naming the key to set; add both pins to its `Chart.yaml` and it renders again.
37+
538
## Which format each version gets
639

740
Selected via `Chart.yaml` (`templateOptions.talosVersion`) or `--talos-version`:
841

942
- **Talos < v1.12** — single YAML document with `machine.network` and `machine.registries` sections (the shape shown in [Node files](node-files.md)).
10-
- **Talos >= v1.12** — multi-document format with separate typed documents instead of the deprecated monolithic fields.
43+
- **Talos >= v1.12** — multi-document format with separate typed documents instead of the deprecated monolithic network and registry fields.
44+
- **Talos >= v1.14** — a second round of the same move, this time for the Kubernetes settings: `machine.kubelet`, `machine.nodeLabels` and the control-plane components each get a document of their own, and a config carrying both shapes is rejected. The shipped charts still write the v1alpha1 fields, which is why the presets pin below this.
1145

1246
## Documents emitted in multi-doc mode
1347

@@ -39,6 +73,6 @@ Multi-NIC nodes therefore produce one document per NIC, not one document total.
3973

4074
## Version compatibility
4175

42-
!!! danger "The configured version must match the Talos actually running on the node"
76+
!!! danger "The configured version must not be newer than the Talos running on the node"
4377

44-
This setting must match the **Talos version actually running on the target node** — i.e. the maintenance ISO/PXE the node booted from for `apply -i`, or the installed Talos for an authenticated apply. It is **not** the same as `install.image`, which only controls what gets written to disk after a successful apply. When the configured contract is newer than the running binary, machinery injects fields (e.g. `machine.install.grubUseUKICmdline` from v1.12) that the running parser does not know, and the apply fails on the node side with `failed to parse config: unknown keys found during decoding: ...`. `talm apply` runs a best-effort pre-flight check against the running version and prints a `warning: pre-flight: ...` line with a hint when it detects this mismatch; if the warning is missed, the same hint is appended to the apply error. Either reboot the node into a maintenance image that matches the configured contract, or lower `templateOptions.talosVersion` / `--talos-version` to match what is running.
78+
This setting must not exceed the **Talos version actually running on the target node** — i.e. the maintenance ISO/PXE the node booted from for `apply -i`, or the installed Talos for an authenticated apply. It is **not** the same as `install.image`, which only controls what gets written to disk after a successful apply. When the configured contract is newer than the running binary, machinery injects fields (e.g. `machine.install.grubUseUKICmdline` from v1.12) that the running parser does not know, and the apply fails on the node side with `failed to parse config: unknown keys found during decoding: ...`. `talm apply` runs a best-effort pre-flight check against the running version and prints a `warning: pre-flight: ...` line with a hint when it detects this mismatch; if the warning is missed, the same hint is appended to the apply error. Either reboot the node into a maintenance image that matches the configured contract, or lower `templateOptions.talosVersion` / `--talos-version` to match what is running.

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,6 @@ talm apply -f nodes/node1.yaml -i
8282

8383
- [Initializing a project](getting-started/init.md) — `--image`, `--cluster-endpoint`, `--root`.
8484
- [Endpoints and VIPs](configuration/endpoints-and-vips.md) — `endpoint`, `floatingIP`, `vipLink`.
85-
- [Talos versions and output format](configuration/talos-versions.md) — the v1.12 multi-document split.
85+
- [Talos versions and output format](configuration/talos-versions.md) — which schema a render targets, and the version keys to pin.
8686
- [Applying with side-patches](operations/side-patches.md) — the `-f` chain.
8787
- [CLI reference](reference/index.md) — generated from the command tree.

‎docs/manual-test-plan.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,21 @@ Watch for:
4848
- `talm.key` written without the security-information banner.
4949
- `.gitignore` not updated.
5050

51+
### A1a. Version pins on a project created before them
52+
53+
Run inside the A1 project (`--preset cozystack`, which pins `talosVersion: "v1.12"`).
54+
55+
```bash
56+
sed -i'' -e 's|talosVersion: .*|talosVersion: ""|' -e 's|kubernetesVersion: .*|kubernetesVersion: ""|' Chart.yaml
57+
talm template --offline --full -t templates/controlplane.yaml
58+
```
59+
60+
Expected: the render stops with `templateOptions.kubernetesVersion is not set` and a hint naming the key.
61+
62+
Restore `talosVersion: "v1.12"` alone and re-run: the render succeeds and warns on stderr that no component images are pinned. Restore the preset's `kubernetesVersion: "v1.34.3"` too and re-run: no warning, and the output carries `kubelet`, `kube-apiserver`, `kube-controller-manager`, `kube-proxy` and `kube-scheduler` images on that version.
63+
64+
`--full` matters here. A default render is a diff against the same bundle, so the images cancel out on both sides and never appear in a node file either way.
65+
5166
### A2. `talm init` second run without `--force`
5267

5368
```bash

‎docs/operations/safety-gates.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111

1212
## 1. Declared-resource existence
1313

14-
Before sending the config to the node, the gate walks the rendered MachineConfig, extracts every reference to a host-side resource (network links from v1.12 multi-doc — `LinkConfig.name`, `BondConfig.links[]`, `VLANConfig.parent`, `BridgeConfig.links[]`, `VRFConfig.links[]`, `Layer2VIPConfig.link`, `HCloudVIPConfig.link`, `DHCPv4Config.name` / `DHCPv6Config.name` / `EthernetConfig.name`, and from v1.14 `BGPInstanceConfig.advertise[]` and `BGPInstanceConfig.neighbors[].link`; v1.11 legacy `machine.network.interfaces[].interface`; install disk via `machine.install.disk` literal or `machine.install.diskSelector`; `UserVolumeConfig.provisioning.diskSelector`), and verifies each against the node's COSI `LinkStatus`/`Disk` snapshots. A reference that doesn't resolve fails the apply with a `[blocker]` line listing the available names so the typo or migration miss is fixable from the values without re-running discovery.
14+
Before sending the config to the node, the gate walks the rendered MachineConfig, extracts every reference to a host-side resource (network links from v1.12 multi-doc — `LinkConfig.name`, `BondConfig.links[]`, `VLANConfig.parent`, `BridgeConfig.links[]`, `Layer2VIPConfig.link`, `HCloudVIPConfig.link`, `DHCPv4Config.name` / `DHCPv6Config.name` / `EthernetConfig.name`; from v1.13 `VRFConfig.links[]`; from v1.14 `BGPInstanceConfig.advertise[]` and `BGPInstanceConfig.neighbors[].link`; v1.11 legacy `machine.network.interfaces[].interface`; install disk via `machine.install.disk` literal or `machine.install.diskSelector`; `UserVolumeConfig.provisioning.diskSelector`), and verifies each against the node's COSI `LinkStatus`/`Disk` snapshots. A reference that doesn't resolve fails the apply with a `[blocker]` line listing the available names so the typo or migration miss is fixable from the values without re-running discovery.
1515

1616
Disk selectors must match at least one (non-readonly, non-CDROM, non-virtual) disk — zero matches block, multiple matches warn (install picks the first).
1717

‎docs/operations/talosctl-commands.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ Talos v1.14 registers `meta`'s `--insecure` on that command's local flag set ins
3232
That is the only way to write a META key over the maintenance service, which is what you do before a node has a machine config, so talm re-publishes such flags as persistent on its wrapper. `talm meta write --insecure` and the `-i` shorthand keep working, and `--cert-fingerprint` travels with them.
3333

3434
Reported upstream as [siderolabs/talos#14346](https://github.com/siderolabs/talos/issues/14346) and fixed by [siderolabs/talos#14347](https://github.com/siderolabs/talos/pull/14347), which is merged to `main` but not backported to the v1.14 branch. Once the fix ships in a Talos release talm depends on, the wrapper step becomes redundant and is dropped.
35+
3536
## `talm apply` — `--mode=reboot` is gone on Talos v1.14
3637

3738
Upstream stopped registering `reboot` among the values of the apply mode flag in v1.14, so `talm apply --mode=reboot` fails with `invalid argument "reboot" for "-m, --mode" flag`. Use `--mode=auto`, which the node promotes to a reboot when the change requires one. Shell completion reads the accepted values back from the flag, so it no longer suggests `reboot` either.
@@ -44,6 +45,14 @@ Upstream stopped registering `--insecure` on `upgrade` in v1.14 (it had been dep
4445

4546
Upstream dropped `--insecure` from `reset` in v1.14, so `talm reset --insecure` fails with `unknown flag: --insecure`. Resetting a node that has no valid configuration is done from the maintenance side instead: boot the node into a maintenance image and apply a fresh config, rather than resetting over an unauthenticated connection.
4647

48+
## `talm support` — the bundle is encrypted by default on Talos v1.14
49+
50+
Upstream changed the default: `talm support` now encrypts the generated bundle with age, to a built-in list of Sidero Labs recipients. `--no-encryption` turns that off, `--encryption-recipients` sends it to recipients you choose, and `--encryption-no-default-recipients` keeps yours while dropping theirs. Worth knowing before collecting a bundle from a cluster whose contents you would rather not hand to a third party, encrypted or otherwise.
51+
52+
## `talm containers`, `logs`, `restart`, `stats` — `-k` / `--kubernetes` is deprecated
53+
54+
Upstream replaced the flag with `--namespace`, which takes `system`, `cri` or `taloscontainers`. The old spelling still works and prints `Flag --kubernetes has been deprecated, use --namespace cri instead`, so scripts keep running for now; move them over before the flag goes the way of the others on this page.
55+
4756
## `talm reset` — META-preserving default
4857

4958
`talm reset` diverges from upstream `talosctl reset` on one default. Upstream defaults to `--wipe-mode=all`, which wipes the Talos META partition along with STATE and EPHEMERAL — the node cannot self-recover and comes up in maintenance mode requiring a full re-apply. Talm instead populates `--system-labels-to-wipe=STATE,EPHEMERAL` when neither `--wipe-mode` nor `--system-labels-to-wipe` was passed, which preserves META so the node rejoins the cluster from its META-stored bootstrap config on the next boot.

0 commit comments

Comments
 (0)