Skip to content

Commit 3525abf

Browse files
committed
fix(docs+commands): branch-review pass 1 — README accuracy + Phase 2C UX
Address blockers + significant issues from the branch-review: Documentation (blockers #1, #2, #3, #4): - README safety-gates block now mirrors the code's actual YAML field names: VLANConfig.parent (not .link), BridgeConfig.links (not .ports). Dropped the addresses[].linkName / routes[].linkName mentions — those fields don't exist in v1alpha1. - README Phase 2A bullet flipped: --dry-run RUNS the gate (was documented as 'skipped'; code has run-on-dry-run since the earlier fix(commands): run Phase 2A drift preview on --dry-run commit). - README adds Phase 2C bullet documenting --skip-post-upgrade-verify + the silent-rollback / slow-boot detection. - README documents the virtual-creator-skip list and the out-of-scope machine.disks[].device. - apply-safety-gates-test-plan.md: VLAN parent test now uses the correct 'parent:' key, bridge slave test uses 'links:'. Public IPs replaced with a parameterized placeholder per RFC 5737 (private cluster identity no longer leaked). Code (significant issues): - Phase 2C error message + hint softened: two hypotheses produce the same symptom (auto-rollback vs slow boot beyond the 90s reconcile window). The hint instructs the operator to re-run after a minute to distinguish. - parseTargetVersion digest-pin detection now uses the authoritative '@sha256:' / '@sha512:' marker instead of a heuristic '/' check inside the post-':' substring. Tests cover both the realistic hex-only digest and the rare slashed form. - runPostApplyGates extracted as a separate function. Phase 2B now collects per-node findings via errors.Join before surfacing, mirroring ValidateRefs's collect-then-block pattern; pre-fix the first node's blocker hid the rest. - shouldRunDriftPreview pure predicate added, mirroring shouldRunPostApplyVerify. New regression-pin test prevents future --dry-run skips from sneaking back in. - cosiLinksDisksReader unreachable-snapshot path: drop the duplicate stderr warning, surface a single error+hint chain. --skip-resource-validation now appears once, not twice. All existing tests still pass; new tests pin Phase 2A dry-run behaviour and digest-pin edge cases. Refs: #172 Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent cab19c1 commit 3525abf

7 files changed

Lines changed: 99 additions & 36 deletions

‎README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -170,15 +170,17 @@ cluster:
170170

171171
> **Version compatibility (`templateOptions.talosVersion` / `--talos-version`).** This setting must match the **Talos version actually running on the target node** — i.e. the maintenance ISO/PXE the node booted from for `apply -i`, or the installed Talos for an authenticated apply. It is **not** the same as `install.image`, which only controls what gets written to disk after a successful apply. When the configured contract is newer than the running binary, machinery injects fields (e.g. `machine.install.grubUseUKICmdline` from v1.12) that the running parser does not know, and the apply fails on the node side with `failed to parse config: unknown keys found during decoding: ...`. `talm apply` runs a best-effort pre-flight check against the running version and prints a `warning: pre-flight: ...` line with a hint when it detects this mismatch; if the warning is missed, the same hint is appended to the apply error. Either reboot the node into a maintenance image that matches the configured contract, or lower `templateOptions.talosVersion` / `--talos-version` to match what is running.
172172

173-
> **Apply-time safety gates.** `talm apply` runs three additional gates around each apply:
173+
> **Apply-time safety gates.** `talm apply` and `talm upgrade` run additional gates around each operation:
174174
>
175-
> 1. **Declared-resource existence** (`--skip-resource-validation` opt-out, default on). Before sending the config to the node, the gate walks the rendered MachineConfig, extracts every reference to a host-side resource (network links: `LinkConfig.name`, `BondConfig.links[]`, `VLANConfig.link`, `BridgeConfig.ports[]`, `Layer2VIPConfig.link`, `addresses[].linkName`, `routes[].linkName`, the legacy `machine.network.interfaces[].interface`; install disk literal or `diskSelector`; `UserVolumeConfig.provisioning.diskSelector`), and verifies each against the node's COSI `LinkStatus`/`Disk` snapshots. A reference that doesn't resolve fails the apply with a `[blocker]` line listing the available names so the typo or migration miss is fixable from the values without re-running discovery. Disk selectors must match at least one (non-readonly, non-CDROM) disk — zero matches block, multiple matches warn (install picks the first). Pass `--skip-resource-validation` for recovery into a maintenance image with mismatched hardware or pre-staging values for hardware that isn't installed yet.
175+
> 1. **Declared-resource existence** (`--skip-resource-validation` opt-out, default on). Before sending the config to the node, the gate walks the rendered MachineConfig, extracts every reference to a host-side resource (network links from v1.12 multi-doc — `LinkConfig.name`, `BondConfig.links[]`, `VLANConfig.parent`, `BridgeConfig.links[]`, `Layer2VIPConfig.link`, `HCloudVIPConfig.link`, `DHCPv4Config.name` / `DHCPv6Config.name` / `EthernetConfig.name`; v1.11 legacy `machine.network.interfaces[].interface`; install disk via `machine.install.disk` literal or `machine.install.diskSelector`; `UserVolumeConfig.provisioning.diskSelector`), and verifies each against the node's COSI `LinkStatus`/`Disk` snapshots. A reference that doesn't resolve fails the apply with a `[blocker]` line listing the available names so the typo or migration miss is fixable from the values without re-running discovery. Disk selectors must match at least one (non-readonly, non-CDROM, non-virtual) disk — zero matches block, multiple matches warn (install picks the first). Virtual-link-creator documents (`BondConfig.name`, `VLANConfig.name`, `BridgeConfig.name`, `WireguardConfig.name`, `DummyLinkConfig.name`, `LinkAliasConfig.name`) are intentionally NOT validated against existing links — those `.name` fields describe new virtual links the apply is creating, not references to pre-existing host resources. Out of scope today: `machine.disks[].device` (extra-disk partitioning); track in a follow-up if you need it. Pass `--skip-resource-validation` for recovery into a maintenance image with mismatched hardware or pre-staging values for hardware that isn't installed yet.
176176
>
177-
> 2. **Pre-apply drift preview** (`--skip-drift-preview` opt-out, default on). Reads the node's current MachineConfig via COSI and prints a `+`/`-`/`~`/`=` diff of what's about to change, keyed by `(kind, name)`. Informational only — never blocks. The `-` lines are the most useful: they surface stale documents from a previous apply that the new render no longer emits (e.g. an `eth1` LinkConfig lingering after a migration to `eth0`). Reading the current config requires the auth path — `MachineConfig` is a Sensitive COSI resource and is unreachable on the `--insecure` maintenance connection; the gate prints `drift verification unavailable on maintenance connection` and proceeds in that case. `--dry-run` skips this gate (nothing's about to apply).
177+
> 2. **Pre-apply drift preview** (`--skip-drift-preview` opt-out, default on). Reads the node's current MachineConfig via COSI and prints a `+`/`-`/`~`/`=` diff of what's about to change, keyed by `(kind, name)`. Informational only — never blocks. The `-` lines are the most useful: they surface stale documents from a previous apply that the new render no longer emits (e.g. an `eth1` LinkConfig lingering after a migration to `eth0`). Reading the current config requires the auth path — `MachineConfig` is a Sensitive COSI resource and is unreachable on the `--insecure` maintenance connection; the gate prints `drift verification unavailable on maintenance connection` and proceeds in that case. **`--dry-run` runs this gate** — the diff is read-only and "show me what would change" is exactly the dry-run contract.
178178
>
179-
> 3. **Post-apply state verification** (`--skip-post-apply-verify` opt-out, **default off** until the Talos-mutated-field allowlist lands — see [#172](https://github.com/cozystack/talm/issues/172)). After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig and structurally compares it against the bytes that were sent. Divergence blocks the apply chain with a per-document diff, primarily catching silent doc drops (Talos parser ignored an unknown field) and controller reverts. Disabled by default because Talos mutates a handful of leaf fields post-apply (cert hashes, timestamps) that would surface as false-positive divergence without an allowlist. `--mode=staged` and `--mode=try` always skip this gate: the contract for those modes diverges from "what was sent is what is on the node now". `--dry-run` skips it too.
179+
> 3. **Post-apply state verification** (`--skip-post-apply-verify` opt-out, **default off** until the Talos-mutated-field allowlist lands — see [#172](https://github.com/cozystack/talm/issues/172)). After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig and structurally compares it against the bytes that were sent. Divergence blocks the apply chain with a per-document diff, primarily catching silent doc drops (Talos parser ignored an unknown field) and controller reverts. Disabled by default because Talos mutates a handful of leaf fields post-apply (cert hashes, timestamps) that would surface as false-positive divergence without an allowlist. `--mode=staged`, `--mode=try`, `--mode=reboot` always skip this gate (each for a documented reason — staged stores rather than activates, try auto-rolls back, reboot kills the COSI connection mid-verify). `--dry-run` skips it too.
180180
>
181-
> The three skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.
181+
> 4. **Post-upgrade version verify** (`--skip-post-upgrade-verify` opt-out, default on — the gate runs). After `talm upgrade` reports success, waits 90s for the node to finish booting then reads `runtime.Version` COSI and compares the running version's `(Major, Minor)` contract against the contract parsed from the target image tag. Point releases share a minor contract; cross-minor mismatch surfaces as a hint-bearing blocker. Catches the silent A/B rollback case where the upgrade RPC acks success but Talos rolled back to the previous partition (cross-vendor image, missing extensions, failed boot readiness check, slow boot exceeding the reconcile window). Best-effort surrender on digest-pinned images and unparseable tags. See [#175](https://github.com/cozystack/talm/issues/175) for the dev17 reproduction.
182+
>
183+
> The skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.
182184

183185
Apply config:
184186
```bash

‎docs/apply-safety-gates-test-plan.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ Run all matrix cells against the binary at `/tmp/talm-safety`. Use the dev17 ref
1818
| --- | --- | --- |
1919
| Typoed `LinkConfig.name` | Add `LinkConfig{name: eth9999}` to a node body | `[blocker] declared link "eth9999" not found …` plus available-links hint |
2020
| Typoed bond slave | Add `BondConfig{name: bond0, links: [ghost0, ens5]}` | Blocker on `ghost0` only; `bond0` (new bond) NOT flagged |
21-
| Typoed VLAN parent | Add `VLANConfig{name: ens5.99, link: ghost0, vlanID: 99}` | Blocker on `link: ghost0`; `name: ens5.99` not flagged (new VLAN) |
22-
| Typoed bridge port | Add `BridgeConfig{name: br99, ports: [ghost0]}` | Blocker on `ghost0` only; `br99` not flagged |
21+
| Typoed VLAN parent | Add `VLANConfig{name: ens5.99, parent: ghost0, vlanID: 99}` (YAML key is `parent`, NOT `link` — `vlan.go ParentLinkConfig`) | Blocker on `parent: ghost0`; `name: ens5.99` not flagged (new VLAN) |
22+
| Typoed bridge slave | Add `BridgeConfig{name: br99, links: [ghost0]}` (YAML key is `links`, NOT `ports` — `bridge.go BridgeLinks`) | Blocker on `ghost0` only; `br99` not flagged |
2323
| Typoed Layer2VIP link | Set `vipLink: ghost0` in values | Blocker on `link: ghost0` |
2424
| Legacy v1.11 interface | `machine.network.interfaces[].interface: eth9999` | Blocker; same hint shape |
2525

@@ -90,7 +90,7 @@ Before requesting human review, exercise the gates against a live Talos node.
9090
cd ~/git/github.com/lexfrei/work/env/dev17/talm
9191
```
9292

93-
dev17 carries an OCI 3-node Talos v1.12.6 cluster (158.101.113.227 / 129.158.237.173 / 157.151.143.81). The vendored talm library in `charts/talm/` may need `talm init --update --preset cozystack` to pick up new helpers; preset templates (`templates/_helpers.tpl`) require interactive confirm and won't auto-update non-tty (known gap, see #174).
93+
Use a 3-node Talos cluster (replace placeholders below with your own node IPs — examples use RFC 5737 documentation ranges). The vendored talm library in `charts/talm/` may need `talm init --update --preset cozystack` to pick up new helpers; preset templates (`templates/_helpers.tpl`) require interactive confirm and won't auto-update non-tty (known gap, see #174).
9494

9595
### Sanity check
9696

‎pkg/commands/apply.go‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -362,14 +362,25 @@ func applyOneFileDirectPatchMode(configFile, withSecretsPath string) error {
362362

363363
helpers.PrintApplyResults(resp)
364364

365-
for _, node := range targetNodes {
366-
if err := runPostApplyGate(client.WithNode(ctx, node), c, result, os.Stderr); err != nil {
367-
return err
368-
}
365+
return runPostApplyGates(ctx, c, result, targetNodes)
366+
})
367+
}
368+
369+
// runPostApplyGates fans Phase 2B verification across every target
370+
// node, collecting per-node findings before surfacing them. Apply
371+
// already happened on every node — short-circuiting on the first
372+
// divergence would hide later nodes' state. Mirrors ValidateRefs's
373+
// collect-then-block pattern.
374+
func runPostApplyGates(ctx context.Context, c *client.Client, result []byte, targetNodes []string) error {
375+
var perNodeErrs []error
376+
377+
for _, node := range targetNodes {
378+
if err := runPostApplyGate(client.WithNode(ctx, node), c, result, os.Stderr); err != nil {
379+
perNodeErrs = append(perNodeErrs, errors.Wrapf(err, "node %s", node))
369380
}
381+
}
370382

371-
return nil
372-
})
383+
return errors.Join(perNodeErrs...)
373384
}
374385

375386
// runPreApplyGates wires the two pre-apply safety gates against the
@@ -389,13 +400,22 @@ func runPreApplyGates(ctx context.Context, c *client.Client, rendered []byte, w
389400
}
390401
}
391402

392-
if applyCmdFlags.skipDriftPreview {
403+
if !shouldRunDriftPreview(applyCmdFlags.skipDriftPreview) {
393404
return nil
394405
}
395406

396407
return previewDrift(ctx, cosiMachineConfigReader(c, applyCmdFlags.insecure), rendered, w)
397408
}
398409

410+
// shouldRunDriftPreview is the testable predicate for Phase 2A
411+
// scheduling. Pure function for pin-testing the contract: dry-run
412+
// is intentionally NOT a skip reason here (the preview is read-only;
413+
// dry-run wants the diff). Only the explicit skip flag suppresses
414+
// the gate.
415+
func shouldRunDriftPreview(skip bool) bool {
416+
return !skip
417+
}
418+
399419
// runPostApplyGate wires Phase 2B (post-apply state verification).
400420
// Skipped on dry-run (no real apply) and on the staged/try/reboot
401421
// apply modes:

‎pkg/commands/preflight_apply_safety.go‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,12 +64,13 @@ func preflightValidateResources(
6464
) error {
6565
snapshot, ok := read(ctx)
6666
if !ok {
67-
_, _ = fmt.Fprintln(w, "warning: could not read host snapshot for resource validation; pass --skip-resource-validation to proceed without it")
68-
67+
// Single message via the error + hint chain. An earlier
68+
// version printed a duplicate warning line to w, so the
69+
// operator saw --skip-resource-validation mentioned twice.
6970
//nolint:wrapcheck // cockroachdb/errors.WithHint at boundary.
7071
return errors.WithHint(
71-
errors.New("pre-flight: host resource snapshot unavailable"),
72-
"could not read links/disks from the node — pass --skip-resource-validation to bypass.",
72+
errors.New("pre-flight: could not read host links/disks snapshot from the node"),
73+
"the node's COSI links/disks resources were unreachable — fix the connection or pass --skip-resource-validation to bypass.",
7374
)
7475
}
7576

‎pkg/commands/preflight_apply_safety_test.go‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,36 @@ func TestVerifyAppliedState_InsecurePath_NoBlock(t *testing.T) {
284284
}
285285
}
286286

287+
// TestShouldRunDriftPreview_RespectsOnlySkipFlag pins the Phase 2A
288+
// scheduling contract: dry-run does NOT skip the drift preview (the
289+
// gate is read-only and dry-run wants the diff); only the explicit
290+
// --skip-drift-preview flag suppresses it. This pins the
291+
// fix(commands): run Phase 2A drift preview on --dry-run commit
292+
// against future flips that might re-introduce the dry-run skip.
293+
func TestShouldRunDriftPreview_RespectsOnlySkipFlag(t *testing.T) {
294+
t.Parallel()
295+
296+
tests := []struct {
297+
name string
298+
skip bool
299+
want bool
300+
}{
301+
{"default: runs", false, true},
302+
{"--skip-drift-preview: skipped", true, false},
303+
}
304+
305+
for _, tc := range tests {
306+
t.Run(tc.name, func(t *testing.T) {
307+
t.Parallel()
308+
309+
got := shouldRunDriftPreview(tc.skip)
310+
if got != tc.want {
311+
t.Errorf("shouldRunDriftPreview(skip=%v) = %v, want %v", tc.skip, got, tc.want)
312+
}
313+
})
314+
}
315+
}
316+
287317
// TestShouldRunPostApplyVerify_RespectsModeAndDryRun pins the post-apply
288318
// gate's skip predicate. staged/try modes have contracts that diverge
289319
// from 'sent == on-node' (staged: not yet active until reboot; try:

‎pkg/commands/preflight_upgrade_verify.go‎

Lines changed: 23 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,14 @@ import (
2424
machineryconfig "github.com/siderolabs/talos/pkg/machinery/config"
2525
)
2626

27-
const postUpgradeVersionMismatchHint = "Talos auto-rolled back the new install. " +
28-
"Cross-vendor upgrades (e.g. cozystack-bundled image -> vanilla siderolabs installer) " +
29-
"drop bundled extensions and fail the boot readiness check; use the cozystack-built " +
30-
"image at the target version, or pass --skip-post-upgrade-verify to bypass."
27+
const postUpgradeVersionMismatchHint = "two hypotheses produce this symptom: " +
28+
"(1) Talos auto-rolled back after the new partition failed its boot readiness check — " +
29+
"cross-vendor upgrades (e.g. cozystack-bundled image -> vanilla siderolabs installer) " +
30+
"drop bundled extensions and trigger this. " +
31+
"(2) The node is slower than the 90s reconcile window — large image pulls or cold " +
32+
"hardware can exceed it. Re-run `talm get version` after a minute to distinguish: if " +
33+
"the version updated, the node was just slow; if it's still the old version, the " +
34+
"rollback case is real. Pass --skip-post-upgrade-verify to bypass."
3135

3236
// verifyPostUpgradeVersion is the Phase 2C gate: after talosctl upgrade
3337
// returns, re-read the node's runtime.Version COSI resource and compare
@@ -80,7 +84,7 @@ func verifyPostUpgradeVersion(
8084

8185
//nolint:wrapcheck // cockroachdb/errors.WithHint at boundary.
8286
return errors.WithHint(
83-
errors.Newf("post-upgrade: requested upgrade to %s but running version is %s — Talos auto-rolled back the new install", target, running),
87+
errors.Newf("post-upgrade: requested upgrade to %s but running version is %s — either Talos auto-rolled back, or the node is still booting beyond the 90s window", target, running),
8488
postUpgradeVersionMismatchHint,
8589
)
8690
}
@@ -93,25 +97,28 @@ func verifyPostUpgradeVersion(
9397
// - factory.talos.dev/installer/<sha256>:v1.13.0 -> v1.13.0
9498
// - "" or no `:` separator -> "" (caller treats as "version unknown")
9599
//
100+
// Digest-pinned references (image@sha256:...) are rejected up front:
101+
// Phase 2C can only verify when the tag carries the version literal.
102+
// A digest-pinned upgrade silently passes the check (the operator
103+
// opted into a content-addressed install and can read it back via
104+
// talos image list).
105+
//
96106
// Returns the tag verbatim; comparison is done via machineryconfig
97107
// VersionContract so v1.13 and v1.13.0 are equivalent at the minor
98108
// level (point-release upgrades pass silently).
99109
func parseTargetVersion(image string) string {
100-
idx := strings.LastIndex(image, ":")
101-
if idx < 0 || idx == len(image)-1 {
110+
// Detect digest pins up front. The "@sha256:" marker is the
111+
// authoritative signal — a `/` in the post-last-colon substring
112+
// catches some shapes but misses hex-only digests like
113+
// `image@sha256:abc123def456`.
114+
if strings.Contains(image, "@sha256:") || strings.Contains(image, "@sha512:") {
102115
return ""
103116
}
104117

105-
tag := image[idx+1:]
106-
107-
// Reject digest references (image@sha256:...) — Phase 2C can only
108-
// verify when the tag carries the version literal. A digest-pinned
109-
// upgrade silently passes the check (the operator opted into a
110-
// content-addressed install and can read it back via talos image
111-
// list).
112-
if strings.Contains(tag, "/") {
118+
idx := strings.LastIndex(image, ":")
119+
if idx < 0 || idx == len(image)-1 {
113120
return ""
114121
}
115122

116-
return tag
123+
return image[idx+1:]
117124
}

‎pkg/commands/preflight_upgrade_verify_test.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,9 @@ func TestParseTargetVersion(t *testing.T) {
3838
{"factory installer", "factory.talos.dev/installer/abcd1234:v1.13.0", "v1.13.0"},
3939
{"no tag", "ghcr.io/cozystack/cozystack/talos", ""},
4040
{"empty tag", "ghcr.io/cozystack/cozystack/talos:", ""},
41-
{"digest pin", "ghcr.io/cozystack/cozystack/talos@sha256:abc/def", ""},
41+
{"digest pin with slash in hex (rare)", "ghcr.io/cozystack/cozystack/talos@sha256:abc/def", ""},
42+
{"realistic digest pin (hex only)", "ghcr.io/cozystack/cozystack/talos@sha256:abc123def456", ""},
43+
{"sha512 digest", "ghcr.io/foo/bar@sha512:0123456789abcdef", ""},
4244
{"empty input", "", ""},
4345
{"only colon", ":", ""},
4446
}
@@ -135,6 +137,7 @@ func TestVerifyPostUpgradeVersion_UnparseableTag_Skip(t *testing.T) {
135137
"",
136138
"no-tag-image",
137139
"ghcr.io/foo/bar@sha256:abc/def",
140+
"ghcr.io/foo/bar@sha256:abc123def456",
138141
"ghcr.io/foo/bar:not-a-version-string",
139142
}
140143

0 commit comments

Comments
 (0)