You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3525abf
Browse filesBrowse the repository at this point in the historyBrowse files
Address blockers + significant issues from the branch-review:
Documentation (blockers #1, #2, #3, #4):
- README safety-gates block now mirrors the code's actual YAML
field names: VLANConfig.parent (not .link), BridgeConfig.links
(not .ports). Dropped the addresses[].linkName /
routes[].linkName mentions — those fields don't exist in
v1alpha1.
- README Phase 2A bullet flipped: --dry-run RUNS the gate (was
documented as 'skipped'; code has run-on-dry-run since the
earlier fix(commands): run Phase 2A drift preview on --dry-run
commit).
- README adds Phase 2C bullet documenting
--skip-post-upgrade-verify + the silent-rollback / slow-boot
detection.
- README documents the virtual-creator-skip list and the
out-of-scope machine.disks[].device.
- apply-safety-gates-test-plan.md: VLAN parent test now uses the
correct 'parent:' key, bridge slave test uses 'links:'. Public
IPs replaced with a parameterized placeholder per RFC 5737
(private cluster identity no longer leaked).
Code (significant issues):
- Phase 2C error message + hint softened: two hypotheses produce
the same symptom (auto-rollback vs slow boot beyond the 90s
reconcile window). The hint instructs the operator to re-run
after a minute to distinguish.
- parseTargetVersion digest-pin detection now uses the
authoritative '@sha256:' / '@sha512:' marker instead of a
heuristic '/' check inside the post-':' substring. Tests cover
both the realistic hex-only digest and the rare slashed form.
- runPostApplyGates extracted as a separate function. Phase 2B
now collects per-node findings via errors.Join before
surfacing, mirroring ValidateRefs's collect-then-block
pattern; pre-fix the first node's blocker hid the rest.
- shouldRunDriftPreview pure predicate added, mirroring
shouldRunPostApplyVerify. New regression-pin test prevents
future --dry-run skips from sneaking back in.
- cosiLinksDisksReader unreachable-snapshot path: drop the
duplicate stderr warning, surface a single error+hint chain.
--skip-resource-validation now appears once, not twice.
All existing tests still pass; new tests pin Phase 2A
dry-run behaviour and digest-pin edge cases.
Refs: #172
Signed-off-by: Aleksei Sviridkin <[email protected]>
Copy file name to clipboardExpand all lines: README.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -170,15 +170,17 @@ cluster:
170
170
171
171
> **Version compatibility (`templateOptions.talosVersion` / `--talos-version`).** This setting must match the **Talos version actually running on the target node** — i.e. the maintenance ISO/PXE the node booted from for `apply -i`, or the installed Talos for an authenticated apply. It is **not** the same as `install.image`, which only controls what gets written to disk after a successful apply. When the configured contract is newer than the running binary, machinery injects fields (e.g. `machine.install.grubUseUKICmdline` from v1.12) that the running parser does not know, and the apply fails on the node side with `failed to parse config: unknown keys found during decoding: ...`. `talm apply` runs a best-effort pre-flight check against the running version and prints a `warning: pre-flight: ...` line with a hint when it detects this mismatch; if the warning is missed, the same hint is appended to the apply error. Either reboot the node into a maintenance image that matches the configured contract, or lower `templateOptions.talosVersion` / `--talos-version` to match what is running.
172
172
173
-
> **Apply-time safety gates.** `talm apply` runs three additional gates around each apply:
173
+
> **Apply-time safety gates.** `talm apply` and `talm upgrade` run additional gates around each operation:
174
174
>
175
-
> 1. **Declared-resource existence** (`--skip-resource-validation` opt-out, default on). Before sending the config to the node, the gate walks the rendered MachineConfig, extracts every reference to a host-side resource (network links: `LinkConfig.name`, `BondConfig.links[]`, `VLANConfig.link`, `BridgeConfig.ports[]`, `Layer2VIPConfig.link`, `addresses[].linkName`, `routes[].linkName`, the legacy `machine.network.interfaces[].interface`; install disk literal or `diskSelector`; `UserVolumeConfig.provisioning.diskSelector`), and verifies each against the node's COSI `LinkStatus`/`Disk` snapshots. A reference that doesn't resolve fails the apply with a `[blocker]` line listing the available names so the typo or migration miss is fixable from the values without re-running discovery. Disk selectors must match at least one (non-readonly, non-CDROM) disk — zero matches block, multiple matches warn (install picks the first). Pass `--skip-resource-validation` for recovery into a maintenance image with mismatched hardware or pre-staging values for hardware that isn't installed yet.
175
+
> 1. **Declared-resource existence** (`--skip-resource-validation` opt-out, default on). Before sending the config to the node, the gate walks the rendered MachineConfig, extracts every reference to a host-side resource (network links from v1.12 multi-doc — `LinkConfig.name`, `BondConfig.links[]`, `VLANConfig.parent`, `BridgeConfig.links[]`, `Layer2VIPConfig.link`, `HCloudVIPConfig.link`, `DHCPv4Config.name` / `DHCPv6Config.name` / `EthernetConfig.name`; v1.11 legacy `machine.network.interfaces[].interface`; install disk via `machine.install.disk` literal or `machine.install.diskSelector`; `UserVolumeConfig.provisioning.diskSelector`), and verifies each against the node's COSI `LinkStatus`/`Disk` snapshots. A reference that doesn't resolve fails the apply with a `[blocker]` line listing the available names so the typo or migration miss is fixable from the values without re-running discovery. Disk selectors must match at least one (non-readonly, non-CDROM, non-virtual) disk — zero matches block, multiple matches warn (install picks the first). Virtual-link-creator documents (`BondConfig.name`, `VLANConfig.name`, `BridgeConfig.name`, `WireguardConfig.name`, `DummyLinkConfig.name`, `LinkAliasConfig.name`) are intentionally NOT validated against existing links — those `.name` fields describe new virtual links the apply is creating, not references to pre-existing host resources. Out of scope today: `machine.disks[].device` (extra-disk partitioning); track in a follow-up if you need it. Pass `--skip-resource-validation` for recovery into a maintenance image with mismatched hardware or pre-staging values for hardware that isn't installed yet.
176
176
>
177
-
> 2. **Pre-apply drift preview** (`--skip-drift-preview` opt-out, default on). Reads the node's current MachineConfig via COSI and prints a `+`/`-`/`~`/`=` diff of what's about to change, keyed by `(kind, name)`. Informational only — never blocks. The `-` lines are the most useful: they surface stale documents from a previous apply that the new render no longer emits (e.g. an `eth1` LinkConfig lingering after a migration to `eth0`). Reading the current config requires the auth path — `MachineConfig` is a Sensitive COSI resource and is unreachable on the `--insecure` maintenance connection; the gate prints `drift verification unavailable on maintenance connection` and proceeds in that case. `--dry-run` skips this gate (nothing's about to apply).
177
+
> 2. **Pre-apply drift preview** (`--skip-drift-preview` opt-out, default on). Reads the node's current MachineConfig via COSI and prints a `+`/`-`/`~`/`=` diff of what's about to change, keyed by `(kind, name)`. Informational only — never blocks. The `-` lines are the most useful: they surface stale documents from a previous apply that the new render no longer emits (e.g. an `eth1` LinkConfig lingering after a migration to `eth0`). Reading the current config requires the auth path — `MachineConfig` is a Sensitive COSI resource and is unreachable on the `--insecure` maintenance connection; the gate prints `drift verification unavailable on maintenance connection` and proceeds in that case. **`--dry-run` runs this gate** — the diff is read-only and "show me what would change" is exactly the dry-run contract.
178
178
>
179
-
> 3. **Post-apply state verification** (`--skip-post-apply-verify` opt-out, **default off** until the Talos-mutated-field allowlist lands — see [#172](https://github.com/cozystack/talm/issues/172)). After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig and structurally compares it against the bytes that were sent. Divergence blocks the apply chain with a per-document diff, primarily catching silent doc drops (Talos parser ignored an unknown field) and controller reverts. Disabled by default because Talos mutates a handful of leaf fields post-apply (cert hashes, timestamps) that would surface as false-positive divergence without an allowlist. `--mode=staged` and `--mode=try`always skip this gate: the contract for those modes diverges from "what was sent is what is on the node now". `--dry-run` skips it too.
179
+
> 3. **Post-apply state verification** (`--skip-post-apply-verify` opt-out, **default off** until the Talos-mutated-field allowlist lands — see [#172](https://github.com/cozystack/talm/issues/172)). After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig and structurally compares it against the bytes that were sent. Divergence blocks the apply chain with a per-document diff, primarily catching silent doc drops (Talos parser ignored an unknown field) and controller reverts. Disabled by default because Talos mutates a handful of leaf fields post-apply (cert hashes, timestamps) that would surface as false-positive divergence without an allowlist. `--mode=staged`, `--mode=try`, `--mode=reboot` always skip this gate (each for a documented reason — staged stores rather than activates, try auto-rolls back, reboot kills the COSI connection mid-verify). `--dry-run` skips it too.
180
180
>
181
-
> The three skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.
181
+
> 4. **Post-upgrade version verify** (`--skip-post-upgrade-verify` opt-out, default on — the gate runs). After `talm upgrade` reports success, waits 90s for the node to finish booting then reads `runtime.Version` COSI and compares the running version's `(Major, Minor)` contract against the contract parsed from the target image tag. Point releases share a minor contract; cross-minor mismatch surfaces as a hint-bearing blocker. Catches the silent A/B rollback case where the upgrade RPC acks success but Talos rolled back to the previous partition (cross-vendor image, missing extensions, failed boot readiness check, slow boot exceeding the reconcile window). Best-effort surrender on digest-pinned images and unparseable tags. See [#175](https://github.com/cozystack/talm/issues/175) for the dev17 reproduction.
182
+
>
183
+
> The skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.
Copy file name to clipboardExpand all lines: docs/apply-safety-gates-test-plan.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,8 +18,8 @@ Run all matrix cells against the binary at `/tmp/talm-safety`. Use the dev17 ref
18
18
| --- | --- | --- |
19
19
| Typoed `LinkConfig.name`| Add `LinkConfig{name: eth9999}` to a node body |`[blocker] declared link "eth9999" not found …` plus available-links hint |
20
20
| Typoed bond slave | Add `BondConfig{name: bond0, links: [ghost0, ens5]}`| Blocker on `ghost0` only; `bond0` (new bond) NOT flagged |
21
-
| Typoed VLAN parent | Add `VLANConfig{name: ens5.99, link: ghost0, vlanID: 99}`| Blocker on `link: ghost0`; `name: ens5.99` not flagged (new VLAN) |
22
-
| Typoed bridge port| Add `BridgeConfig{name: br99, ports: [ghost0]}`| Blocker on `ghost0` only; `br99` not flagged |
21
+
| Typoed VLAN parent | Add `VLANConfig{name: ens5.99, parent: ghost0, vlanID: 99}`(YAML key is `parent`, NOT `link` — `vlan.go ParentLinkConfig`) | Blocker on `parent: ghost0`; `name: ens5.99` not flagged (new VLAN) |
22
+
| Typoed bridge slave| Add `BridgeConfig{name: br99, links: [ghost0]}` (YAML key is `links`, NOT `ports` — `bridge.go BridgeLinks`)| Blocker on `ghost0` only; `br99` not flagged |
23
23
| Typoed Layer2VIP link | Set `vipLink: ghost0` in values | Blocker on `link: ghost0`|
24
24
| Legacy v1.11 interface |`machine.network.interfaces[].interface: eth9999`| Blocker; same hint shape |
25
25
@@ -90,7 +90,7 @@ Before requesting human review, exercise the gates against a live Talos node.
90
90
cd~/git/github.com/lexfrei/work/env/dev17/talm
91
91
```
92
92
93
-
dev17 carries an OCI 3-node Talos v1.12.6 cluster (158.101.113.227 / 129.158.237.173 / 157.151.143.81). The vendored talm library in `charts/talm/` may need `talm init --update --preset cozystack` to pick up new helpers; preset templates (`templates/_helpers.tpl`) require interactive confirm and won't auto-update non-tty (known gap, see #174).
93
+
Use a 3-node Talos cluster (replace placeholders below with your own node IPs — examples use RFC 5737 documentation ranges). The vendored talm library in `charts/talm/` may need `talm init --update --preset cozystack` to pick up new helpers; preset templates (`templates/_helpers.tpl`) require interactive confirm and won't auto-update non-tty (known gap, see #174).
"(2) The node is slower than the 90s reconcile window — large image pulls or cold "+
32
+
"hardware can exceed it. Re-run `talm get version` after a minute to distinguish: if "+
33
+
"the version updated, the node was just slow; if it's still the old version, the "+
34
+
"rollback case is real. Pass --skip-post-upgrade-verify to bypass."
31
35
32
36
// verifyPostUpgradeVersion is the Phase 2C gate: after talosctl upgrade
33
37
// returns, re-read the node's runtime.Version COSI resource and compare
@@ -80,7 +84,7 @@ func verifyPostUpgradeVersion(
80
84
81
85
//nolint:wrapcheck // cockroachdb/errors.WithHint at boundary.
82
86
returnerrors.WithHint(
83
-
errors.Newf("post-upgrade: requested upgrade to %s but running version is %s — Talos auto-rolled backthe new install", target, running),
87
+
errors.Newf("post-upgrade: requested upgrade to %s but running version is %s — either Talos auto-rolled back, or the node is still booting beyond the 90s window", target, running),
0 commit comments