You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dac1d2c
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/manual-test-plan.md
+64-1Lines changed: 64 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1166,7 +1166,7 @@ go vet ./...
1166
1166
### Known limitations / follow-ups
1167
1167
1168
1168
-**Talos-mutated-field allowlist**: Phase 2B reports cert hashes / timestamps as divergence today; the verify is off by default until an allowlist lands.
1169
-
-**`talm upgrade`has no pre-upgrade gates** (Phase 2C runs *after*, not before): the upgrade flow wraps `talosctl upgrade` and doesn't route through `buildApplyClosure` / `applyOneFileDirectPatchMode`, so Phase 1 / Phase 2A do not run. Phase 2C (post-upgrade version verify) was added precisely to catch the silent-rollback class without that refactor. Full pre-upgrade gates would require reproducing the gate calls in `upgrade_handler.go` or refactoring the apply flow.
1169
+
-**`talm upgrade`does not run the apply-path gates**: the upgrade flow wraps `talosctl upgrade` and doesn't route through `buildApplyClosure` / `applyOneFileDirectPatchMode`, so Phase 1 / Phase 2A do not run. Two gates of its own cover the classes that matter most: the pre-upgrade path check refuses a move Talos will not make, and Phase 2C catches the silent rollback afterwards. Bringing the apply gates over would still require reproducing the gate calls in `upgrade_handler.go` or refactoring the apply flow.
1170
1170
-**Phase 1/2 on `--insecure`**: the safety gates can't run before the chart renders, and the chart's `lookup` calls need an authenticated COSI connection. Insecure path = effectively no gates today.
1171
1171
1172
1172
## D. Apply (insecure / maintenance path)
@@ -1320,6 +1320,69 @@ Expected: stderr emits the "Synced machine.install.image" line immediately after
1320
1320
1321
1321
Regression anchor: contract tests `TestContract_WriteBackInstallImage_*` pin the on-disk shape of the patch (scalar swap, idempotency, silent-skip on orphan files, structural errors, file-list fan-out). A regression that fires the write-back BEFORE verify (or instead of verify) would silently pin the body to an image the node never actually ran — the failure-path scenario above would catch it. Cross-reference: `pkg/commands/upgrade_image_writeback.go`.
1322
1322
1323
+
1324
+
### E5. Node body image divergence is reported
1325
+
1326
+
The upgrade target comes from `values.yaml`, so a node body naming something else is ignored and then rewritten. Both directions are reported, but only one of them is a problem.
1327
+
1328
+
```bash
1329
+
# Point a node body at a NEWER Talos than values.yaml names.
1330
+
sed -i.bak 's|image: ghcr.io/cozystack/cozystack/talos:.*|image: ghcr.io/cozystack/cozystack/talos:<A_NEWER_TAG>|' nodes/node0.yaml
1331
+
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
1332
+
```
1333
+
1334
+
Do not pipe this into `head`. With `2>&1 |` talm's stderr is the pipe, `head` closes it after five lines, and a Go process takes SIGPIPE on fd 2 rather than an error — talm dies after the RPC has already fired, skipping the post-upgrade verify and the `install.image` write-back that E4 covers. The pipeline still exits 0, so nothing says it happened.
1335
+
1336
+
Expected: `Using image from values.yaml: <ref>` followed by a `warning:` line naming `nodes/node0.yaml`, the body's image, and the resolved target, telling you to put that image in values.yaml if the file is the one you meant. The warning fires because the body names a newer Talos than the target: the upgrade is going somewhere the file does not.
1337
+
1338
+
`<A_NEWER_TAG>` only has to be newer than what `values.yaml` names — the comparison is ordered down to the patch, so `v1.13.1` against a `v1.13.0` target warns just like a newer minor does. A node moving backwards by a patch is the same incident one granularity down, and the pre-upgrade path check lets it through because Talos allows patch moves inside a minor.
1339
+
1340
+
Now the ordinary direction. Restore the body, then bump `values.yaml` forward a minor and leave the bodies as rendered, which is what an operator raising the cluster's version actually does:
1341
+
1342
+
```bash
1343
+
mv nodes/node0.yaml.bak nodes/node0.yaml
1344
+
cp values.yaml values.yaml.orig
1345
+
sed -i '''s|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:<A_NEWER_TAG>"|' values.yaml
1346
+
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
1347
+
```
1348
+
1349
+
Expected: the divergence reports as a plain line with no `warning:` label and no advice. That is the canonical flow, where every body trails the target until the write-back resyncs it, so advice to bump values.yaml would be advice to repeat what you just did.
1350
+
1351
+
Watch for:
1352
+
1353
+
- A `warning:` label on the trailing-body case, which is every node after a values.yaml bump.
1354
+
- Silence on a body that names another registry at the same version: that one cannot be shown to trail, so it is a divergence.
1355
+
- The report firing when the body already matches the target, or on a side-patch that declares no `install.image`.
1356
+
1357
+
### E6. Pre-upgrade path check
1358
+
1359
+
The refusal itself stops before the RPC, so that half needs no node that reboots. The escape-hatch flag is a different matter — see the warning below it.
1360
+
1361
+
Pick a target Talos actually refuses: two minors back, or several minors forward. One minor back is a supported rollback and must NOT be blocked.
1362
+
1363
+
```bash
1364
+
sed -i '''s|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:v1.10.5"|' values.yaml
1365
+
talm upgrade -f nodes/node0.yaml
1366
+
```
1367
+
1368
+
Expected: the upgrade is refused with `refusing this upgrade: host version <running> is too new to downgrade to Talos 1.10.5`, and a hint offering the two ways out. Nothing is sent to the node.
1369
+
1370
+
Do not run `--skip-upgrade-path-check` on this fixture to see the escape hatch work. It does work, and that is the problem: the RPC goes out and the node attempts a cross-minor downgrade, which is what the hint two lines above says ends in an A/B rollback or a node that will not boot. Exercise the flag only against a target the node can actually take, or on a stand you are willing to rebuild.
1371
+
1372
+
Watch for:
1373
+
1374
+
- A refusal on one minor back, which Talos supports: the guard must follow the matrix, not a version comparison.
1375
+
- A node that is unreachable blocking the upgrade: that case prints `warning: upgrade-path check skipped for <node>` and carries on.
1376
+
- A missing hint on the refusal: `errors.Join` drops the hints of the errors it joins, so the way out has to be attached to the joined error.
1377
+
1378
+
Restore `values.yaml` afterwards, from the copy E5 took before it edited anything:
1379
+
1380
+
```bash
1381
+
mv values.yaml.orig values.yaml
1382
+
```
1383
+
1384
+
E5 and E6 both edit `values.yaml`, so they must not share one backup: a second `sed -i.bak` overwrites the first backup with already-edited content, and the restore then puts back E5's value rather than the original. The sections that follow read `values.yaml::image`.
Copy file name to clipboardExpand all lines: docs/operations/safety-gates.md
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,7 @@
8
8
| Pre-apply drift preview |`--skip-drift-preview`| on |
9
9
| Post-apply state verification |`--skip-post-apply-verify`|**off**|
10
10
| Post-upgrade version verify |`--skip-post-upgrade-verify`| on |
11
+
| Pre-upgrade path check |`--skip-upgrade-path-check`| on |
11
12
12
13
## 1. Declared-resource existence
13
14
@@ -49,6 +50,14 @@ After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig a
49
50
50
51
After `talm upgrade` reports success, waits the configured reconcile window (default 90s; tune via `--post-upgrade-reconcile-window` for slow hardware / large image pulls) for the node to finish booting, then reads `runtime.Version` COSI and compares the running version's `(Major, Minor)` contract against the contract parsed from the target image tag. Point releases share a minor contract; cross-minor mismatch surfaces as a hint-bearing blocker. Catches the silent A/B rollback case where the upgrade RPC acks success but Talos rolled back to the previous partition (cross-vendor image, missing extensions, failed boot readiness check, slow boot exceeding the configured window). Best-effort surrender on digest-pinned images and unparseable tags.
51
52
53
+
## 5. Pre-upgrade path check
54
+
55
+
Before `talm upgrade` fires the RPC, reads `runtime.Version` COSI from every target node and asks Talos's own compatibility matrix (`pkg/machinery/compatibility`) whether the node can be moved to the target. That is the same matrix the Talos installer runs as its own pre-flight, so this gate answers with the authority rather than a version comparison of its own.
56
+
57
+
Two shapes are refused. A target too far forward — Talos accepts upgrades only from a bounded range of older versions — would otherwise fail inside the installer after the image has been pulled. A downgrade past what Talos allows (it supports at most one minor back) is invisible to gate 4, because once the downgrade takes, running equals target and the verify passes; the post-upgrade write-back then pins the node body to the older image.
58
+
59
+
The usual way in is a `values.yaml` nobody bumped, so the refusal carries Talos's own verdict and offers bumping it or passing `--image`. To try anyway, `--skip-upgrade-path-check`. Best-effort surrender on digest-pinned images, unparseable tags, and a target newer than any minor this talm build's matrix knows — that last one means the binary is too old to have an opinion, not that the path is wrong. A node whose version cannot be read is reported and skipped rather than blocked: an unreachable node is a worse reason to stop an upgrade than the move this prevents.
60
+
52
61
## Skip flags and the maintenance path
53
62
54
63
The skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.
0 commit comments