Skip to content

Commit dac1d2c

Browse files
authored
Merge pull request #243 from cozystack/feat/upgrade-image-visibility
feat(upgrade): make the install image resolution visible and guarded
2 parents 7fb8fb8 + 7203b41 commit dac1d2c

10 files changed

Lines changed: 945 additions & 8 deletions

‎docs/manual-test-plan.md‎

Lines changed: 64 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1166,7 +1166,7 @@ go vet ./...
11661166
### Known limitations / follow-ups
11671167

11681168
- **Talos-mutated-field allowlist**: Phase 2B reports cert hashes / timestamps as divergence today; the verify is off by default until an allowlist lands.
1169-
- **`talm upgrade` has no pre-upgrade gates** (Phase 2C runs *after*, not before): the upgrade flow wraps `talosctl upgrade` and doesn't route through `buildApplyClosure` / `applyOneFileDirectPatchMode`, so Phase 1 / Phase 2A do not run. Phase 2C (post-upgrade version verify) was added precisely to catch the silent-rollback class without that refactor. Full pre-upgrade gates would require reproducing the gate calls in `upgrade_handler.go` or refactoring the apply flow.
1169+
- **`talm upgrade` does not run the apply-path gates**: the upgrade flow wraps `talosctl upgrade` and doesn't route through `buildApplyClosure` / `applyOneFileDirectPatchMode`, so Phase 1 / Phase 2A do not run. Two gates of its own cover the classes that matter most: the pre-upgrade path check refuses a move Talos will not make, and Phase 2C catches the silent rollback afterwards. Bringing the apply gates over would still require reproducing the gate calls in `upgrade_handler.go` or refactoring the apply flow.
11701170
- **Phase 1/2 on `--insecure`**: the safety gates can't run before the chart renders, and the chart's `lookup` calls need an authenticated COSI connection. Insecure path = effectively no gates today.
11711171

11721172
## D. Apply (insecure / maintenance path)
@@ -1320,6 +1320,69 @@ Expected: stderr emits the "Synced machine.install.image" line immediately after
13201320

13211321
Regression anchor: contract tests `TestContract_WriteBackInstallImage_*` pin the on-disk shape of the patch (scalar swap, idempotency, silent-skip on orphan files, structural errors, file-list fan-out). A regression that fires the write-back BEFORE verify (or instead of verify) would silently pin the body to an image the node never actually ran — the failure-path scenario above would catch it. Cross-reference: `pkg/commands/upgrade_image_writeback.go`.
13221322

1323+
1324+
### E5. Node body image divergence is reported
1325+
1326+
The upgrade target comes from `values.yaml`, so a node body naming something else is ignored and then rewritten. Both directions are reported, but only one of them is a problem.
1327+
1328+
```bash
1329+
# Point a node body at a NEWER Talos than values.yaml names.
1330+
sed -i.bak 's|image: ghcr.io/cozystack/cozystack/talos:.*|image: ghcr.io/cozystack/cozystack/talos:<A_NEWER_TAG>|' nodes/node0.yaml
1331+
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
1332+
```
1333+
1334+
Do not pipe this into `head`. With `2>&1 |` talm's stderr is the pipe, `head` closes it after five lines, and a Go process takes SIGPIPE on fd 2 rather than an error — talm dies after the RPC has already fired, skipping the post-upgrade verify and the `install.image` write-back that E4 covers. The pipeline still exits 0, so nothing says it happened.
1335+
1336+
Expected: `Using image from values.yaml: <ref>` followed by a `warning:` line naming `nodes/node0.yaml`, the body's image, and the resolved target, telling you to put that image in values.yaml if the file is the one you meant. The warning fires because the body names a newer Talos than the target: the upgrade is going somewhere the file does not.
1337+
1338+
`<A_NEWER_TAG>` only has to be newer than what `values.yaml` names — the comparison is ordered down to the patch, so `v1.13.1` against a `v1.13.0` target warns just like a newer minor does. A node moving backwards by a patch is the same incident one granularity down, and the pre-upgrade path check lets it through because Talos allows patch moves inside a minor.
1339+
1340+
Now the ordinary direction. Restore the body, then bump `values.yaml` forward a minor and leave the bodies as rendered, which is what an operator raising the cluster's version actually does:
1341+
1342+
```bash
1343+
mv nodes/node0.yaml.bak nodes/node0.yaml
1344+
cp values.yaml values.yaml.orig
1345+
sed -i '' 's|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:<A_NEWER_TAG>"|' values.yaml
1346+
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
1347+
```
1348+
1349+
Expected: the divergence reports as a plain line with no `warning:` label and no advice. That is the canonical flow, where every body trails the target until the write-back resyncs it, so advice to bump values.yaml would be advice to repeat what you just did.
1350+
1351+
Watch for:
1352+
1353+
- A `warning:` label on the trailing-body case, which is every node after a values.yaml bump.
1354+
- Silence on a body that names another registry at the same version: that one cannot be shown to trail, so it is a divergence.
1355+
- The report firing when the body already matches the target, or on a side-patch that declares no `install.image`.
1356+
1357+
### E6. Pre-upgrade path check
1358+
1359+
The refusal itself stops before the RPC, so that half needs no node that reboots. The escape-hatch flag is a different matter — see the warning below it.
1360+
1361+
Pick a target Talos actually refuses: two minors back, or several minors forward. One minor back is a supported rollback and must NOT be blocked.
1362+
1363+
```bash
1364+
sed -i '' 's|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:v1.10.5"|' values.yaml
1365+
talm upgrade -f nodes/node0.yaml
1366+
```
1367+
1368+
Expected: the upgrade is refused with `refusing this upgrade: host version <running> is too new to downgrade to Talos 1.10.5`, and a hint offering the two ways out. Nothing is sent to the node.
1369+
1370+
Do not run `--skip-upgrade-path-check` on this fixture to see the escape hatch work. It does work, and that is the problem: the RPC goes out and the node attempts a cross-minor downgrade, which is what the hint two lines above says ends in an A/B rollback or a node that will not boot. Exercise the flag only against a target the node can actually take, or on a stand you are willing to rebuild.
1371+
1372+
Watch for:
1373+
1374+
- A refusal on one minor back, which Talos supports: the guard must follow the matrix, not a version comparison.
1375+
- A node that is unreachable blocking the upgrade: that case prints `warning: upgrade-path check skipped for <node>` and carries on.
1376+
- A missing hint on the refusal: `errors.Join` drops the hints of the errors it joins, so the way out has to be attached to the joined error.
1377+
1378+
Restore `values.yaml` afterwards, from the copy E5 took before it edited anything:
1379+
1380+
```bash
1381+
mv values.yaml.orig values.yaml
1382+
```
1383+
1384+
E5 and E6 both edit `values.yaml`, so they must not share one backup: a second `sed -i.bak` overwrites the first backup with already-edited content, and the restore then puts back E5's value rather than the original. The sections that follow read `values.yaml::image`.
1385+
13231386
## F. CA rotation
13241387

13251388
### F1. Rotate CA dry-run

‎docs/operations/safety-gates.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
| Pre-apply drift preview | `--skip-drift-preview` | on |
99
| Post-apply state verification | `--skip-post-apply-verify` | **off** |
1010
| Post-upgrade version verify | `--skip-post-upgrade-verify` | on |
11+
| Pre-upgrade path check | `--skip-upgrade-path-check` | on |
1112

1213
## 1. Declared-resource existence
1314

@@ -49,6 +50,14 @@ After `ApplyConfiguration` returns success, re-reads the on-node MachineConfig a
4950

5051
After `talm upgrade` reports success, waits the configured reconcile window (default 90s; tune via `--post-upgrade-reconcile-window` for slow hardware / large image pulls) for the node to finish booting, then reads `runtime.Version` COSI and compares the running version's `(Major, Minor)` contract against the contract parsed from the target image tag. Point releases share a minor contract; cross-minor mismatch surfaces as a hint-bearing blocker. Catches the silent A/B rollback case where the upgrade RPC acks success but Talos rolled back to the previous partition (cross-vendor image, missing extensions, failed boot readiness check, slow boot exceeding the configured window). Best-effort surrender on digest-pinned images and unparseable tags.
5152

53+
## 5. Pre-upgrade path check
54+
55+
Before `talm upgrade` fires the RPC, reads `runtime.Version` COSI from every target node and asks Talos's own compatibility matrix (`pkg/machinery/compatibility`) whether the node can be moved to the target. That is the same matrix the Talos installer runs as its own pre-flight, so this gate answers with the authority rather than a version comparison of its own.
56+
57+
Two shapes are refused. A target too far forward — Talos accepts upgrades only from a bounded range of older versions — would otherwise fail inside the installer after the image has been pulled. A downgrade past what Talos allows (it supports at most one minor back) is invisible to gate 4, because once the downgrade takes, running equals target and the verify passes; the post-upgrade write-back then pins the node body to the older image.
58+
59+
The usual way in is a `values.yaml` nobody bumped, so the refusal carries Talos's own verdict and offers bumping it or passing `--image`. To try anyway, `--skip-upgrade-path-check`. Best-effort surrender on digest-pinned images, unparseable tags, and a target newer than any minor this talm build's matrix knows — that last one means the binary is too old to have an opinion, not that the path is wrong. A node whose version cannot be read is reported and skipped rather than blocked: an unreachable node is a worse reason to stop an upgrade than the move this prevents.
60+
5261
## Skip flags and the maintenance path
5362

5463
The skip flags don't suppress each other — pass them independently. On the `--insecure` (maintenance) path the gates are functionally unreachable for charts that drive discovery via `lookup` — those COSI lookups require an authenticated connection and the render itself errors before any gate runs. Charts that render fully offline (no `lookup` calls) reach the gates on `--insecure` as well, with the Phase 2 hooks degrading gracefully because the `MachineConfig` resource is Sensitive.

‎docs/reference/upgrade.md‎

Lines changed: 24 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ require (
66
filippo.io/age v1.3.1
77
github.com/BurntSushi/toml v1.6.0
88
github.com/Masterminds/sprig/v3 v3.3.0
9+
github.com/blang/semver/v4 v4.0.0
910
github.com/cockroachdb/errors v1.14.0
1011
github.com/cosi-project/runtime v1.16.3
1112
github.com/dustin/go-humanize v1.0.1
@@ -70,7 +71,6 @@ require (
7071
github.com/aws/aws-sdk-go-v2/service/sts v1.45.4 // indirect
7172
github.com/aws/smithy-go v1.27.7 // indirect
7273
github.com/beorn7/perks v1.0.1 // indirect
73-
github.com/blang/semver/v4 v4.0.0 // indirect
7474
github.com/brianvoe/gofakeit/v7 v7.14.1 // indirect
7575
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
7676
github.com/cespare/xxhash/v2 v2.3.0 // indirect

0 commit comments

Comments
 (0)