Skip to content

Commit 64e2f91

Browse files
committed
fix(upgrade): warn when a node body points at a different image
The upgrade target is values.yaml::image, so a body carrying something else is ignored and then overwritten by the post-upgrade write-back. Neither was announced, so an operator who bumped a node file instead of values.yaml upgraded to a version they had not read anywhere. This already surprised one operator and then a second, publicly. Report every -f body whose machine.install.image differs from the resolved target, naming both refs and what happens to the file. Files that carry no install image, or that do not parse, stay silent: side-patches and orphans in the -f list legitimately have neither, and a parse error surfaces from the write-back with a better message than a pre-flight warning could give. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-by: LLM
1 parent 5f13021 commit 64e2f91

5 files changed

Lines changed: 418 additions & 4 deletions

File tree

‎docs/manual-test-plan.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1320,6 +1320,38 @@ Expected: stderr emits the "Synced machine.install.image" line immediately after
13201320

13211321
Regression anchor: contract tests `TestContract_WriteBackInstallImage_*` pin the on-disk shape of the patch (scalar swap, idempotency, silent-skip on orphan files, structural errors, file-list fan-out). A regression that fires the write-back BEFORE verify (or instead of verify) would silently pin the body to an image the node never actually ran — the failure-path scenario above would catch it. Cross-reference: `pkg/commands/upgrade_image_writeback.go`.
13221322

1323+
### E5. Node body image divergence is reported
1324+
1325+
The upgrade target comes from `values.yaml`, so a node body naming something else is ignored and then rewritten. Both directions are reported, but only one of them is a problem.
1326+
1327+
```bash
1328+
# Point a node body at a NEWER Talos than values.yaml names.
1329+
sed -i.bak 's|image: ghcr.io/cozystack/cozystack/talos:.*|image: ghcr.io/cozystack/cozystack/talos:<A_NEWER_MINOR>|' nodes/node0.yaml
1330+
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
1331+
```
1332+
1333+
Do not pipe this into `head`. With `2>&1 |` talm's stderr is the pipe, `head` closes it after five lines, and a Go process takes SIGPIPE on fd 2 rather than an error — talm dies after the RPC has already fired, skipping the post-upgrade verify and the `install.image` write-back that E4 covers. The pipeline still exits 0, so nothing says it happened.
1334+
1335+
Expected: `Using image from values.yaml: <ref>` followed by a `warning:` line naming `nodes/node0.yaml`, the body's image, and the resolved target, telling you to put that image in values.yaml if the file is the one you meant. The warning fires because the body names a newer Talos than the target: the upgrade is going somewhere the file does not.
1336+
1337+
`<A_NEWER_MINOR>` has to be a newer **minor**, not a newer patch. The comparison is at major.minor, so `v1.13.1` against a `v1.13.0` target counts as trailing and prints the plain line instead.
1338+
1339+
Now the ordinary direction. Restore the body, then bump `values.yaml` forward a minor and leave the bodies as rendered, which is what an operator raising the cluster's version actually does:
1340+
1341+
```bash
1342+
mv nodes/node0.yaml.bak nodes/node0.yaml
1343+
sed -i.bak 's|^image: .*|image: "ghcr.io/cozystack/cozystack/talos:<A_NEWER_MINOR>"|' values.yaml
1344+
talm upgrade -f nodes/node0.yaml 2>upgrade.log; head -5 upgrade.log
1345+
```
1346+
1347+
Expected: the divergence reports as a plain line with no `warning:` label and no advice. That is the canonical flow, where every body trails the target until the write-back resyncs it, so advice to bump values.yaml would be advice to repeat what you just did.
1348+
1349+
Watch for:
1350+
1351+
- A `warning:` label on the trailing-body case, which is every node after a values.yaml bump.
1352+
- Silence on a body that names another registry at the same version: that one cannot be shown to trail, so it is a divergence.
1353+
- The report firing when the body already matches the target, or on a side-patch that declares no `install.image`.
1354+
13231355
## F. CA rotation
13241356

13251357
### F1. Rotate CA dry-run

‎docs/reference/upgrade.md‎

Lines changed: 9 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 249 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,249 @@
1+
// Copyright Cozystack Authors
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package commands
16+
17+
import (
18+
"bytes"
19+
"os"
20+
"path/filepath"
21+
"strings"
22+
"testing"
23+
24+
"github.com/spf13/cobra"
25+
)
26+
27+
// After a values.yaml bump every node body still names the previous image:
28+
// the command's own help calls that the canonical way to raise the cluster's
29+
// Talos version and says re-templating first is not required. So a body older
30+
// than the target is the normal shape, and telling that operator to "bump
31+
// values.yaml" is advice to repeat what they just did.
32+
//
33+
// The incident worth prescribing for is the other direction: values.yaml left
34+
// on an old preset default while the node file was edited to something
35+
// current. There the upgrade goes somewhere older than the file says, and the
36+
// file is then rewritten to match it.
37+
func TestContract_UpgradeDivergenceMessageMatchesTheDirection(t *testing.T) {
38+
const target = "ghcr.io/cozystack/cozystack/talos:v1.12.6"
39+
40+
write := func(t *testing.T, image string) string {
41+
t.Helper()
42+
43+
path := filepath.Join(t.TempDir(), "node0.yaml")
44+
body := "machine:\n type: controlplane\n install:\n disk: /dev/sda\n image: " + image + "\n"
45+
46+
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
47+
t.Fatal(err)
48+
}
49+
50+
return path
51+
}
52+
53+
t.Run("body behind the target is reported without prescribing", func(t *testing.T) {
54+
var buf bytes.Buffer
55+
56+
warnNodeBodyImageDivergence(&buf, []string{write(t, "ghcr.io/cozystack/cozystack/talos:v1.11.0")}, target)
57+
58+
out := buf.String()
59+
if !strings.Contains(out, "v1.11.0") || !strings.Contains(out, "v1.12.6") {
60+
t.Errorf("the stale body was not reported at all:\n%s", out)
61+
}
62+
63+
if strings.Contains(out, "Bump values.yaml") {
64+
t.Errorf("told the operator to bump values.yaml, which is what put the body behind:\n%s", out)
65+
}
66+
67+
if strings.Contains(out, "warning:") {
68+
t.Errorf("labelled the canonical flow as a warning:\n%s", out)
69+
}
70+
})
71+
72+
t.Run("body ahead of the target warns and prescribes", func(t *testing.T) {
73+
var buf bytes.Buffer
74+
75+
warnNodeBodyImageDivergence(&buf, []string{write(t, "ghcr.io/cozystack/cozystack/talos:v1.13.0")}, target)
76+
77+
out := buf.String()
78+
if !strings.Contains(out, "warning:") {
79+
t.Errorf("a body the upgrade will not honour was not flagged:\n%s", out)
80+
}
81+
82+
if !strings.Contains(out, "values.yaml") {
83+
t.Errorf("the way out was not named:\n%s", out)
84+
}
85+
})
86+
87+
// A patch ahead is the same incident one granularity down: the node moves
88+
// backwards, and the pre-upgrade guard lets it through because Talos allows
89+
// patch moves inside a minor. Ordering below the minor is what catches it.
90+
t.Run("body a patch ahead of the target warns", func(t *testing.T) {
91+
var buf bytes.Buffer
92+
93+
warnNodeBodyImageDivergence(&buf, []string{write(t, "ghcr.io/cozystack/cozystack/talos:v1.12.9")}, target)
94+
95+
if out := buf.String(); !strings.Contains(out, "warning:") {
96+
t.Errorf("a body a patch ahead was reported as the canonical flow:\n%s", out)
97+
}
98+
})
99+
100+
t.Run("body a patch behind the target is the canonical flow", func(t *testing.T) {
101+
var buf bytes.Buffer
102+
103+
warnNodeBodyImageDivergence(&buf, []string{write(t, "ghcr.io/cozystack/cozystack/talos:v1.12.1")}, target)
104+
105+
if out := buf.String(); strings.Contains(out, "warning:") {
106+
t.Errorf("a trailing patch was flagged as going somewhere else:\n%s", out)
107+
}
108+
})
109+
110+
t.Run("a body already on the target says nothing", func(t *testing.T) {
111+
var buf bytes.Buffer
112+
113+
warnNodeBodyImageDivergence(&buf, []string{write(t, target)}, target)
114+
115+
if out := buf.String(); out != "" {
116+
t.Errorf("reported a body that is already on the target, which is every node on an ordinary upgrade:\n%s", out)
117+
}
118+
})
119+
120+
t.Run("an empty image declares nothing", func(t *testing.T) {
121+
var buf bytes.Buffer
122+
123+
warnNodeBodyImageDivergence(&buf, []string{write(t, `""`)}, target)
124+
125+
if out := buf.String(); out != "" {
126+
t.Errorf("reported an empty image, which would print a message with a hole in it:\n%s", out)
127+
}
128+
})
129+
130+
t.Run("a body with no tag warns", func(t *testing.T) {
131+
var buf bytes.Buffer
132+
133+
warnNodeBodyImageDivergence(&buf, []string{write(t, "ghcr.io/cozystack/cozystack/talos")}, target)
134+
135+
if out := buf.String(); !strings.Contains(out, "warning:") {
136+
t.Errorf("an untagged body cannot be shown to trail the target, so it is a divergence:\n%s", out)
137+
}
138+
})
139+
140+
t.Run("a body naming another image warns", func(t *testing.T) {
141+
var buf bytes.Buffer
142+
143+
warnNodeBodyImageDivergence(&buf, []string{write(t, "registry.example.com/custom/talos:v1.12.6")}, target)
144+
145+
if out := buf.String(); !strings.Contains(out, "warning:") {
146+
t.Errorf("an image the upgrade will not go to was not flagged:\n%s", out)
147+
}
148+
})
149+
}
150+
151+
// The godoc promises silence on a file the walker cannot make sense of. Each
152+
// of these reaches a different surrender: an unreadable path, a stream that
153+
// does not parse, and a structurally wrong machine.install.image.
154+
func TestContract_UpgradeDivergenceWarningSurrendersQuietly(t *testing.T) {
155+
dir := t.TempDir()
156+
157+
badYAML := filepath.Join(dir, "broken.yaml")
158+
if err := os.WriteFile(badYAML, []byte("machine:\n\tinstall: [\n"), 0o600); err != nil {
159+
t.Fatal(err)
160+
}
161+
162+
wrongShape := filepath.Join(dir, "wrong-shape.yaml")
163+
if err := os.WriteFile(wrongShape, []byte("machine:\n install: \"oops\"\n"), 0o600); err != nil {
164+
t.Fatal(err)
165+
}
166+
167+
for _, tc := range []struct {
168+
name string
169+
path string
170+
}{
171+
{name: "unreadable file", path: filepath.Join(dir, "does-not-exist.yaml")},
172+
{name: "unparseable stream", path: badYAML},
173+
{name: "structurally wrong install", path: wrongShape},
174+
} {
175+
t.Run(tc.name, func(t *testing.T) {
176+
var buf bytes.Buffer
177+
178+
warnNodeBodyImageDivergence(&buf, []string{tc.path}, "ghcr.io/cozystack/cozystack/talos:v1.12.6")
179+
180+
if out := buf.String(); out != "" {
181+
t.Errorf("warned instead of surrendering:\n%s", out)
182+
}
183+
})
184+
}
185+
}
186+
187+
// The report only exists if the handler still calls it. Replacing the call
188+
// with `_ = filesToProcess` leaves every test above green, because they all
189+
// drive the function directly — so this one goes through cmd.RunE.
190+
//
191+
// The downgrade guard has the same pin for the same reason; without this the
192+
// two halves of the branch are covered unevenly.
193+
func TestContract_UpgradeDivergenceReportReachesTheOperator(t *testing.T) {
194+
root := t.TempDir()
195+
196+
// DetectAndSetRootFromFiles anchors on these two, so the -f file has to sit
197+
// inside something shaped like a talm project.
198+
for name, content := range map[string]string{
199+
"Chart.yaml": "apiVersion: v2\nname: t\nversion: 0.1.0\n",
200+
"secrets.yaml": "cluster:\n id: x\n",
201+
} {
202+
if err := os.WriteFile(filepath.Join(root, name), []byte(content), 0o600); err != nil {
203+
t.Fatal(err)
204+
}
205+
}
206+
207+
if err := os.WriteFile(filepath.Join(root, "values.yaml"),
208+
[]byte("image: ghcr.io/cozystack/cozystack/talos:v1.12.6\n"), 0o600); err != nil {
209+
t.Fatal(err)
210+
}
211+
212+
nodeFile := filepath.Join(root, "node0.yaml")
213+
body := "# talm: nodes=[\"192.0.2.10\"]\nmachine:\n type: controlplane\n install:\n image: ghcr.io/cozystack/cozystack/talos:v1.13.0\n"
214+
215+
if err := os.WriteFile(nodeFile, []byte(body), 0o600); err != nil {
216+
t.Fatal(err)
217+
}
218+
219+
savedRoot, savedVerify, savedNodes := Config.RootDir, upgradeCmdFlags.skipPostUpgradeVerify, GlobalArgs.Nodes
220+
221+
t.Cleanup(func() {
222+
Config.RootDir = savedRoot
223+
upgradeCmdFlags.skipPostUpgradeVerify = savedVerify
224+
GlobalArgs.Nodes = savedNodes
225+
})
226+
227+
Config.RootDir = root
228+
229+
cmd := &cobra.Command{Use: upgradeCmdName}
230+
cmd.Flags().StringP("image", "i", "", "")
231+
cmd.Flags().Bool("stage", false, "")
232+
cmd.Flags().StringSliceP("file", "f", nil, "")
233+
wrapUpgradeCommand(cmd, func(*cobra.Command, []string) error { return nil })
234+
235+
if err := cmd.Flags().Set("file", nodeFile); err != nil {
236+
t.Fatal(err)
237+
}
238+
239+
upgradeCmdFlags.skipPostUpgradeVerify = true
240+
241+
// The RunE error is deliberately ignored: the report is written during
242+
// image resolution, before anything that needs a cluster, so whatever
243+
// happens later does not affect what the operator already saw.
244+
out := captureStderr(t, func() { _ = cmd.RunE(cmd, nil) })
245+
246+
if !strings.Contains(out, "node0.yaml") || !strings.Contains(out, "v1.13.0") {
247+
t.Errorf("the divergence never reached the operator through the real flow:\n%s", out)
248+
}
249+
}

‎pkg/commands/upgrade_handler.go‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,8 +90,15 @@ Image resolution (when -f is provided):
9090
9191
The first -f file anchors the project root (Chart.yaml +
9292
secrets.yaml); its modeline supplies the nodes / endpoints. The
93-
node body's machine.install.image is no longer consulted by the
94-
upgrade flow.
93+
node body's machine.install.image no longer selects the target:
94+
it is read only to report it back, and rewritten afterwards.
95+
96+
- A body naming an older Talos than the target is the normal
97+
shape after a values.yaml bump, since refreshing node files
98+
first is not required. It is reported as a plain line.
99+
- A body naming a newer Talos, or an image from somewhere else,
100+
means the upgrade is not going where that file says. That one
101+
is a warning, and it names the way out.
95102
96103
Post-upgrade sync (when the upgrade succeeds):
97104
- talm point-patches machine.install.image in every -f node body
@@ -199,6 +206,7 @@ Post-upgrade sync (when the upgrade succeeds):
199206
fmt.Fprintf(os.Stderr, "Warning: failed to set --image flag: %v\n", err)
200207
} else {
201208
fmt.Fprintf(os.Stderr, "Using image from values.yaml: %s\n", image)
209+
warnNodeBodyImageDivergence(os.Stderr, filesToProcess, image)
202210
}
203211
}
204212

0 commit comments

Comments
 (0)