Skip to content

Commit 3dc254a

Browse files
committed
fix(cli): make strict mode block on unverifiable or missing drift baselines
decideDrift downgraded every check failure to a non-fatal warning even with strictCharts enforcement on, and a missing baseline passed even more quietly: deleting .talm-preset.lock or charts/talm/ was the cheapest bypass of the check. Under strict, a corrupted or unreadable baseline now aborts with a repair hint, and a missing one aborts with a pin-the-baseline hint (via an ErrNoBaseline sentinel the checks return); without strict the contracts are unchanged — failures degrade to a neutral "could not check drift" warning and a missing baseline stays silent, so pre-pinning projects are not nagged. The strict drift error no longer carries the warning's "(or ignore if this is intentional)" suffix, which contradicted the refusal to run. Touched not-exist checks moved to the unwrap-tolerant errors.Is idiom. Documented in README and the manual test plan. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent e28d34e commit 3dc254a

6 files changed

Lines changed: 227 additions & 37 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -259,7 +259,7 @@ WARN: project's vendored charts/talm/ library differs from the copy built into t
259259

260260
The remediation needs the preset name because `talm init --update` resolves the preset from `Chart.yaml`, which an init'd project does not record — pass `--preset <your-preset>` (the one you ran `talm init` with) explicitly.
261261

262-
Teams that want this enforced can turn the warning into a hard error (exit 1): set `strictCharts: true` in `Chart.yaml` so the whole team and CI inherit it, or pass `--strict-charts` for a single run. Strict mode applies to every config-loading command, including read-only ones such as `talm get` — run `talm init --update --preset <preset>`, or drop the flag / unset `strictCharts`, to unblock. The check stays silent for `dev`/source builds, whose embedded charts are a moving target the developer controls.
262+
Teams that want this enforced can turn the warning into a hard error (exit 1): set `strictCharts: true` in `Chart.yaml` so the whole team and CI inherit it, or pass `--strict-charts` for a single run. Strict mode applies to every config-loading command, including read-only ones such as `talm get` — run `talm init --update --preset <preset>`, or drop the flag / unset `strictCharts`, to unblock. Strict mode also escalates a check that cannot run at all — an unreadable `charts/talm/` or a corrupted `.talm-preset.lock` — into the same hard error, where the default behaviour degrades it to a `WARN: could not check drift` line: an unverifiable baseline passing silently would defeat the enforcement. A *missing* baseline (no `charts/talm/`, no `.talm-preset.lock`) blocks under strict for the same reason — deleting the baseline must not be a quieter bypass than corrupting it — while staying silent without strict, so projects generated before baseline pinning are not nagged. The check stays silent for `dev`/source builds, whose embedded charts are a moving target the developer controls.
263263

264264
### Preset drift
265265

@@ -276,7 +276,7 @@ A release build then compares the binary's *current* preset hash against that pi
276276
WARN: project's cozystack preset differs from the copy built into talm <version>; run `talm init --update --preset cozystack` to pull the new preset defaults (your templates/ edits are preserved via the interactive diff)
277277
```
278278

279-
`talm init --update --preset <preset>` shows you an interactive diff of the new preset against your `templates/`, lets you merge what you want, and advances the baseline — which clears the warning even if you decline individual diffs to keep your customizations. `--strict-charts` / `strictCharts: true` escalate this to a hard error exactly as for the library. Projects with no `.talm-preset.lock` (generated before preset pinning) stay silent — there is no baseline to compare. Commit `.talm-preset.lock` so the baseline is shared across your team.
279+
`talm init --update --preset <preset>` shows you an interactive diff of the new preset against your `templates/`, lets you merge what you want, and advances the baseline — which clears the warning even if you decline individual diffs to keep your customizations. `--strict-charts` / `strictCharts: true` escalate this to a hard error exactly as for the library. Projects with no `.talm-preset.lock` (generated before preset pinning) stay silent — there is no baseline to compare — unless strict mode is on, which treats a missing baseline as a blocker. Commit `.talm-preset.lock` so the baseline is shared across your team.
280280

281281
## Apply with side-patches
282282

‎docs/manual-test-plan.md‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -272,7 +272,7 @@ talm template -f nodes/node0.yaml --offline 2>&1 >/dev/null | grep '^WARN:.*pres
272272

273273
`init --update` rewrites `.talm-preset.lock` to the current baseline, so the warning clears even if you declined individual preset-file diffs. Expected: `OK: preset drift cleared`.
274274

275-
Regression anchors: `TestPresetLock_RoundTrip_NoDrift`, `TestCheckPresetDrift_StaleBaseline_Drift`, `TestCheckPresetDrift_OperatorEditedTemplates_NoDrift` (the false-positive guard), `TestCheckPresetDrift_NoLock_Silent`, and `TestEvaluatePresetDrift` pin every branch above.
275+
Regression anchors: `TestPresetLock_RoundTrip_NoDrift`, `TestCheckPresetDrift_StaleBaseline_Drift`, `TestCheckPresetDrift_OperatorEditedTemplates_NoDrift` (the false-positive guard), `TestCheckPresetDrift_NoLock_NoBaselineSentinel`, and `TestEvaluatePresetDrift` pin every branch above.
276276

277277
Strict mode turns the same drift into a hard error (exit 1) raised before the command body runs. Opt in per project via `strictCharts: true` in `Chart.yaml`, or per invocation via `--strict-charts`:
278278

@@ -282,11 +282,36 @@ talm template -f nodes/node0.yaml --offline --strict-charts; echo "exit=$?"
282282

283283
Expected: exit 1, the drift message plus a hint pointing at `talm init --update --preset <preset>`, and no rendered output — the command body never runs (no modeline/render errors follow the hint).
284284

285+
### B5b. Drift check failure: warn by default, block under strict
286+
287+
A check that cannot run at all (corrupted `.talm-preset.lock`, unreadable `charts/talm/`) degrades to a non-fatal warning by default, but under strict mode it is a hard error — an unverifiable baseline silently passing would defeat the enforcement the team opted into:
288+
289+
```bash
290+
printf 'preset: [unclosed\n' > .talm-preset.lock
291+
talm template -f nodes/node0.yaml --offline 2>&1 >/dev/null | grep '^WARN: could not check drift'; echo "exit=$?"
292+
talm template -f nodes/node0.yaml --offline --strict-charts >/dev/null; echo "exit=$?"
293+
talm init --update --preset <your-preset> --force # restore a valid lock
294+
```
295+
296+
Expected: the first render prints `WARN: could not check drift: ...` and exits 0; the `--strict-charts` render fails (exit 1) with `drift check failed under strict mode` and a hint to repair the baseline; `init --update` rewrites a valid lock. Regression anchors: the error-path subtests of `TestEvaluateChartDrift` / `TestEvaluatePresetDrift`.
297+
298+
A *missing* baseline follows the same split — silent by default (projects generated before baseline pinning are not nagged), blocked under strict (deleting the baseline must not bypass enforcement):
299+
300+
```bash
301+
rm .talm-preset.lock
302+
talm template -f nodes/node0.yaml --offline 2>&1 >/dev/null | grep '^WARN:'; echo "exit=$?" # no WARN, exit 0 from talm
303+
talm template -f nodes/node0.yaml --offline --strict-charts >/dev/null; echo "exit=$?" # exit 1: drift baseline missing under strict mode
304+
talm init --update --preset <your-preset> --force # re-pin the baseline
305+
```
306+
307+
Regression anchors: the missing-baseline subtests of `TestEvaluateChartDrift` / `TestEvaluatePresetDrift`, `TestCheckChartDrift_MissingVendoredDir_NoBaselineSentinel`, `TestCheckPresetDrift_NoLock_NoBaselineSentinel`.
308+
285309
No-false-alarm checks — each MUST stay silent (no `WARN:` line):
286310

287311
- A project vendored by an older release, run under a newer binary whose `charts/talm/` is byte-identical: the version stamp differs but the content does not.
288312
- A `dev`/source build: its embedded charts are a moving target the developer controls, so the check is a no-op.
289313
- A freshly synced project (`talm init --update --preset <preset>`) under the matching binary.
314+
- A vendored `charts/talm/` whose files carry CRLF line endings (a Windows clone with `core.autocrlf=true`): line endings are checkout artifacts, not content (pinned by `TestCheckChartDrift_CRLFVendoredCopy_NoDrift`).
290315

291316
### B6. Scope-filter symmetry across v1.11 and v1.12 renders
292317

‎main.go‎

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -299,15 +299,41 @@ func evaluatePresetDrift(rawVersion, rootDir string, strict bool) (string, error
299299
}
300300

301301
// decideDrift folds a (drift, msg, err) result into the (warning, error)
302-
// outcome shared by both drift checks: an I/O error downgrades to a non-fatal
303-
// warning (best effort, never blocks a command); drift under strict is a hard
304-
// error with a remediation hint; drift otherwise is a warning; no drift is
305-
// silent.
302+
// outcome shared by both drift checks: a check failure downgrades to a
303+
// non-fatal warning (best effort, never blocks a command) — except under
304+
// strict, where an unverifiable baseline is a hard error: the operator opted
305+
// into enforcement, and a corrupted lock or unreadable vendored tree passing
306+
// silently would defeat it exactly when the baseline broke; a MISSING
307+
// baseline (commands.ErrNoBaseline) is silence without strict — projects
308+
// from before baseline pinning should not nag — but a blocker under strict,
309+
// where deleting the baseline must not pass more quietly than corrupting
310+
// it; drift under strict is a hard error with a remediation hint; drift
311+
// otherwise is a warning; no drift is silent.
306312
func decideDrift(drift bool, msg string, err error, strict bool) (string, error) {
307313
switch {
314+
case errors.Is(err, commands.ErrNoBaseline) && !strict:
315+
return "", nil
316+
case errors.Is(err, commands.ErrNoBaseline):
317+
//nolint:wrapcheck // cockroachdb/errors.WithHint at boundary; project idiom.
318+
return "", errors.WithHint(
319+
errors.Wrap(err, "drift baseline missing under strict mode"),
320+
"run `talm init --update --preset <preset>` to vendor the library and pin the preset baseline, or unset strictCharts / drop --strict-charts",
321+
)
322+
case err != nil && strict:
323+
//nolint:wrapcheck // cockroachdb/errors.WithHint at boundary; project idiom.
324+
return "", errors.WithHint(
325+
errors.Wrap(err, "drift check failed under strict mode"),
326+
"repair the baseline (re-run `talm init --update --preset <preset>`), or unset strictCharts / drop --strict-charts to downgrade this to a warning",
327+
)
308328
case err != nil:
309-
return fmt.Sprintf("could not check chart drift: %v", err), nil
329+
return fmt.Sprintf("could not check drift: %v", err), nil
310330
case drift && strict:
331+
// The shared drift message ends with "(or ignore if this is
332+
// intentional)" — sound advice on a warning, contradictory on an
333+
// error the command just refused to run past. Strip it here
334+
// rather than threading a second message through both checkers.
335+
msg = strings.Replace(msg, " (or ignore if this is intentional)", "", 1)
336+
311337
//nolint:wrapcheck // originating error built with errors.New; WithHint adds operator-facing guidance and is the project idiom.
312338
return "", errors.WithHint(
313339
errors.New(msg),

‎main_test.go‎

Lines changed: 124 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -339,6 +339,9 @@ func TestEvaluateChartDrift(t *testing.T) {
339339
if !hintsContain(err, "talm init --update --preset") {
340340
t.Errorf("strict-mode error must hint at `talm init --update --preset`, hints: %v", errors.GetAllHints(err))
341341
}
342+
if strings.Contains(err.Error(), "or ignore if this is intentional") {
343+
t.Errorf("strict-mode error must not carry the warning's ignore suggestion — the command just refused to run: %v", err)
344+
}
342345
})
343346

344347
t.Run("non-strict drift warns without blocking", func(t *testing.T) {
@@ -353,6 +356,73 @@ func TestEvaluateChartDrift(t *testing.T) {
353356
}
354357
})
355358

359+
t.Run("unreadable vendored tree under strict blocks the command", func(t *testing.T) {
360+
// charts/talm as a file: the drift check cannot read the tree.
361+
// Strict mode exists for enforcement; an unverifiable baseline
362+
// passing silently would defeat it exactly when the project is
363+
// broken.
364+
root := t.TempDir()
365+
if err := os.MkdirAll(filepath.Join(root, "charts"), 0o755); err != nil {
366+
t.Fatalf("mkdir: %v", err)
367+
}
368+
if err := os.WriteFile(filepath.Join(root, "charts", "talm"), []byte("not a directory"), 0o644); err != nil {
369+
t.Fatalf("write file: %v", err)
370+
}
371+
372+
warning, err := evaluateChartDrift("0.30.0", root, true)
373+
if err == nil {
374+
t.Fatal("a drift check failure must block under strict mode, not silently pass")
375+
}
376+
if warning != "" {
377+
t.Errorf("strict error path must not also emit a warning, got %q", warning)
378+
}
379+
})
380+
381+
t.Run("unreadable vendored tree without strict downgrades to a warning", func(t *testing.T) {
382+
root := t.TempDir()
383+
if err := os.MkdirAll(filepath.Join(root, "charts"), 0o755); err != nil {
384+
t.Fatalf("mkdir: %v", err)
385+
}
386+
if err := os.WriteFile(filepath.Join(root, "charts", "talm"), []byte("not a directory"), 0o644); err != nil {
387+
t.Fatalf("write file: %v", err)
388+
}
389+
390+
warning, err := evaluateChartDrift("0.30.0", root, false)
391+
if err != nil {
392+
t.Fatalf("a non-strict drift check failure must not block the command: %v", err)
393+
}
394+
if !strings.Contains(warning, "could not check drift") {
395+
t.Errorf("expected a could-not-check warning, got %q", warning)
396+
}
397+
})
398+
399+
t.Run("missing vendored library under strict blocks the command", func(t *testing.T) {
400+
// Deleting charts/talm/ is the cheapest bypass of the check —
401+
// under opt-in enforcement a missing baseline must block, not
402+
// pass more quietly than a corrupted one.
403+
root := t.TempDir()
404+
405+
warning, err := evaluateChartDrift("0.30.0", root, true)
406+
if err == nil {
407+
t.Fatal("a missing vendored library must block under strict mode, not silently pass")
408+
}
409+
if warning != "" {
410+
t.Errorf("strict error path must not also emit a warning, got %q", warning)
411+
}
412+
if !hintsContain(err, "talm init --update --preset") {
413+
t.Errorf("strict-mode error must hint at `talm init --update --preset`, hints: %v", errors.GetAllHints(err))
414+
}
415+
})
416+
417+
t.Run("missing vendored library without strict is silent", func(t *testing.T) {
418+
root := t.TempDir()
419+
420+
warning, err := evaluateChartDrift("0.30.0", root, false)
421+
if err != nil || warning != "" {
422+
t.Errorf("a project with nothing vendored must stay silent without strict, got warning=%q err=%v", warning, err)
423+
}
424+
})
425+
356426
t.Run("dev build is silent even with drift present and strict set", func(t *testing.T) {
357427
root := writeDriftedTalmProject(t)
358428

@@ -425,12 +495,63 @@ func TestEvaluatePresetDrift(t *testing.T) {
425495
}
426496
})
427497

428-
t.Run("project without a preset lock is silent", func(t *testing.T) {
498+
t.Run("project without a preset lock is silent without strict", func(t *testing.T) {
429499
root := t.TempDir()
430500

431-
warning, err := evaluatePresetDrift("0.30.0", root, true)
501+
warning, err := evaluatePresetDrift("0.30.0", root, false)
432502
if err != nil || warning != "" {
433-
t.Errorf("a project with no preset lock must be silent, got warning=%q err=%v", warning, err)
503+
t.Errorf("a project with no preset lock must be silent without strict, got warning=%q err=%v", warning, err)
504+
}
505+
})
506+
507+
t.Run("missing preset lock under strict blocks the command", func(t *testing.T) {
508+
// A merge conflict resolved as "delete .talm-preset.lock" must
509+
// not pass MORE quietly than a corrupted lock — under opt-in
510+
// enforcement a missing baseline is indistinguishable from a
511+
// deleted one.
512+
root := t.TempDir()
513+
514+
warning, err := evaluatePresetDrift("0.30.0", root, true)
515+
if err == nil {
516+
t.Fatal("a missing preset baseline must block under strict mode, not silently pass")
517+
}
518+
if warning != "" {
519+
t.Errorf("strict error path must not also emit a warning, got %q", warning)
520+
}
521+
if !hintsContain(err, "talm init --update --preset") {
522+
t.Errorf("strict-mode error must hint at `talm init --update --preset`, hints: %v", errors.GetAllHints(err))
523+
}
524+
})
525+
526+
t.Run("malformed lock under strict blocks the command", func(t *testing.T) {
527+
// A lock corrupted by a bad merge is exactly the moment the
528+
// team's strictCharts enforcement must fire, not silently pass.
529+
root := t.TempDir()
530+
if err := os.WriteFile(filepath.Join(root, ".talm-preset.lock"), []byte("preset: [unclosed\n"), 0o644); err != nil {
531+
t.Fatalf("write lock: %v", err)
532+
}
533+
534+
warning, err := evaluatePresetDrift("0.30.0", root, true)
535+
if err == nil {
536+
t.Fatal("an unreadable preset baseline must block under strict mode, not silently pass")
537+
}
538+
if warning != "" {
539+
t.Errorf("strict error path must not also emit a warning, got %q", warning)
540+
}
541+
})
542+
543+
t.Run("malformed lock without strict downgrades to a warning", func(t *testing.T) {
544+
root := t.TempDir()
545+
if err := os.WriteFile(filepath.Join(root, ".talm-preset.lock"), []byte("preset: [unclosed\n"), 0o644); err != nil {
546+
t.Fatalf("write lock: %v", err)
547+
}
548+
549+
warning, err := evaluatePresetDrift("0.30.0", root, false)
550+
if err != nil {
551+
t.Fatalf("a non-strict drift check failure must not block the command: %v", err)
552+
}
553+
if !strings.Contains(warning, "could not check drift") {
554+
t.Errorf("expected a could-not-check warning, got %q", warning)
434555
}
435556
})
436557
}

0 commit comments

Comments
 (0)