Skip to content

Commit f209c47

Browse files
authored
test(e2e): read the opensearch release history through the pinned name (cozystack#4689)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The opensearch suite now reads HelmRelease history through the shared jsonpath, and a unit test fails if any Chainsaw suite goes back to a copied literal. Before this, the suite carried its own copy of the expression, and the upstream contract test in `internal/fluxcontract` only checks `HELMRELEASE_HISTORY_JSONPATH`. I checked the copy against helm-controller `api` v1.5.1, the version go.mod pins: the history field is `history` and each snapshot has `status`. So the read works today, it was just not pinned. The suite passes `-o "jsonpath=${HELMRELEASE_HISTORY_JSONPATH}"` now, which gives kubectl exactly the same argument as before. The new test reads every suite under `hack/e2e-chainsaw/`. A suite that calls `helmrelease_has_remediation_cycle` must fill the local it hands the guard from the shared name, and no suite may assign that name itself. It reuses the matchers that already pin `run-kubernetes.sh`. The test is red on main and green with the fix. One known limit. The second-assignment check counts over the whole file, which fits `run-kubernetes.sh` because it runs as one shell. A Chainsaw suite runs each `script` step in its own shell, so an unrelated `statuses=` in another opensearch step would turn the test red with nothing broken. That fails toward red, never toward a silent pass, so I left it. Closes cozystack#4550 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note test(e2e): the opensearch suite reads HelmRelease history through the shared jsonpath, which is now pinned against the upstream Flux type for every Chainsaw suite that runs the remediation guard. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded validation across Chainsaw end-to-end suites to check consistent HelmRelease history handling in remediation checks, including the order and isolation of script steps. * The OpenSearch suite now uses the shared history lookup. Empty-history checks and remediation-cycle failure handling are unchanged, and there is no change to end-user behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b4fbaf0 + 2df578b commit f209c47

2 files changed

Lines changed: 221 additions & 10 deletions

File tree

‎hack/e2e-chainsaw/opensearch/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -551,7 +551,7 @@ spec:
551551
cd ../../..
552552
. hack/e2e-chainsaw/_lib/remediation-guard.sh
553553
statuses=$(kubectl -n "$NAMESPACE" get helmrelease opensearch-test \
554-
-o jsonpath='{range .status.history[*]}{.status}{"\n"}{end}')
554+
-o "jsonpath=${HELMRELEASE_HISTORY_JSONPATH}")
555555
# Always emit the raw value so a silent future-Flux field rename shows
556556
# up as "empty history on a Ready HR" in CI logs rather than vanishing.
557557
# The helper reads empty input as "no cycle observed", so without this

‎internal/fluxcontract/snapshot_shape_test.go‎

Lines changed: 220 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,12 @@
2424
package fluxcontract
2525

2626
import (
27+
"bufio"
28+
"bytes"
2729
"encoding/json"
2830
"errors"
31+
"fmt"
32+
"io"
2933
"os"
3034
"os/exec"
3135
"path/filepath"
@@ -34,14 +38,20 @@ import (
3438
"testing"
3539

3640
helmv2 "github.com/fluxcd/helm-controller/api/v2"
41+
utilyaml "k8s.io/apimachinery/pkg/util/yaml"
3742
"k8s.io/client-go/util/jsonpath"
43+
"sigs.k8s.io/yaml"
3844
)
3945

4046
const (
4147
// The shell library that owns the expression, and the script that runs it.
4248
guardLibrary = "../../hack/e2e-chainsaw/_lib/remediation-guard.sh"
4349
guardCaller = "../../hack/e2e-chainsaw/_lib/run-kubernetes.sh"
4450

51+
// The Chainsaw suites, one directory each, which may call the guard from a
52+
// script step.
53+
chainsawSuites = "../../hack/e2e-chainsaw"
54+
4555
sharedName = "HELMRELEASE_HISTORY_JSONPATH"
4656
)
4757

@@ -73,11 +83,13 @@ var nameFollowedByEquals = regexp.MustCompile(regexp.QuoteMeta(sharedName) + `=`
7383
// free, because this script mixes forms that behave identically. Each is a
7484
// case in the accept table below.
7585
//
76-
// The reach is one named caller and calls that pass a local. A second script
77-
// calling the guard, or this one passing a substitution written out in the
78-
// argument, is outside what this sees. Neither exists today, and the rule for
79-
// authors is prose rather than a pattern: docs/agents/e2e-testing.md tells them
80-
// to take the expression from the shared name.
86+
// The reach is one named caller, the Chainsaw suites, and calls that pass a
87+
// local. Any other script that reads history for the guard, or a caller passing
88+
// a substitution written out in the argument, is outside what this sees. Neither
89+
// exists today (the bats tests hand the guard literal input and read no
90+
// history), and the rule for authors is prose rather than a pattern:
91+
// docs/agents/e2e-testing.md tells them to take the expression from the shared
92+
// name.
8193
var guardConsumers = regexp.MustCompile(
8294
`(?m)^[ \t]*(?:if[ \t]+)?(?:![ \t]*)?helmrelease_has_remediation_cycle[ \t]+"?\$\{?([A-Za-z_][A-Za-z0-9_]*)`)
8395

@@ -471,31 +483,230 @@ func TestGuardCallerReferencesTheSharedExpansion(t *testing.T) {
471483

472484
src := readGuardFile(t, guardCaller)
473485

474-
consumers := guardConsumers.FindAllSubmatch(src, -1)
486+
consumers := guardConsumers.FindAllSubmatchIndex(src, -1)
475487
if len(consumers) == 0 {
476488
t.Fatalf("%s never calls helmrelease_has_remediation_cycle with an expanded local, so there "+
477489
"is no read for this pin to follow. Either the caller stopped running the guard, or it "+
478490
"calls it in a shape %s cannot see.", guardCaller, guardConsumers)
479491
}
480492

493+
assertConsumedLocalsTakeTheSharedRead(t, guardCaller, src, consumers)
494+
}
495+
496+
func assertConsumedLocalsTakeTheSharedRead(t testing.TB, path string, src []byte, consumers [][]int) {
497+
t.Helper()
481498
for _, m := range consumers {
482-
local := string(m[1])
499+
local := string(src[m[2]:m[3]])
483500
n := len(localFollowedByEquals(local).FindAll(src, -1))
484501
if n != 1 {
485502
t.Errorf("%s writes %s, the local it hands the guard, followed by = %d times; want "+
486503
"exactly one. The count is of occurrences, so a comment mentioning one counts too; "+
487504
"what it looks for is a second assignment, which the shell would take instead of "+
488505
"the one this pin matched, putting an unpinned expression in front of the guard.",
489-
guardCaller, local, n)
506+
path, local, n)
490507
continue
491508
}
492509
if !sharedReadBy(local).Match(src) {
493510
t.Errorf("%s fills %s, the local it hands the guard, in a shape that does not take its "+
494511
"expression from %s. The pattern is the exact form accepted: %s. Either the read "+
495512
"stopped using the shared assignment, or it uses it in a shape the pattern cannot "+
496-
"see.", guardCaller, local, sharedName, sharedReadBy(local))
513+
"see.", path, local, sharedName, sharedReadBy(local))
514+
continue
515+
}
516+
if !sharedReadPrecedes(src, m) {
517+
t.Errorf("%s fills %s from %s only after handing it to the guard, so the guard runs "+
518+
"on whatever the local held before the read.", path, local, sharedName)
519+
}
520+
}
521+
}
522+
523+
// sharedReadPrecedes reports whether the shared read of the local consumed by
524+
// the guard call m starts before that call. m holds the call's offsets as
525+
// FindAllSubmatchIndex returns them, so the call is placed where it matched and
526+
// not wherever its text first appears, a comment included. Text order stands
527+
// in for run order, which holds for a read and a call in one straight-line
528+
// block, as in every caller this file reads today.
529+
func sharedReadPrecedes(src []byte, m []int) bool {
530+
read := sharedReadBy(string(src[m[2]:m[3]])).FindIndex(src)
531+
return read != nil && read[0] < m[0]
532+
}
533+
534+
// A Chainsaw suite that sources the guard library from a script step is a
535+
// caller too, and one this file would otherwise never read: the pin above names
536+
// one script, so a suite writing the jsonpath out as a literal runs an
537+
// expression nothing here checked.
538+
//
539+
// Every suite is read rather than a named one, because the next suite to adopt
540+
// the guard is the likeliest place for a copied literal and naming suites here
541+
// would need the same edit that introduces it. A suite cannot assign the shared
542+
// name either: its script sources the library first, so its own assignment
543+
// lands afterwards and replaces the value the upstream test resolved.
544+
//
545+
// Both sets are asserted non-empty. A glob that stops matching, or suites that
546+
// all stop calling the guard, leave the loop with nothing to check, and that
547+
// reads exactly like a pass.
548+
func TestChainsawSuitesReadHistoryThroughTheSharedName(t *testing.T) {
549+
var suites []string
550+
for _, pattern := range []string{"chainsaw-test.yaml", "chainsaw-test.yml"} {
551+
found, err := filepath.Glob(filepath.Join(chainsawSuites, "*", pattern))
552+
if err != nil {
553+
t.Fatalf("globbing the Chainsaw suites: %v", err)
497554
}
555+
suites = append(suites, found...)
556+
}
557+
if len(suites) == 0 {
558+
t.Fatalf("no Chainsaw suites found under %s, so this test asserted nothing", chainsawSuites)
498559
}
560+
561+
// Each script step runs in a shell of its own, so the read the guard sees is
562+
// the one in its own step. Checked against the whole file, a read in one
563+
// step would satisfy a call in another, and an unrelated assignment of the
564+
// same local elsewhere would fail a correct one.
565+
t.Run("a read in another step does not count", func(t *testing.T) {
566+
steps := scriptSteps(t, "fixture", []byte(`spec:
567+
steps:
568+
- try:
569+
- script:
570+
content: |
571+
statuses=$(kubectl get hr -o "jsonpath=${HELMRELEASE_HISTORY_JSONPATH}")
572+
--- # a separator may carry a comment
573+
spec:
574+
steps:
575+
- try:
576+
- script:
577+
content: |
578+
helmrelease_has_remediation_cycle "$statuses"
579+
`))
580+
if len(steps) != 2 {
581+
t.Fatalf("split the fixture into %d script steps, want 2", len(steps))
582+
}
583+
rec := &errorRecorder{TB: t}
584+
if n := checkSuiteSteps(rec, "fixture", steps); n != 1 {
585+
t.Fatalf("counted %d guard callers in the fixture, want 1", n)
586+
}
587+
if len(rec.errs) == 0 {
588+
t.Error("a guard call whose only read sits in another step passed the check")
589+
}
590+
})
591+
592+
// The read has to come before the call: one after it fills the local only
593+
// once the guard has already run, on an empty or unset value.
594+
t.Run("a read after the guard call does not count", func(t *testing.T) {
595+
const read = "statuses=$(kubectl get hr -o \"jsonpath=${HELMRELEASE_HISTORY_JSONPATH}\")\n"
596+
const call = "helmrelease_has_remediation_cycle \"$statuses\"\n"
597+
for _, tc := range []struct {
598+
name, step string
599+
want bool
600+
}{
601+
{"read first", read + call, true},
602+
{"call first", call + read, false},
603+
// The call's own text in a comment above the read is not the call.
604+
{"call text in a comment first", "# then " + call + read + call, true},
605+
} {
606+
ms := guardConsumers.FindAllSubmatchIndex([]byte(tc.step), -1)
607+
if len(ms) != 1 {
608+
t.Fatalf("%s: found %d guard calls, want 1", tc.name, len(ms))
609+
}
610+
m := ms[0]
611+
if got := sharedReadPrecedes([]byte(tc.step), m); got != tc.want {
612+
t.Errorf("%s: sharedReadPrecedes = %v, want %v", tc.name, got, tc.want)
613+
}
614+
// The same step through the suite check, so the order test is
615+
// one the check runs and not only a property of the helper.
616+
rec := &errorRecorder{TB: t}
617+
checkSuiteSteps(rec, "fixture", []string{tc.step})
618+
if refused := len(rec.errs) > 0; refused == tc.want {
619+
t.Errorf("%s: the suite check refused the step = %v, want %v: %v", tc.name, refused, !tc.want, rec.errs)
620+
}
621+
}
622+
})
623+
624+
callers := 0
625+
for _, path := range suites {
626+
src := readGuardFile(t, path)
627+
if n := len(nameFollowedByEquals.FindAll(src, -1)); n != 0 {
628+
t.Errorf("%s writes %s followed by = %d times; only %s may assign it. A suite sources "+
629+
"the library before its own lines run, so its assignment replaces the value the "+
630+
"upstream test checked.", path, sharedName, n, guardLibrary)
631+
}
632+
callers += checkSuiteSteps(t, path, scriptSteps(t, path, src))
633+
}
634+
if callers == 0 {
635+
t.Fatalf("no Chainsaw suite under %s calls helmrelease_has_remediation_cycle with an "+
636+
"expanded local, so this test asserted nothing. If no suite runs the guard any more, "+
637+
"this test goes with it.", chainsawSuites)
638+
}
639+
}
640+
641+
// The content of every script step in a Chainsaw file, wherever it sits: try,
642+
// catch, finally, or a nested operation.
643+
func scriptSteps(t testing.TB, path string, src []byte) []string {
644+
t.Helper()
645+
var steps []string
646+
var walk func(any)
647+
walk = func(node any) {
648+
switch v := node.(type) {
649+
case map[string]any:
650+
if script, ok := v["script"].(map[string]any); ok {
651+
if content, ok := script["content"].(string); ok {
652+
steps = append(steps, content)
653+
}
654+
}
655+
for _, child := range v {
656+
walk(child)
657+
}
658+
case []any:
659+
for _, child := range v {
660+
walk(child)
661+
}
662+
}
663+
}
664+
// The reader splits on a separator line that carries a comment too, which a
665+
// pattern for a bare `---` would leave inside one document, and Unmarshal
666+
// reads only the first document it is given.
667+
docs := utilyaml.NewYAMLReader(bufio.NewReader(bytes.NewReader(src)))
668+
for {
669+
doc, err := docs.Read()
670+
if errors.Is(err, io.EOF) {
671+
break
672+
}
673+
if err != nil {
674+
t.Fatalf("splitting %s into documents: %v", path, err)
675+
}
676+
var tree any
677+
if err := yaml.Unmarshal(doc, &tree); err != nil {
678+
t.Fatalf("parsing %s: %v", path, err)
679+
}
680+
walk(tree)
681+
}
682+
return steps
683+
}
684+
685+
// checkSuiteSteps holds each script step that calls the guard to its own read,
686+
// since every step runs in a shell of its own, and returns how many did.
687+
func checkSuiteSteps(t testing.TB, path string, steps []string) int {
688+
t.Helper()
689+
callers := 0
690+
for _, step := range steps {
691+
consumers := guardConsumers.FindAllSubmatchIndex([]byte(step), -1)
692+
if len(consumers) == 0 {
693+
continue
694+
}
695+
callers++
696+
assertConsumedLocalsTakeTheSharedRead(t, path, []byte(step), consumers)
697+
}
698+
return callers
699+
}
700+
701+
// errorRecorder collects Errorf calls instead of failing, so a subtest can
702+
// assert that a check reports a fixture it must refuse.
703+
type errorRecorder struct {
704+
testing.TB
705+
errs []string
706+
}
707+
708+
func (r *errorRecorder) Errorf(format string, args ...any) {
709+
r.errs = append(r.errs, fmt.Sprintf(format, args...))
499710
}
500711

501712
// No shell library but the one that owns the name assigns it. Sourcing the

0 commit comments

Comments
 (0)