Skip to content

Commit dc9c2f7

Browse files
test(monitoring): remove orphaned OIDC BATS suites
The two pre-Chainsaw monitoring OIDC suites have had no invocation site since the app suite migrated to Chainsaw. Remove them instead of carrying dead tests that force the selector to run every live suite. Keep the existing selector and capture guards for the historical path shape, and update their comments now that the directory is empty. Retain a named chart assertion for the no-KeycloakRealmGroup ownership invariant; the existing count and kind assertions already enforce the same behavior. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent e89617d commit dc9c2f7

9 files changed

Lines changed: 44 additions & 455 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
7979

8080
`hack/select-e2e.sh` walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under `hack/e2e-chainsaw/` containing a `chainsaw-test.yaml`.
8181

82-
- Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh` helpers, the `hack/e2e-*.bats` the sandbox runs and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. `hack/e2e-apps/<name>.bats` — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. What remains in that directory is wired to nothing after the migration, and it is deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested against the suite list on the spot, and a name no suite carries escalates there and then, naming the file. Deferring that to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under `hack/e2e-apps/` — a nested path, a non-`.bats` file — is shared material for those suites the way `hack/e2e-chainsaw/_lib/` is for the Chainsaw ones, and escalates for the same reason.
82+
- Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh` helpers, the `hack/e2e-*.bats` the sandbox runs and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. `hack/e2e-apps/<name>.bats` — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. The files formerly kept there were wired to nothing after migration, and the path remains deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested against the suite list on the spot, and a name no suite carries escalates there and then, naming the file. Deferring that to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under `hack/e2e-apps/` — a nested path, a non-`.bats` file — is shared material for those suites the way `hack/e2e-chainsaw/_lib/` is for the Chainsaw ones, and escalates for the same reason.
8383
- **Every path is classified, and an unclassified one escalates.** The selector has three selection outcomes — full suite, a scoped set, or nothing — and each is reached by an explicit rule; the fourth outcome in the bullet below is not a selection but a refusal to produce one. Several rules produce the empty one, which is worth knowing before reading an empty selection as "the inert list matched": documentation, dashboards and `*.md` are dropped first wherever they live, a `*.disabled` Chainsaw suite and a `examples/backups/<app>/` edit whose app has no suite are dropped where chainsaw and backup-example paths are matched, and `inert_config_pattern` covers repo meta, agent config, `hack/testdata/`, the helm-unittest fixture charts under `packages/tests/`, the three `.gitattributes` that hold nothing but `linguist-generated` markers, which reach no build, chart or test (enumerated rather than matched by filename, because the justification is what those files contain and the name does not carry it — `.gitattributes` can also set `filter`, `eol`, `working-tree-encoding` and `export-subst`, each of which changes what lands in the working tree and therefore what gets built, so a by-name rule would make such a file inert in silence) and the top-level `hack/*.bats` that are not `hack/e2e-*.bats`. That last one is the unit lane, and the root `Makefile` is the authority on the split: `BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))` keeps every non-E2E file for `make unit-tests`, while `packages/core/testing` runs the E2E-prefixed files, so the sandbox never executes a unit file and no Chainsaw suite can regress from an edit to one. Being inert here does not leave them untested: the `unit-tests` job is gated on the `plan` job's `code` output, which `pull-requests.yaml` computes as "any changed path outside `docs/`" and never from `select-e2e.sh`, so a bats-only pull request still runs the whole unit lane — plus install and the OpenAPI tests, since the `e2e` job also reads `code` rather than the selection. `full_suite_pattern` covers shared build inputs and the e2e harness, and the graph lookup handles `packages/(apps|system|extra)/`. A path matching none of them escalates to the full suite and logs why, because both lanes read an empty selection as "skip Chainsaw" and then post `E2E Tests` green — so a silent default is a green gate with nothing tested (#3392). When that escalation fires for something genuinely inert, add it to `inert_config_pattern`; do not relax the fall-through. The escalation list and the inert list are checked in that order, so a specific escalation beats a broad inert directory.
8484
- **Coverage is decided per changed path.** A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources — `system/postgres-operator` belongs to two, one of which reaches no suite — and deciding per source would escalate on that half and run everything for every change to it.
8585
- **Two sources are excluded from the reverse-dependency walk as install-ordering hubs.** Every `*-application` source depends on `cozystack.cozystack-engine` so its `*-rd` HelmRelease waits for the ApplicationDefinition CRD, and the sources that depend on `cozystack.cozystack-basics` do so to get a namespace or a platform-wide policy in place first. Neither edge says the dependent's behaviour changes when the hub does, so carrying selection across them goes wrong in both directions: through the engine a `cert-manager` change fans out to every app in the tree, and through basics the platform's namespace-and-policy package narrows to whichever single suite happens to land downstream of it — 22 suites down to one on #3426, where the new site-router suite sits under `kubevirt-cdi`, which depends on basics. Dropping the reverse edges leaves both hubs reachable and stops them propagating; a change to either still runs everything, because its own group then reaches no suite and the per-path escalation fires on it. Adding a third one is a real decision, not bookkeeping: it says every existing edge into that source is ordering rather than behaviour, so read them before extending the list.

‎hack/bats-no-exit-trap.bats‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -571,9 +571,9 @@ bnet_add_two_handlers_on_one_line() {
571571
}
572572

573573
@test "a bats file in a subdirectory is audited too" {
574-
# The ban is on every bats file under hack/, and hack/e2e-apps/ already holds
575-
# two. A flat glob would leave the claim about "every file added later" false
576-
# for anything a directory down.
574+
# The ban covers every bats file under hack/. Two pre-Chainsaw suites once
575+
# lived one directory down; this synthetic fixture pins recursion after their
576+
# removal.
577577
tmp=$(mktemp -d)
578578
bnet_new_fixture "$tmp"
579579
bnet_add_nested_fixture "$tmp"

‎hack/cozytest-capture-gate.bats‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -175,9 +175,9 @@ run_fixture() {
175175
}
176176

177177
@test "a live cluster suite under an e2e directory keeps its captures" {
178-
# hack/e2e-apps/ holds suites whose own filenames carry no prefix. Matching
179-
# the basename alone disarms them, which takes the captures away from a suite
180-
# that runs against a real cluster and therefore has state worth capturing.
178+
# The pre-Chainsaw suites under hack/e2e-apps/ used plain filenames. This
179+
# fixture pins the directory-based guard so that layout retains captures if it
180+
# returns, even though no live suite currently uses it.
181181
dir=$(make_fixture_dir e2e-apps/plain-named-suite.bats)
182182
run_fixture "$dir" e2e-apps/plain-named-suite.bats
183183
grep -qF "$DATAPLANE_MARKER" "$dir/out"

‎hack/cozytest.sh‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -161,12 +161,12 @@ COZY_REPORT_DIR="${COZY_REPORT_DIR:-_out/cozyreport}"
161161
# packages/core/testing/Makefile runs against a live cluster instead.
162162
#
163163
# That wildcard is not recursive, so it says nothing about subdirectories, and
164-
# hack/e2e-apps/ holds live-cluster suites whose own filenames carry no prefix.
165-
# Matching the directory too keeps them armed; matching only the basename would
166-
# take the captures away from suites that need them, which is the opposite of
167-
# the fix. Neither test is a claim that every future live-cluster suite will be
168-
# named this way -- it is the only signal the runner has, and a suite placed
169-
# outside both shapes gets no captures.
164+
# hack/e2e-apps/ historically held live-cluster suites whose own filenames
165+
# carried no prefix. Keep matching the directory so captures stay armed if that
166+
# layout returns; matching only the basename would silently disarm them. Neither
167+
# test claims every future live-cluster suite will be named this way -- these are
168+
# the only signals the runner has, and a suite outside both shapes gets no
169+
# captures.
170170
#
171171
# Deliberately not a reachability probe. Gating on "can I talk to an apiserver"
172172
# would disarm the captures in the case they exist for: a failing e2e run is

‎hack/e2e-apps/monitoring-oidc-customconfig.bats‎

Lines changed: 0 additions & 202 deletions
This file was deleted.

0 commit comments

Comments
 (0)