Skip to content

Commit e5a1ca8

Browse files
committed
fix(ci): run every Keycloak client suite on a Keycloak change
With a monitoring suite in the tree, a change to system/keycloak or system/keycloak-operator stopped escalating to the full run and selected monitoring alone, because monitoring-application is the only source that declares the keycloak-operator edge. That dropped kubernetes-latest, whose OIDC lifecycle also creates a KeycloakClient and waits on the operator's finalizer to remove it. The mapping is explicit because kubernetes-application has no dependsOn on keycloak-operator. A postgres-operator change now reaches kubernetes-latest too, through Keycloak's database. The selector comment and the e2e doc no longer name system/postgres-operator as a path with a source that reaches no suite, which stopped being true when monitoring got its suite. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 472f962 commit e5a1ca8

3 files changed

Lines changed: 29 additions & 7 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
9696

9797
- Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh` helpers, the `hack/e2e-*.bats` the sandbox runs and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/` and `examples/backups/_lib/` (the place for helpers shared by the backup walkthroughs), the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. The directory name is membership-tested against the suite list on the spot, so a directory holding no suite — shared material beside `_lib/`, or a suite nested deeper than discovery looks — escalates there and then, naming the file. `hack/e2e-apps/<name>.bats` — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. The files formerly kept there were wired to nothing after migration, and the path remains deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested the same way. Deferring either test to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under `hack/e2e-apps/` — a nested path, a non-`.bats` file — is shared material for those suites the way `hack/e2e-chainsaw/_lib/` is for the Chainsaw ones, and escalates for the same reason.
9898
- **Every path is classified, and an unclassified one escalates.** The selector has three selection outcomes — full suite, a scoped set, or nothing — and each is reached by an explicit rule; the fourth outcome in the bullet below is not a selection but a refusal to produce one. Several rules produce the empty one, which is worth knowing before reading an empty selection as "the inert list matched": documentation, dashboards and `*.md` are dropped first wherever they live, a `*.disabled` Chainsaw suite and a `examples/backups/<app>/` edit whose app has no suite are dropped where chainsaw and backup-example paths are matched, and `inert_config_pattern` covers repo meta, agent config, `hack/testdata/`, the helm-unittest fixture charts under `packages/tests/`, the three `.gitattributes` that hold nothing but `linguist-generated` markers, which reach no build, chart or test (enumerated rather than matched by filename, because the justification is what those files contain and the name does not carry it — `.gitattributes` can also set `filter`, `eol`, `working-tree-encoding` and `export-subst`, each of which changes what lands in the working tree and therefore what gets built, so a by-name rule would make such a file inert in silence) and the top-level `hack/*.bats` that are not `hack/e2e-*.bats`. That last one is the unit lane, and the root `Makefile` is the authority on the split: `BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))` keeps every non-E2E file for `make unit-tests`, while `packages/core/testing` runs the E2E-prefixed files, so the sandbox never executes a unit file and no Chainsaw suite can regress from an edit to one. Being inert here does not leave them untested: the `unit-tests` job is gated on the `plan` job's `code` output, which `pull-requests.yaml` computes as "any changed path outside `docs/`" and never from `select-e2e.sh`, so a bats-only pull request still runs the whole unit lane — plus install and the OpenAPI tests, since the `e2e` job also reads `code` rather than the selection. `full_suite_pattern` covers shared build inputs and the e2e harness, and the graph lookup handles `packages/(apps|system|extra)/`. A path matching none of them escalates to the full suite and logs why, because both lanes read an empty selection as "skip Chainsaw" and then post `E2E Tests` green — so a silent default is a green gate with nothing tested (#3392). When that escalation fires for something genuinely inert, add it to `inert_config_pattern`; do not relax the fall-through. The escalation list and the inert list are checked in that order, so a specific escalation beats a broad inert directory.
99-
- **Coverage is decided per changed path.** A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources — `system/postgres-operator` belongs to two, one of which reaches no suite — and deciding per source would escalate on that half and run everything for every change to it.
99+
- **Coverage is decided per changed path.** A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources, not all of which need reach a suite, and deciding per source would escalate on one that reaches none and run everything for every change to the path.
100100
- **Two sources are excluded from the reverse-dependency walk as install-ordering hubs.** Every `*-application` source depends on `cozystack.cozystack-engine` so its `*-rd` HelmRelease waits for the ApplicationDefinition CRD, and the sources that depend on `cozystack.cozystack-basics` do so to get a namespace or a platform-wide policy in place first. Neither edge says the dependent's behaviour changes when the hub does, so carrying selection across them goes wrong in both directions: through the engine a `cert-manager` change fans out to every app in the tree, and through basics the platform's namespace-and-policy package narrows to whichever single suite happens to land downstream of it — 22 suites down to one on #3426, where the new site-router suite sits under `kubevirt-cdi`, which depends on basics. Dropping the reverse edges leaves both hubs reachable and stops them propagating; a change to either still runs everything, because its own group then reaches no suite and the per-path escalation fires on it. Adding a third one is a real decision, not bookkeeping: it says every existing edge into that source is ordering rather than behaviour, so read them before extending the list.
101101
- **There is a fourth outcome, and it is a failed step rather than a selection.** It fires when the suite list itself is unavailable: `find` cannot list `hack/e2e-chainsaw/`, or that directory holds no `chainsaw-test.yaml` or `chainsaw-test.yml`. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting `E2E Tests` green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under `bash -e`, so the job fails and the required check goes red. A red `Select E2E tests` step is a real signal, not a flake: read its stderr line before rerunning. A `yq` failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot.
102102
- **Every escalation names its cause on stderr.** "Run everything" is the same list of suite names whichever rule produced it, so a reason line is the only thing that distinguishes the branches that can reach it, and without one the answer to "why did this pull request run the whole suite" has to be re-derived by hand from the diff. Four of them used to be silent — the shared `hack/e2e-chainsaw/_lib/` and `.chainsaw.yaml` case, the `full_suite_pattern` match, a `packages/` path no PackageSource claims, and the empty-selection backstop — and `full_suite_pattern` is the commonest cause of a full run by a wide margin, so the usual answer was the one the log never carried. Each line names the path or the sources that caused it, and a unit test asserts each message, because a reason that regresses to silence changes no selection and is invisible to every other test. The backstop is the exception: every rule that selects a suite by name membership-tests it where the path is classified, so nothing reaches the backstop and no test asserts its line; it stays as the guard for a future rule that forgets to. The lines go to **stderr** and must stay there: stdout is the suite list and both lanes parse it, so a reason line on stdout would be read as a suite name.

‎hack/select-e2e.sh‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,11 @@ src_to_suites() {
257257
# serves. etcd's round-trip is left out because CI gates it off. A test
258258
# derives this set from the suites.
259259
seaweedfs-application) echo "bucket clickhouse harbor kafka kafka-metadata mariadb mongodb postgres rabbitmq redis seaweedfs" ;;
260+
# Both suites create a KeycloakClient and wait for the operator's finalizer
261+
# to remove it. kubernetes-application has no dependsOn on keycloak-operator
262+
# (#4680), so the graph reaches only monitoring and would drop the tenant
263+
# Kubernetes OIDC lifecycle in kubernetes-latest.
264+
keycloak|keycloak-operator) echo "kubernetes-latest monitoring" ;;
260265
*-application) echo "${1%-application}" ;;
261266
*) echo "$1" ;;
262267
esac
@@ -463,9 +468,9 @@ while IFS= read -r file || [ -n "$file" ]; do
463468
if [ -n "$src" ]; then
464469
# Keep this path's owning sources together as one comma-joined group. The
465470
# unit coverage is decided over is the changed PATH, not the single
466-
# source: system/postgres-operator belongs to two PackageSources, one of
467-
# which reaches no suite, and deciding per source would escalate on that
468-
# half and run everything for every change to it.
471+
# source: a path can belong to several PackageSources, some of which
472+
# reach no suite, and deciding per source would escalate on that half
473+
# and run everything for every change to it.
469474
selected_groups="$selected_groups $(echo "$src" | paste -sd , -)"
470475
trigger_any=1
471476
else

‎hack/select-e2e_test.bats‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -876,8 +876,9 @@ assert_full_suite() {
876876
@test "a path owned by several sources counts as covered if any one is" {
877877
# system/postgres-operator belongs to two PackageSources: cozystack.monitoring
878878
# and cozystack.postgres-operator, which reaches postgres-application,
879-
# harbor-application (Harbor uses postgres as its backing DB) and
880-
# monitoring-application. Coverage is decided over the path's sources
879+
# harbor-application (Harbor uses postgres as its backing DB),
880+
# monitoring-application, and keycloak, which maps to the suites driving
881+
# Keycloak clients. Coverage is decided over the path's sources
881882
# together, which is why the unit is the path and not the source — deciding
882883
# per source would escalate on a half that reaches nothing and run everything
883884
# for every change here. cozystack.monitoring maps to the monitoring
@@ -897,7 +898,7 @@ assert_full_suite() {
897898
[ "$owners" -ge 2 ]
898899
echo "packages/system/postgres-operator/values.yaml" > "$tmp/diff"
899900
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
900-
[ "$output" = "harbor monitoring postgres" ]
901+
[ "$output" = "harbor kubernetes-latest monitoring postgres" ]
901902
rm -rf "$tmp"
902903
}
903904

@@ -935,6 +936,22 @@ assert_full_suite() {
935936
rm -rf "$tmp"
936937
}
937938

939+
@test "a keycloak change selects both suites that drive Keycloak clients" {
940+
# monitoring and kubernetes-latest both create a KeycloakClient and assert
941+
# it is removed again, which goes through the operator's finalizer. Only
942+
# monitoring-application declares the keycloak-operator edge, so the graph
943+
# alone reaches monitoring and drops the tenant Kubernetes OIDC lifecycle.
944+
tmp=$(mktemp -d)
945+
cp -r packages/core/platform/sources "$tmp/sources"
946+
for path in packages/system/keycloak/values.yaml packages/system/keycloak-operator/values.yaml; do
947+
echo "$path" > "$tmp/diff"
948+
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
949+
assert_selection "a change to $path must run every Keycloak client lifecycle" \
950+
"$output" "kubernetes-latest monitoring"
951+
done
952+
rm -rf "$tmp"
953+
}
954+
938955
@test "an etcd-operator change selects the etcd suite, not the whole run" {
939956
# cozystack.etcd-operator reached no runnable suite, so every change to the
940957
# operator ran all 21 -- 5 of the last 150 merged pull requests. The suite

0 commit comments

Comments
 (0)