Skip to content

Commit 6ee56e3

Browse files
authored
fix(ci): decide E2E test coverage per changed path (#3441)
## What this PR does `hack/select-e2e.sh` escalates to the full Chainsaw suite when a changed package reaches no suite through the dependency graph, because otherwise that change gets no E2E coverage at all. The escalation was read off the merged selection, so it depended on the whole diff rather than on the path that needed it. Any other changed path contributing a suite name made the selection non-empty and the escalation vanished with nothing in the output to say it had. Editing a platform component together with one unrelated suite's Chainsaw test ran that single suite instead of everything, which is an ordinary PR shape: change a component, adjust the e2e near it. Coverage is now decided per changed path. Each path resolves through its owning sources to the suites that exist, and a path reaching none of them forces the full run whatever the rest of the diff selected. The unit is the path rather than the source, because `system/postgres-operator` belongs to two PackageSources where one reaches no suite, and deciding per source would run everything on every change to it. The other half is the mapping that left packages unreachable from their own suites. `src_to_suites()` covered `*-application` names plus `external-dns`, and the final filter dropped the rest, so `cozystack.kuberture` and `cozystack.securitygroup-controller` were walked to and then discarded. A CNI change ran neither of the two controller suites most likely to regress from it, and a change to either package escalated to the full run rather than selecting the suite that covers it. Every reached source now maps through the table, which is the inverse of `select-install.sh`'s `suite_to_source()`; a test walks every suite through both, so the next source name fitting neither convention fails there instead of quietly escalating forever. ## What was measured Behaviour was diffed against the current selector over one `values.yaml` per package under `packages/{apps,system,extra}`: 159 packages, 8 selections change, all of them the mapping repair. Five networking packages and `prometheus-operator-crds` widen — 19 to 21 suites and 10 to 11, both gaining `kuberture` and `securitygroup`. `kuberture` and `securitygroup-controller` narrow from the full suite to their own single suite. Nothing narrows that should not. That sweep is one path per diff, so it says nothing about the per-path escalation, which only shows on a diff with more than one path. Pairing each package with one unrelated Chainsaw edit gives that number. 82 of the 159 packages escalate on their own and stop escalating once the pair is added, which is the size of the hole: for those, adding an unrelated suite edit to the same pull request replaced the full run with one suite. 88 packages select something different after this change, the extra six being the mapping repair widening a selection rather than restoring an escalation. Worth stating alongside it, because it decides whether the branch is a corner or the common case: 83 of the 159 escalate on a single-path diff today, and 81 do after this change — the two that leave are `kuberture` and `securitygroup-controller`, which now select the suite that covers them. So the escalation is the branch most packages take, and the swallowed one was the branch most packages took. Counting that needs an instrument that separates a graph walk which reached everything from an escalation which printed everything, since the two agree once the mapping is fixed: the count above comes from a scratch suite tree carrying one extra suite that no source names, which only an escalation can emit. The suite selection is what decides which tests run, so a rule that stops firing does not go red, it disappears. The tests pin the property rather than a worked example: that a path reaching no suite escalates whatever else the diff holds, that a covered path never drags the full suite in with it, and that every suite round-trips between the two mapping tables. Each was checked by mutation — removing the rule it guards turns it red. ## Left for a follow-up `resolve_suites` writes into the caller's scope because POSIX sh has no `local`, and what keeps that safe is the single call site sitting inside `$( )`, pinned by a test that counts occurrences of the name. Declaring the function with `( … )` instead of `{ … }` would contain those variables by construction and let both the test and the two paragraphs explaining the constraint go away. A count-the-name grep is the weakest kind of pin and it exists only because the function leaks. The output contract at the top of the script lists the causes of a full-suite selection but not the backstop, which after this change is a mechanism of its own: an edit under `hack/e2e-chainsaw/<dir>/` where that directory holds no `chainsaw-test.yaml`. The behaviour is pinned by a test and explained where it lives; only the header block is short. In the same area, the escalation line names the owning sources rather than the changed path, and reports the first uncovered group rather than all of them. ## Overlap #3434 fixes the same escalation from a different angle: it decides the escalation from the whole source-derived selection rather than per path. That closes the "package with no coverage plus an unrelated suite edit" shape, and leaves "package with no coverage plus another package that does select suites", where the source-derived selection is non-empty and the escalation is swallowed again. There is a test for that second shape here. ### Screenshots Not applicable, no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The changed files are `hack/select-e2e.sh`, `hack/select-e2e_test.bats` and `docs/agents/e2e-testing.md`. The only `hack/`-related trigger is cozystack/ccp, which gates on `hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh` and on what a make target does; nothing here is moved, renamed, or changed in make-target behaviour. The ansible-cozystack and talm triggers are about `hack/e2e-prepare-cluster.bats`, which is untouched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(ci): a changed package that no E2E suite covers now runs the full suite even when the same pull request also edits an unrelated suite, and changes to the kuberture and securitygroup packages select their own suite instead of escalating ``` Closes #3330 Closes #3665
2 parents 677d80a + 587a7b9 commit 6ee56e3

3 files changed

Lines changed: 298 additions & 59 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
7070

7171
- Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh|*.bats` helpers and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test.
7272
- **Every path is classified, and an unclassified one escalates.** The selector has three selection outcomes — full suite, a scoped set, or nothing — and each is reached by an explicit rule; the fourth outcome in the bullet below is not a selection but a refusal to produce one. Several rules produce the empty one, which is worth knowing before reading an empty selection as "the inert list matched": documentation, dashboards and `*.md` are dropped first wherever they live, a `*.disabled` Chainsaw suite and a `examples/backups/<app>/` edit whose app has no suite are dropped where chainsaw and backup-example paths are matched, and `inert_config_pattern` covers repo meta, agent config and `hack/testdata/`. `full_suite_pattern` covers shared build inputs and the e2e harness, and the graph lookup handles `packages/(apps|system|extra)/`. A path matching none of them escalates to the full suite and logs why, because both lanes read an empty selection as "skip Chainsaw" and then post `E2E Tests` green — so a silent default is a green gate with nothing tested (#3392). When that escalation fires for something genuinely inert, add it to `inert_config_pattern`; do not relax the fall-through. The escalation list and the inert list are checked in that order, so a specific escalation beats a broad inert directory.
73+
- **Coverage is decided per changed path.** A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources — `system/postgres-operator` belongs to two, one of which reaches no suite — and deciding per source would escalate on that half and run everything for every change to it.
7374
- **There is a fourth outcome, and it is a failed step rather than a selection.** It fires when the suite list itself is unavailable: `find` cannot list `hack/e2e-chainsaw/`, or that directory holds no `chainsaw-test.yaml`. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting `E2E Tests` green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under `bash -e`, so the job fails and the required check goes red. A red `Select E2E tests` step is a real signal, not a flake: read its stderr line before rerunning. A `yq` failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot.
7475
- **The one path the fall-through cannot catch is a new e2e workflow.** `inert_config_pattern` makes `.github/` inert as a directory, and the workflows that run the suite are exempted from it by name in `full_suite_pattern`. A workflow added under `.github/workflows/` therefore reads as inert until someone adds it to that enumeration, and the fall-through never sees it. That is why the list carries a `rg -l test-chainsaw .github/workflows/` reminder: the enumeration is the guard, not a convenience.
7576
- The `full-e2e` PR label forces the whole suite.
@@ -78,6 +79,8 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
7879

7980
When adding a new app package, confirm `select-e2e.sh` maps it correctly (it has a unit test, `hack/select-e2e_test.bats`). A new app whose suite directory does not yet exist will not be selected — add `hack/e2e-chainsaw/<app>/chainsaw-test.yaml` first.
8081

82+
`src_to_suites()` maps a PackageSource name to the suites it owns and is the inverse of `select-install.sh`'s `suite_to_source()`. A test walks every suite through both tables, so a source whose name fits neither convention — not `<suite>-application`, not the suite name itself — fails there rather than quietly making its own suite unreachable, which turns every change to that package into a full run (#3665).
83+
8184
#### What TIA does and does not do
8285

8386
Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs," and the wiring has consequences worth knowing before you rely on it or change it.

‎hack/select-e2e.sh‎

Lines changed: 103 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@
1313
# inert_config_pattern for repo meta and agent config
1414
# - <suite names> selected per the PackageSource dependency graph
1515
# - full list any path that affects all tests, OR an unrecognised
16-
# packages/* path, OR a per-app source whose graph has
17-
# no *-application descendants, OR a path matching NEITHER
16+
# packages/* path, OR a changed package that reaches no
17+
# runnable suite through the graph, OR a path matching NEITHER
1818
# the full-suite nor the inert list, OR a yq that failed to
1919
# build the dependency graph (conservative fallbacks)
2020
# - nothing, and the suite list itself is unavailable: find failed, or
@@ -148,13 +148,29 @@ escalate_to_full_suite() {
148148
}
149149

150150
# PackageSource name -> Chainsaw suite name(s). Most *-application sources map
151-
# by stripping the suffix; explicit overrides for the few that don't.
151+
# by stripping the suffix, a source that is not an app carries its own name
152+
# (cozystack.kuberture owns the kuberture suite), and the few that match neither
153+
# are listed explicitly.
154+
#
155+
# This is the inverse of select-install.sh's suite_to_source() and has to stay
156+
# in step with it — a suite that does not round-trip is unreachable from its own
157+
# package, so every change to that package escalates to the full run instead of
158+
# selecting the one suite that covers it (#3665). The round-trip is pinned by a
159+
# test rather than left to review.
160+
#
161+
# Mapping optimistically is safe: a name that is not a suite is dropped by
162+
# intersect_suites() downstream.
163+
#
164+
# Only names the two general arms get wrong are listed. `postgres-application`
165+
# and `external-dns` used to sit here and no longer do: the first is what
166+
# stripping the suffix already produces, and the second is what carrying the
167+
# source name through already produces, so both said the rule twice and made
168+
# this list read as longer than the set of real exceptions.
152169
src_to_suites() {
153170
case "$1" in
154-
postgres-application) echo postgres ;;
155171
vm-instance-application) echo vminstance ;;
156172
kubernetes-application) echo "kubernetes-latest kubernetes-previous kubernetes-oidc-system kubernetes-oidc-customconfig" ;;
157-
external-dns) echo external-dns ;;
173+
securitygroup-controller) echo securitygroup ;;
158174
*-application) echo "${1%-application}" ;;
159175
*) echo "$1" ;;
160176
esac
@@ -177,8 +193,8 @@ build_owners_index() {
177193
# Without this filter, postgres-operator -> keycloak -> cozystack-engine -> EVERY
178194
# app, defeating test-impact analysis. Dropping the engine reverse edges keeps
179195
# the engine reachable but stops it propagating selection downstream. A change to
180-
# the engine itself still triggers the full suite via the no-app-descendants
181-
# safety-net at the bottom of this script.
196+
# the engine itself still triggers the full suite: its group reaches no suite, so
197+
# the per-path escalation below fires on it.
182198
build_reverse_deps() {
183199
yq -rN '.metadata.name as $n | .spec.variants[]?.dependsOn[]? | select(. != null and . != "" and . != "cozystack.cozystack-engine") | . + "\t" + $n' "$SOURCES_DIR"/*.yaml
184200
}
@@ -207,7 +223,7 @@ REVERSE=$(printf '%s\n' "$REVERSE" | sort -u)
207223

208224
trigger_full=0
209225
trigger_any=0
210-
selected_sources=""
226+
selected_groups=""
211227
selected_apps=""
212228

213229
# `|| [ -n "$file" ]` is what makes the classification below total. POSIX read
@@ -281,7 +297,12 @@ while IFS= read -r file || [ -n "$file" ]; do
281297
if [ -n "$rel" ]; then
282298
src=$(echo "$OWNERS" | awk -v p="$rel" -F'\t' '$1==p {print $2}')
283299
if [ -n "$src" ]; then
284-
selected_sources="$selected_sources $src"
300+
# Keep this path's owning sources together as one comma-joined group. The
301+
# unit coverage is decided over is the changed PATH, not the single
302+
# source: system/postgres-operator belongs to two PackageSources, one of
303+
# which reaches no suite, and deciding per source would escalate on that
304+
# half and run everything for every change to it.
305+
selected_groups="$selected_groups $(echo "$src" | paste -sd , -)"
285306
trigger_any=1
286307
else
287308
# Inside packages/ but no graph entry — be conservative.
@@ -306,43 +327,85 @@ if [ "$trigger_any" = 0 ]; then
306327
exit 0 # nothing to run
307328
fi
308329

309-
# Transitive closure: walk reverse-deps from each selected source.
310-
all_sources="$selected_sources"
311-
while :; do
312-
new=""
313-
for s in $all_sources; do
314-
deps=$(echo "$REVERSE" | awk -v src="$s" -F'\t' '$1==src {print $2}')
315-
for d in $deps; do
316-
case " $all_sources " in *" $d "*) ;; *) new="$new $d";; esac
330+
# Deduplicate a space-separated suite list; keep only names that are suites.
331+
intersect_suites() {
332+
echo "$1" | tr ' ' '\n' | sort -u | grep -v '^$' | while read -r app; do
333+
if echo "$all_apps" | grep -Fxq "$app"; then
334+
echo "$app"
335+
fi
336+
done | paste -sd ' ' -
337+
}
338+
339+
# Transitive closure over reverse-deps from the given PackageSource names, every
340+
# reached source mapped through src_to_suites. Emits candidate suite names, not
341+
# yet filtered — pass the result through intersect_suites.
342+
#
343+
# POSIX sh has no `local`, so this scribbles on all_sources/new/final. Its one
344+
# call site invokes it inside $( ), which contains the damage; a direct call
345+
# would clobber the caller's variables. A test pins that there is still exactly
346+
# one call.
347+
resolve_suites() {
348+
all_sources="$*"
349+
while :; do
350+
new=""
351+
for s in $all_sources; do
352+
deps=$(echo "$REVERSE" | awk -v src="$s" -F'\t' '$1==src {print $2}')
353+
for d in $deps; do
354+
case " $all_sources " in *" $d "*) ;; *) new="$new $d";; esac
355+
done
317356
done
357+
[ -z "$new" ] && break
358+
all_sources="$all_sources $new"
318359
done
319-
[ -z "$new" ] && break
320-
all_sources="$all_sources $new"
321-
done
322360

323-
# Filter to *-application sources, then map to Chainsaw suite names.
324-
final=""
325-
for s in $all_sources; do
326-
app=${s#cozystack.}
327-
case "$app" in
328-
*-application) final="$final $(src_to_suites "$app")" ;;
329-
external-dns) final="$final external-dns" ;;
330-
esac
331-
done
332-
333-
# Add directly-selected suites from per-suite edits.
334-
final="$final $selected_apps"
361+
final=""
362+
for s in $all_sources; do
363+
final="$final $(src_to_suites "${s#cozystack.}")"
364+
done
365+
echo "$final"
366+
}
335367

336-
# Deduplicate; intersect with available Chainsaw suites.
337-
final_apps=$(echo "$final" | tr ' ' '\n' | sort -u | grep -v '^$' | while read -r app; do
338-
if echo "$all_apps" | grep -Fxq "$app"; then
339-
echo "$app"
368+
# Escalation is a property of the changed path, not of the whole diff. A path
369+
# that reaches no runnable suite is covered by nothing and forces the full run
370+
# on its own account, whatever the rest of the diff selected.
371+
#
372+
# Reading that off an empty FINAL selection instead — which is what the old
373+
# safety net at the bottom did, and still does as a backstop — let any other
374+
# changed path that contributed a suite name swallow the escalation, with
375+
# nothing in the output to record that it happened (#3330). The shape that hits
376+
# it is ordinary rather than exotic: change a platform component, adjust the
377+
# Chainsaw test next to it, and the component's escalation disappears behind
378+
# that one suite.
379+
group_suites=""
380+
for g in $(echo "$selected_groups" | tr ' ' '\n' | sort -u); do
381+
s=$(intersect_suites "$(resolve_suites "$(echo "$g" | tr ',' ' ')")")
382+
if [ -z "$s" ]; then
383+
# Say which changed path bought the full run. Every other escalation here
384+
# names its cause on stderr, and this one is the common case rather than the
385+
# rare one — most packages reach no suite of their own — so without a line
386+
# "why did this pull request run everything" has no answer in the log. The
387+
# sources are printed rather than the path because they are what the walk
388+
# started from; the path that owns them is the one this group came from in
389+
# the classification loop above.
390+
echo "select-e2e: no runnable suite covers $(echo "$g" | tr ',' ' ') — escalating to the full suite" >&2
391+
escalate_to_full_suite
340392
fi
341-
done | paste -sd ' ' -)
393+
group_suites="$group_suites $s"
394+
done
395+
396+
# Union of the graph-selected suites and the directly-selected ones from
397+
# per-suite edits. Walking the graph once per group rather than once over all of
398+
# them costs an extra walk per changed path and buys the escalation above; the
399+
# answer is unchanged, because reverse reachability distributes over union, so
400+
# the union of the per-group results is the set the single walk produced.
401+
final_apps=$(intersect_suites "$group_suites $selected_apps")
342402

343-
# Safety net: a system source with no *-application descendants would otherwise
344-
# silently skip E2E. Fall back to full suite so a path inside the graph is
345-
# never silently dropped.
403+
# Backstop. Every graph path above either escalates or contributes a suite that
404+
# exists, so what still reaches this is a per-suite edit naming a directory that
405+
# holds no chainsaw-test.yaml — shared material beside _lib/, or a suite nested
406+
# deeper than the depth-2 scan looks. Selecting nothing for those would skip
407+
# E2E outright, so failing towards the full suite is the only safe way to be
408+
# wrong here.
346409
if [ -z "$final_apps" ]; then
347410
escalate_to_full_suite
348411
fi

0 commit comments

Comments
 (0)