Skip to content

Commit e53a76a

Browse files
ci: run nothing but the backport-batch tests for a backport-batch change
A pull request that changed only cmd/backport-batch counted as code: it built every image, installed the platform in the e2e sandbox and ran the whole unit lane. The backport-batch tests themselves ran on every code pull request, whatever it touched. The plan job's `code` output is now false when every changed path is under docs/ or one of the CI-only tools, cmd/backport-batch/ and cmd/api-gate/, so builds, finalize and e2e are skipped and e2e-report posts "E2E Tests" green as it already did for docs. e2e-fork.yaml skips on the same prefixes, and gate-concurrency-contract.bats now holds the two lists, and the tools the selectors leave alone, equal. test-backport-batch leaves `make unit-tests`. The checks job runs it when hack/backport-batch-tests-needed.sh finds a changed path listed in cmd/backport-batch/test-inputs.txt: the tool itself, the in-repo packages it imports, the docs pages its tests read, go.mod and go.sum. TestInputsListed compares that list with `go list -deps -test` and fails when an import is missing, so it cannot fall behind the code without the pull request that moved it going red. A new `docs` output keeps the Bats lane for docs changes and drops it for tool-only ones. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent d1f3f49 commit e53a76a

11 files changed

Lines changed: 375 additions & 28 deletions

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,8 @@ jobs:
6161
# base-repo PR file list — NOT from the presence/absence of an artifact the
6262
# fork's own build controls. Fail-closed if the fork build failed or the
6363
# pr-patch artifact is missing; green-and-stop only for a PR whose file list is
64-
# entirely under docs/ (verified against the base-repo API).
64+
# entirely under docs/ or the CI-only tools cmd/backport-batch/ and
65+
# cmd/api-gate/ (verified against the base-repo API).
6566
resolve:
6667
name: Resolve fork PR
6768
# Only fork PRs: head_repository differs from the base repository. Same-repo
@@ -266,15 +267,21 @@ jobs:
266267
await decide('failure', 'PR file list hit the 3000-file API cap — cannot verify docs-only');
267268
return;
268269
}
270+
// Paths with nothing to build and nothing for e2e to install: docs,
271+
// and the CI-only Go tools no image builds. pull-requests.yaml's
272+
// plan job answers the same question with the same prefixes, and
273+
// hack/gate-concurrency-contract.bats holds the two lists equal.
274+
//
269275
// A rename reports the NEW path in `filename` and the old one in
270276
// `previous_filename`. Checking only `filename` would read a
271277
// packages/… -> docs/… rename as docs-only, when it in fact removes
272-
// a code path. BOTH sides must be under docs/ to count as docs-only.
273-
const isDocs = p => typeof p === 'string' && p.startsWith('docs/');
274-
const hasNonDocs = files.some(f =>
275-
!isDocs(f.filename) || (f.previous_filename && !isDocs(f.previous_filename)));
276-
if (!hasNonDocs) {
277-
await decide('success', 'docs-only PR (verified via PR file list) — e2e not required');
278+
// a code path. BOTH sides must be under these prefixes to count.
279+
const noRuntimePrefixes = ['docs/', 'cmd/backport-batch/', 'cmd/api-gate/'];
280+
const isNoRuntime = p => typeof p === 'string' && noRuntimePrefixes.some(x => p.startsWith(x));
281+
const hasRuntime = files.some(f =>
282+
!isNoRuntime(f.filename) || (f.previous_filename && !isNoRuntime(f.previous_filename)));
283+
if (!hasRuntime) {
284+
await decide('success', 'docs or CI-only tools (verified via PR file list) — e2e not required');
278285
return;
279286
}
280287

‎.github/workflows/pull-requests.yaml‎

Lines changed: 38 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ jobs:
7575
statuses: write
7676
outputs:
7777
code: ${{ steps.p.outputs.code }}
78+
docs: ${{ steps.p.outputs.docs }}
79+
backport_batch: ${{ steps.p.outputs.backport_batch }}
7880
matrix: ${{ steps.p.outputs.matrix }}
7981
any: ${{ steps.p.outputs.any }}
8082
touched: ${{ steps.p.outputs.touched }}
@@ -167,11 +169,27 @@ jobs:
167169
# alone here rather than widened by a side effect.
168170
git diff --name-only --no-renames "origin/${BASE_REF}...HEAD" > /tmp/changed_norenames.txt
169171
echo "Changed files:"; sed 's/^/ /' /tmp/changed_norenames.txt
170-
if grep -qvE '^docs/' /tmp/changed_norenames.txt; then
172+
# `code` is false when every path is under docs/ or in a CI-only Go
173+
# tool that no image builds (cmd/backport-batch, cmd/api-gate): there
174+
# is nothing to build and nothing for e2e to install. e2e-fork.yaml
175+
# answers the same question from the PR file list with the same
176+
# prefixes, and hack/gate-concurrency-contract.bats holds the two lists
177+
# equal.
178+
if grep -qvE '^(docs/|cmd/backport-batch/|cmd/api-gate/)' /tmp/changed_norenames.txt; then
171179
echo "code=true" >> "$GITHUB_OUTPUT"
172180
else
173181
echo "code=false" >> "$GITHUB_OUTPUT"
174182
fi
183+
# The Bats contracts read docs, so a docs change runs them even when
184+
# `code` is false; a change confined to the CI-only tools does not.
185+
if grep -qE '^docs/' /tmp/changed_norenames.txt; then
186+
echo "docs=true" >> "$GITHUB_OUTPUT"
187+
else
188+
echo "docs=false" >> "$GITHUB_OUTPUT"
189+
fi
190+
backport_batch="$(hack/backport-batch-tests-needed.sh /tmp/changed_norenames.txt)"
191+
echo "backport-batch tests needed: $backport_batch"
192+
echo "backport_batch=$backport_batch" >> "$GITHUB_OUTPUT"
175193
matrix="$(hack/build-matrix.sh /tmp/changed.txt)"
176194
echo "Build matrix: $matrix"
177195
echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
@@ -293,9 +311,11 @@ jobs:
293311
contents: read
294312
needs: ["plan"]
295313
# Unit contracts read both code and documentation. Run this job for every
296-
# non-release PR: code changes take the complete unit path, while docs-only
297-
# changes run the Bats lane that owns those cross-tree contracts. Builds and
298-
# E2E remain gated on `code` below.
314+
# non-release PR: code changes take the complete unit path, docs changes run
315+
# the Bats lane that owns those cross-tree contracts, and the backport-batch
316+
# tests run whenever one of their inputs changed and at no other time. A
317+
# change confined to the CI-only tools runs no other lane. Builds and E2E
318+
# remain gated on `code` below.
299319
if: |
300320
!contains(github.event.pull_request.labels.*.name, 'release')
301321
steps:
@@ -306,29 +326,31 @@ jobs:
306326
# The module cache is restored from the key build-main.yaml writes on
307327
# main, which every PR run can read, so a PR's first run starts warm. The
308328
# download fetches whatever the cache lacks, the whole set on a miss. The
309-
# Go steps are for code changes only: a docs-only run is the Bats lane alone.
329+
# Go steps are for code changes and the backport-batch tests only: a
330+
# docs-only run is the Bats lane alone.
310331
- name: Locate Go module cache
311-
if: needs.plan.outputs.code == 'true'
332+
if: needs.plan.outputs.code == 'true' || needs.plan.outputs.backport_batch == 'true'
312333
id: gomodcache
313334
run: echo "dir=$(go env GOMODCACHE)" >> "$GITHUB_OUTPUT"
314335
- name: Cache Go modules
315-
if: needs.plan.outputs.code == 'true'
336+
if: needs.plan.outputs.code == 'true' || needs.plan.outputs.backport_batch == 'true'
316337
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
317338
with:
318339
path: ${{ steps.gomodcache.outputs.dir }}
319340
key: go-mod-${{ runner.os }}-${{ hashFiles('**/go.sum') }}
320341
restore-keys: go-mod-${{ runner.os }}-
321342
- name: Download Go modules
322-
if: needs.plan.outputs.code == 'true'
343+
if: needs.plan.outputs.code == 'true' || needs.plan.outputs.backport_batch == 'true'
323344
run: |
324345
for mod in $(git ls-files go.mod '*/go.mod' | xargs -n1 dirname); do
325346
(cd "$mod" && "$GITHUB_WORKSPACE/hack/go-mod-download.sh")
326347
done
327348
# Ephemeral runners don't carry the helm-unittest plugin the persistent
328349
# self-hosted runner accreted. The plugin embeds its own Helm, so its
329350
# version is pinned: a plugin release must not change test results. Bats is
330-
# installed on every run, docs-only ones included.
351+
# installed whenever a Bats lane runs, docs-only ones included.
331352
- name: Set up test toolchain
353+
if: needs.plan.outputs.code == 'true' || needs.plan.outputs.docs == 'true'
332354
env:
333355
CODE_CHANGED: ${{ needs.plan.outputs.code }}
334356
HELM_UNITTEST_VERSION: "1.1.2"
@@ -374,8 +396,13 @@ jobs:
374396
if: needs.plan.outputs.code == 'true'
375397
run: make unit-tests test-controllers test-controllers-envtest -j4 -k --output-sync=target
376398
- name: Run Bats unit tests for docs-only changes
377-
if: needs.plan.outputs.code == 'false'
399+
if: needs.plan.outputs.code == 'false' && needs.plan.outputs.docs == 'true'
378400
run: make bats-unit-tests
401+
# Not part of `make unit-tests`: hack/backport-batch-tests-needed.sh
402+
# decides from cmd/backport-batch/test-inputs.txt, whatever `code` is.
403+
- name: Run backport-batch tests
404+
if: needs.plan.outputs.backport_batch == 'true'
405+
run: make test-backport-batch
379406
- name: Upload unit test report
380407
if: always()
381408
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
@@ -1427,7 +1454,7 @@ jobs:
14271454
description = `release packages candidate did not verify (result: ${CANDIDATE_RESULT})`;
14281455
} else if (CODE !== 'true') {
14291456
state = 'success';
1430-
description = 'docs-only PR — e2e not required';
1457+
description = 'docs or CI-only tools — e2e not required';
14311458
} else if (IS_RELEASE === 'true' && IS_FULL_E2E !== 'true') {
14321459
// Release promotion PRs deliberately do not repeat e2e: the rc was
14331460
// already verified against the immutable candidate whose digests

‎Makefile‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ test:
120120
make -C packages/core/testing apply
121121
make -C packages/core/testing e2e
122122

123-
unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness test-backport-batch migrations-target-check
123+
unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness migrations-target-check
124124

125125
helm-unit-tests:
126126
hack/helm-unit-tests.sh
@@ -196,6 +196,9 @@ test-check-readiness:
196196
go test ./test/check-readiness/ -count=1
197197

198198
# ./cmd/... is excluded from go-unit-tests, so the batch tool needs its own target.
199+
# It is not a prerequisite of unit-tests: the checks job in pull-requests.yaml
200+
# runs it only when the diff touches a path listed in
201+
# cmd/backport-batch/test-inputs.txt.
199202
test-backport-batch:
200203
go test ./cmd/backport-batch/... -count=1
201204

‎cmd/backport-batch/inputs_test.go‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package main
18+
19+
import (
20+
"os"
21+
"os/exec"
22+
"strings"
23+
"testing"
24+
)
25+
26+
const modulePath = "github.com/cozystack/cozystack/"
27+
28+
// readTestInputs returns the entries of test-inputs.txt, the list CI reads to
29+
// decide whether a pull request runs these tests.
30+
func readTestInputs(t *testing.T) []string {
31+
t.Helper()
32+
data, err := os.ReadFile("test-inputs.txt")
33+
if err != nil {
34+
t.Fatal(err)
35+
}
36+
var entries []string
37+
for _, line := range strings.Split(string(data), "\n") {
38+
line = strings.TrimSpace(line)
39+
if line == "" || strings.HasPrefix(line, "#") {
40+
continue
41+
}
42+
entries = append(entries, line)
43+
}
44+
if len(entries) == 0 {
45+
t.Fatal("test-inputs.txt lists no path")
46+
}
47+
return entries
48+
}
49+
50+
// covered reports whether a change to path matches an entry the way the plan
51+
// job matches it: a trailing slash is a directory prefix, anything else is
52+
// one file.
53+
func covered(entries []string, path string) bool {
54+
for _, e := range entries {
55+
if strings.HasSuffix(e, "/") && strings.HasPrefix(path, e) || path == e {
56+
return true
57+
}
58+
}
59+
return false
60+
}
61+
62+
// A pull request that changes an input of these tests has to run them, and CI
63+
// knows the inputs only from test-inputs.txt. An import added here, or added to
64+
// a package this tool imports, lands in a pull request that touches a listed
65+
// path, so it runs this test, which fails until the new package is listed.
66+
func TestInputsListed(t *testing.T) {
67+
entries := readTestInputs(t)
68+
69+
out, err := exec.Command("go", "list", "-deps", "-test", "-f", "{{.ImportPath}}", ".").Output()
70+
if err != nil {
71+
t.Fatalf("go list: %v", err)
72+
}
73+
var deps int
74+
for _, pkg := range strings.Split(strings.TrimSpace(string(out)), "\n") {
75+
// A test variant prints as "pkg [pkg.test]"; the synthesized main
76+
// package as "pkg.test". Both are this directory.
77+
pkg, _, _ = strings.Cut(pkg, " ")
78+
pkg = strings.TrimSuffix(pkg, ".test")
79+
rel, ok := strings.CutPrefix(pkg, modulePath)
80+
if !ok {
81+
continue
82+
}
83+
deps++
84+
if !covered(entries, rel+"/x.go") {
85+
t.Errorf("these tests import %s, which test-inputs.txt does not list: add %s/", pkg, rel)
86+
}
87+
}
88+
if deps == 0 {
89+
t.Fatalf("go list named no package of %s, so nothing was checked:\n%s", modulePath, out)
90+
}
91+
92+
for _, page := range []string{policyPage, "docs/release/backport-tasks/x.md"} {
93+
if !covered(entries, page) {
94+
t.Errorf("the tests read %s, which test-inputs.txt does not cover", page)
95+
}
96+
}
97+
for _, f := range []string{"go.mod", "go.sum"} {
98+
if !covered(entries, f) {
99+
t.Errorf("test-inputs.txt does not list %s", f)
100+
}
101+
}
102+
}
103+
104+
// A listed path that no longer exists makes the list read as broader than it
105+
// is, and usually means a dependency moved and its new home is missing.
106+
func TestInputsExist(t *testing.T) {
107+
for _, e := range readTestInputs(t) {
108+
if _, err := os.Stat("../../" + e); err != nil {
109+
t.Errorf("test-inputs.txt lists %s: %v", e, err)
110+
}
111+
}
112+
}

‎cmd/backport-batch/test-inputs.txt‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Every path whose change can change the result of
2+
# `go test ./cmd/backport-batch/...`. The plan job in
3+
# .github/workflows/pull-requests.yaml runs those tests when, and only when, the
4+
# pull request touches one of these paths.
5+
#
6+
# One repository-relative path per line. A trailing slash names a directory and
7+
# matches everything below it; anything else matches that file alone. Blank lines
8+
# and lines starting with # are ignored.
9+
#
10+
# TestInputsListed in inputs_test.go keeps this list in step with the code: it
11+
# fails when the tests' in-repo package dependencies (go list -deps -test) or the
12+
# docs pages they read are missing from it.
13+
14+
cmd/backport-batch/
15+
go.mod
16+
go.sum
17+
18+
# In-repo packages the tool and its tests import.
19+
internal/apigate/
20+
pkg/apis/apps/v1alpha1/
21+
pkg/apis/apps/validation/
22+
pkg/config/
23+
24+
# Pages the tests read from the tree.
25+
docs/release/backports.md
26+
docs/release/backport-tasks/

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
121121
- **It trims only which suites `chainsaw test` runs, not the expensive stages.** `Prepare environment` and `Install Cozystack` (the full platform install) run regardless of the selection. TIA only decides the `CHAINSAW_SUITES` passed to the final E2E step. A perfect narrow saves the suite tail, not the bulk of wall-time. (Package image builds are scoped separately by `hack/build-matrix.sh`; the Talos leg runs when its direct or shared inputs changed and remains unconditional for forks, while installer/finalize runs for every non-docs PR. That is a distinct mechanism from TIA's suite trimming.)
122122
- **`Select E2E tests` runs *after* `Install Cozystack` and has no `always()`.** GitHub implicitly ANDs a step's `if:` with `success()`, so if install fails the selector (and `Run E2E tests`) are skipped entirely. On a red install you will not see TIA narrow anything — that is the ordering, not a selector bug.
123123
- **Most PRs legitimately escalate to the full suite.** Platform manifests live under `packages/core/`, Go under `api/`/`internal/`, helpers under `hack/`, plus the `Makefile` — touching any one triggers the full suite. So "all suites selected" is usually correct, not a failure to narrow.
124-
- **Two skip layers, with a gap.** The `plan` job computes a coarse docs-only gate (`code`, true for any change outside `docs/**`) that skips the whole build+E2E pipeline for docs-only PRs, plus a per-package build matrix (`hack/build-matrix.sh`) that scopes which images build. TIA's finer skip (`select-e2e.sh` ignoring `docs/`, `dashboards/`, `*.md` and the rest of `inert_config_pattern`) only trims the suite selection. A PR touching only `dashboards/*.json` therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the `plan` docs-gate, not `select-e2e.sh`.
124+
- **Two skip layers, with a gap.** The `plan` job computes a coarse gate (`code`, false only when every changed path is under `docs/` or one of the CI-only Go tools no image builds, `cmd/backport-batch/` and `cmd/api-gate/`) that skips the whole build+E2E pipeline for such PRs, and `e2e-fork.yaml` skips on the same prefixes (`hack/gate-concurrency-contract.bats` holds the two lists equal), plus a per-package build matrix (`hack/build-matrix.sh`) that scopes which images build. TIA's finer skip (`select-e2e.sh` ignoring `docs/`, `dashboards/`, `*.md` and the rest of `inert_config_pattern`) only trims the suite selection. A PR touching only `dashboards/*.json` therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the `plan` docs-gate, not `select-e2e.sh`.
125125

126126
### 8. Failure must be self-explanatory — diagnostics are first-class
127127

‎docs/agents/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ A chart also declares the limits on its own resource name there, with `## @name
106106

107107
## Testing
108108

109-
- **Helm and BATS unit tests:** `make helm-unit-tests` runs `hack/helm-unit-tests.sh` over every package that defines a `test` target. A `test` target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as `packages/core/testing` already does with its sandbox flow. `make bats-unit-tests` requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with `pre-commit run bats-unit-tests --hook-stage manual --all-files`; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose `/bin/sh` differs from CI, `make BATS_POSIX_SHELL=dash bats-posix-compat-tests` selects the same interpreter explicitly. `make unit-tests` runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under `./internal/...`; run `make test-controllers` as well. CI invokes both targets together.
109+
- **Helm and BATS unit tests:** `make helm-unit-tests` runs `hack/helm-unit-tests.sh` over every package that defines a `test` target. A `test` target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as `packages/core/testing` already does with its sandbox flow. `make bats-unit-tests` requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with `pre-commit run bats-unit-tests --hook-stage manual --all-files`; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose `/bin/sh` differs from CI, `make BATS_POSIX_SHELL=dash bats-posix-compat-tests` selects the same interpreter explicitly. `make unit-tests` runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under `./internal/...`; run `make test-controllers` as well. CI invokes both targets together. The backport-batch tests are not part of it: run `make test-backport-batch` after changing anything listed in `cmd/backport-batch/test-inputs.txt`, which is also what CI keys them on.
110110
- **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md).
111111
- **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers.
112112

0 commit comments

Comments
 (0)