Skip to content

Commit 0788e94

Browse files
feat(ci): test-impact analysis E2E (default-on) + release E2E workflow
Three changes that compose into a single E2E policy: 1. **Test-impact analysis selector** (`hack/select-e2e.sh`). Reads `packages/core/platform/sources/*.yaml` (the PackageSource dependency graph also rendered by `cozypkg dot`) and emits the bats files affected by a PR diff. Library/core/networking/CI changes, shared `hack/*.sh` helpers, the install bats, and the `pull-requests` / `release-e2e` workflows trigger the full suite. Doc/dashboard/`*.md` diffs select nothing. Per-app bats edits select only that app. Anything inside `packages/` with no graph entry, or a system source whose graph has no `*-application` descendants, falls back to the full suite (conservative). 2. **Default-on TIA in `pull-requests.yaml`** with `full-e2e` opt-out. Drops the previous `e2e-tia` opt-in shape — the selector's behaviour has been validated, so making it the default gives PR runtime savings out of the box. PRs that need the full suite (e.g. cross-cutting refactors that the selector can't reason about) carry the `full-e2e` label and get the unchanged pre-TIA flow. Docs-only PRs (selector emits nothing) skip the `Run E2E tests` step entirely. 3. **`release-e2e.yaml`** runs the full suite on every tag push (`v*`). Independent of `tags.yaml`'s release publish flow — a failure here is an alarm, not a release blocker. Closes the coverage gap left by making PR runs use TIA: the full suite is now exercised against shipped code at tag-cut time. Bug fixes vs. the original `feat/select-e2e-script` draft: - `dashboard/` -> `dashboards/` (path is plural in this repo). - `hack/e2e-apps/<name>.sh` and `hack/<name>.bats` were silently ignored; they now trigger the full suite (test infrastructure affecting all apps). - A path matching the graph but yielding zero `*-application` descendants now falls back to the full suite instead of silently emitting nothing (system-only changes are no longer silently skipped). - Per-app bats edits are matched before the full-suite pattern, so editing one bats file no longer escalates to the full suite. - Removed the dead `skip_pattern` variable (PCRE syntax that `grep -qE` doesn't support; it was never matched). Tests: `hack/select-e2e_test.bats` covers all 11 cases (the original 6 plus 5 for the bug fixes above). Workflow inputs are passed via env vars rather than direct `${{ }}` expansion into shell commands, per workflow injection guidance. Signed-off-by: Myasnikov Daniil <[email protected]> Assisted-By: Claude <[email protected]>
1 parent 5786afe commit 0788e94

4 files changed

Lines changed: 441 additions & 2 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -249,18 +249,51 @@ jobs:
249249
done
250250
echo "✅ The task completed successfully after $attempt attempts"
251251
252+
# Test Impact Analysis (TIA): walk the PackageSource graph for the PR
253+
# diff and select only the bats files affected. Runs by default; opt
254+
# out via the `full-e2e` label to run the full suite.
255+
# `skip=true` means nothing to test (docs-only PR).
256+
- name: Select E2E tests
257+
id: select
258+
if: ${{ !contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
259+
env:
260+
BASE_REF: ${{ github.base_ref }}
261+
run: |
262+
cd /tmp/$SANDBOX_NAME
263+
git diff --name-only "origin/${BASE_REF}...HEAD" > /tmp/changed.txt
264+
apps=$(./hack/select-e2e.sh /tmp/changed.txt)
265+
echo "Selected apps: ${apps:-<none>}"
266+
echo "apps=${apps}" >> $GITHUB_OUTPUT
267+
if [ -z "${apps}" ]; then
268+
echo "skip=true" >> $GITHUB_OUTPUT
269+
else
270+
echo "skip=false" >> $GITHUB_OUTPUT
271+
fi
272+
252273
- name: Run OpenAPI tests
253274
run: |
254275
cd /tmp/$SANDBOX_NAME
255276
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
256277
257-
# ▸ Run E2E tests
278+
# Run E2E tests. With `full-e2e` label the selector is skipped, its
279+
# outputs are empty, the != 'true' guard lets this step run, and the
280+
# full bats list is used. Without the label, the selector decides;
281+
# an empty selection (docs-only) skips this step entirely.
258282
- name: Run E2E tests
259283
id: e2e_tests
284+
if: ${{ steps.select.outputs.skip != 'true' }}
285+
env:
286+
SELECTED_APPS: ${{ steps.select.outputs.apps }}
287+
FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
260288
run: |
261289
cd /tmp/$SANDBOX_NAME
290+
if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]; then
291+
apps_list=$(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1)
292+
else
293+
apps_list="$SELECTED_APPS"
294+
fi
262295
failed_tests=""
263-
for app in $(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1); do
296+
for app in $apps_list; do
264297
echo "::group::Testing $app"
265298
attempt=0
266299
success=false

‎.github/workflows/release-e2e.yaml‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
name: Release E2E
2+
3+
# Runs the full E2E suite against the tagged commit. Independent of the
4+
# release publish flow (`tags.yaml`) — a failure here is an alarm, not a
5+
# release blocker. PR-time E2E runs Test Impact Analysis by default, so
6+
# tags are the only place the full suite is exercised against shipped code.
7+
8+
env:
9+
# TODO: unhardcode this
10+
REGISTRY: iad.ocir.io/idyksih5sir9/cozystack
11+
on:
12+
push:
13+
tags:
14+
- 'v*.*.*'
15+
- 'v*.*.*-rc.*'
16+
- 'v*.*.*-beta.*'
17+
- 'v*.*.*-alpha.*'
18+
19+
concurrency:
20+
group: release-e2e-${{ github.ref }}
21+
cancel-in-progress: false
22+
23+
jobs:
24+
build:
25+
name: Build (release E2E)
26+
runs-on: [self-hosted]
27+
timeout-minutes: 30
28+
permissions:
29+
contents: read
30+
packages: write
31+
32+
steps:
33+
- name: Checkout code
34+
uses: actions/checkout@v4
35+
with:
36+
fetch-depth: 0
37+
fetch-tags: true
38+
39+
- name: Login to registry
40+
uses: docker/login-action@v3
41+
with:
42+
registry: iad.ocir.io
43+
username: ${{ secrets.OCI_REGISTRY_USERNAME }}
44+
password: ${{ secrets.OCI_REGISTRY_PASSWORD }}
45+
46+
- name: Build images
47+
run: make build PUSH=1
48+
49+
- name: Build Talos image
50+
run: make -C packages/core/talos build
51+
52+
- name: Upload Talos image
53+
uses: actions/upload-artifact@v4
54+
with:
55+
name: talos-image
56+
path: _out/assets/nocloud-amd64.raw.xz
57+
retention-days: 1
58+
59+
e2e:
60+
name: Release E2E (full suite)
61+
runs-on: [self-hosted]
62+
timeout-minutes: 120
63+
permissions:
64+
contents: read
65+
packages: read
66+
needs: ["build"]
67+
68+
steps:
69+
- name: Checkout code
70+
uses: actions/checkout@v4
71+
72+
- name: Download Talos image
73+
uses: actions/download-artifact@v4
74+
with:
75+
name: talos-image
76+
path: _out/assets
77+
78+
- name: Set sandbox ID
79+
run: echo "SANDBOX_NAME=cozy-e2e-release-$(echo "${GITHUB_REPOSITORY}:${GITHUB_REF}" | sha256sum | cut -c1-10)" >> $GITHUB_ENV
80+
81+
- name: Prepare workspace
82+
env:
83+
WORKSPACE: ${{ github.workspace }}
84+
run: |
85+
rm -rf /tmp/$SANDBOX_NAME
86+
cp -r "$WORKSPACE" /tmp/$SANDBOX_NAME
87+
88+
- name: Prepare environment
89+
run: |
90+
cd /tmp/$SANDBOX_NAME
91+
attempt=0
92+
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
93+
attempt=$((attempt + 1))
94+
if [ $attempt -ge 3 ]; then
95+
echo "❌ Attempt $attempt failed, exiting..."
96+
exit 1
97+
fi
98+
echo "❌ Attempt $attempt failed, retrying..."
99+
done
100+
echo "✅ The task completed successfully after $attempt attempts"
101+
102+
- name: Install Cozystack into sandbox
103+
run: |
104+
cd /tmp/$SANDBOX_NAME
105+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack
106+
107+
- name: Run OpenAPI tests
108+
run: |
109+
cd /tmp/$SANDBOX_NAME
110+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
111+
112+
# Always full suite — release E2E intentionally does not honour the
113+
# `full-e2e` / TIA mechanism used by PR runs.
114+
- name: Run E2E tests (full suite)
115+
id: e2e_tests
116+
run: |
117+
cd /tmp/$SANDBOX_NAME
118+
failed_tests=""
119+
for app in $(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1); do
120+
echo "::group::Testing $app"
121+
if make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-apps-$app; then
122+
echo "::endgroup::"
123+
echo "✅ Test $app completed successfully"
124+
else
125+
echo "::endgroup::"
126+
echo "❌ Test $app failed"
127+
failed_tests="$failed_tests $app"
128+
echo "::group::Diagnostics for $app"
129+
kubectl get hr -A -o wide 2>&1 | tail -50 || true
130+
kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -50 || true
131+
echo "::endgroup::"
132+
fi
133+
done
134+
if [ -n "$failed_tests" ]; then
135+
echo "❌ Failed tests:$failed_tests"
136+
exit 1
137+
fi
138+
echo "✅ All E2E tests passed"
139+
140+
- name: Collect report
141+
if: always()
142+
run: |
143+
cd /tmp/$SANDBOX_NAME
144+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME collect-report || true
145+
146+
- name: Upload cozyreport.tgz
147+
if: always()
148+
uses: actions/upload-artifact@v4
149+
with:
150+
name: cozyreport-${{ github.ref_name }}
151+
path: /tmp/${{ env.SANDBOX_NAME }}/_out/cozyreport.tgz
152+
153+
- name: Tear down sandbox
154+
if: always()
155+
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
156+
157+
- name: Remove workspace
158+
if: always()
159+
run: rm -rf /tmp/$SANDBOX_NAME

‎hack/select-e2e.sh‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#!/bin/sh
2+
# Read a list of changed files (one per line) and emit space-separated app
3+
# names whose bats files in hack/e2e-apps/ should run.
4+
#
5+
# Usage: hack/select-e2e.sh <changed-files> [<sources-dir>]
6+
# Defaults: sources-dir = packages/core/platform/sources
7+
#
8+
# Output:
9+
# - empty no E2E impact (docs / dashboards / *.md only)
10+
# - <app names> selected per the PackageSource dependency graph
11+
# - full list any path that affects all tests, OR an unrecognised
12+
# packages/* path, OR a per-app source whose graph has
13+
# no *-application descendants (conservative fallback)
14+
set -eu
15+
16+
CHANGED="${1:?missing changed-files arg}"
17+
SOURCES_DIR="${2:-packages/core/platform/sources}"
18+
19+
# Anything matching this pattern triggers the full bats suite. Per-app bats
20+
# (hack/e2e-apps/<name>.bats) are matched BEFORE this so editing one bats
21+
# file doesn't escalate to the full suite.
22+
full_suite_pattern='^(packages/library/|packages/core/|api/|cmd/|internal/|hack/[^/]+\.sh$|hack/[^/]+\.bats$|hack/e2e-apps/[^/]+\.sh$|Makefile$|\.github/workflows/(pull-requests|release-e2e)\.yaml$)'
23+
24+
# All known per-app bats files
25+
all_apps=$(ls hack/e2e-apps/*.bats 2>/dev/null | xargs -n1 basename | sed 's/\.bats$//')
26+
27+
# PackageSource name -> bats name(s). Most *-application sources map by
28+
# stripping the suffix; explicit overrides for the few that don't.
29+
app_to_bats() {
30+
case "$1" in
31+
postgres-application) echo postgres ;;
32+
vm-instance-application) echo vminstance ;;
33+
kubernetes-application) echo "kubernetes-latest kubernetes-previous" ;;
34+
external-dns) echo external-dns ;;
35+
*-application) echo "${1%-application}" ;;
36+
*) echo "$1" ;;
37+
esac
38+
}
39+
40+
# yq: path -> PackageSource name
41+
build_owners_index() {
42+
for f in "$SOURCES_DIR"/*.yaml; do
43+
src=$(yq -r '.metadata.name' "$f")
44+
yq -r '.spec.variants[]?.components[]?.path // ""' "$f" | while read -r path; do
45+
if [ -n "$path" ]; then echo "$path $src"; fi
46+
done
47+
done
48+
}
49+
50+
# yq: PackageSource name -> sources that depend on it (reverse of dependsOn)
51+
build_reverse_deps() {
52+
for f in "$SOURCES_DIR"/*.yaml; do
53+
src=$(yq -r '.metadata.name' "$f")
54+
yq -r '.spec.variants[]?.dependsOn[]? // ""' "$f" | while read -r dep; do
55+
if [ -n "$dep" ]; then echo "$dep $src"; fi
56+
done
57+
done
58+
}
59+
60+
OWNERS=$(build_owners_index | sort -u)
61+
REVERSE=$(build_reverse_deps | sort -u)
62+
63+
trigger_full=0
64+
trigger_any=0
65+
selected_sources=""
66+
selected_apps=""
67+
68+
while IFS= read -r file; do
69+
[ -z "$file" ] && continue
70+
71+
# 1. Skip: docs, dashboards, *.md
72+
if echo "$file" | grep -qE '^(docs/|dashboards/)' || echo "$file" | grep -qE '\.md$'; then
73+
continue
74+
fi
75+
76+
# 2. Per-app bats first — editing one bats file selects only that app.
77+
if echo "$file" | grep -qE '^hack/e2e-apps/[^/]+\.bats$'; then
78+
app=$(basename "$file" .bats)
79+
selected_apps="$selected_apps $app"
80+
trigger_any=1
81+
continue
82+
fi
83+
84+
# 3. Full-suite trigger
85+
if echo "$file" | grep -qE "$full_suite_pattern"; then
86+
trigger_full=1
87+
continue
88+
fi
89+
90+
# 4. Component change: lookup in PackageSource graph
91+
rel=$(echo "$file" | sed -nE 's,^packages/(apps|system|extra)/([^/]+)/.*,\1/\2,p')
92+
if [ -n "$rel" ]; then
93+
src=$(echo "$OWNERS" | awk -v p="$rel" -F'\t' '$1==p {print $2}')
94+
if [ -n "$src" ]; then
95+
selected_sources="$selected_sources $src"
96+
trigger_any=1
97+
else
98+
# Inside packages/ but no graph entry — be conservative.
99+
trigger_full=1
100+
fi
101+
fi
102+
# Anything else (e.g. unrelated workflow files, top-level configs) is
103+
# silently ignored.
104+
done < "$CHANGED"
105+
106+
if [ "$trigger_full" = 1 ]; then
107+
echo "$all_apps" | tr '\n' ' '
108+
exit 0
109+
fi
110+
111+
if [ "$trigger_any" = 0 ]; then
112+
exit 0 # nothing to run
113+
fi
114+
115+
# Transitive closure: walk reverse-deps from each selected source.
116+
all_sources="$selected_sources"
117+
while :; do
118+
new=""
119+
for s in $all_sources; do
120+
deps=$(echo "$REVERSE" | awk -v src="$s" -F'\t' '$1==src {print $2}')
121+
for d in $deps; do
122+
case " $all_sources " in *" $d "*) ;; *) new="$new $d";; esac
123+
done
124+
done
125+
[ -z "$new" ] && break
126+
all_sources="$all_sources $new"
127+
done
128+
129+
# Filter to *-application sources, then map to bats names.
130+
final=""
131+
for s in $all_sources; do
132+
app=${s#cozystack.}
133+
case "$app" in
134+
*-application) final="$final $(app_to_bats "$app")" ;;
135+
external-dns) final="$final external-dns" ;;
136+
esac
137+
done
138+
139+
# Add directly-selected apps from per-app bats edits.
140+
final="$final $selected_apps"
141+
142+
# Deduplicate; intersect with available bats files.
143+
final_apps=$(echo "$final" | tr ' ' '\n' | sort -u | grep -v '^$' | while read -r app; do
144+
if echo "$all_apps" | grep -qw "$app"; then
145+
echo "$app"
146+
fi
147+
done | paste -sd ' ' -)
148+
149+
# Safety net: a system source with no *-application descendants would otherwise
150+
# silently skip E2E. Fall back to full suite so a path inside the graph is
151+
# never silently dropped.
152+
if [ -z "$final_apps" ]; then
153+
echo "$all_apps" | tr '\n' ' '
154+
exit 0
155+
fi
156+
157+
echo "$final_apps"

0 commit comments

Comments
 (0)