Skip to content

Commit b14f856

Browse files
myasnikovdaniilmyasnikovdaniil
committed
ci(workflows): run the E2E suite on approval instead of on every push
The suite is 176 of a pull request's ~180 minutes and it runs on every push: 4.5 times per pull request on average, and 41 times on one renovate branch. Nothing about that buys coverage. A pull request that reaches main is approved by definition, since renovate carries automerge: false and the ruleset requires one approving review, so gating the expensive job on an approving review costs nothing and stops paying for the suite four times over. dismiss_stale_reviews is already on, so a push after approval invalidates it and the next approval earns a fresh run against the new head. Everything ahead of the suite still runs on a push, so a broken build is still reported in minutes. Only the e2e job waits. Two ways to reach the lane without an approval, which a draft or an author wanting an early run needs. Both are labels, and both are keyed on the label EVENT rather than on the label being present, which makes e2e/run a one-shot button with no code to strip it afterwards: applying it fires `labeled` and that run goes heavy, while the next push does not look at the label list. Removing and re-applying presses it again. Reading the list instead would need pull-requests: write, which a fork's read-only token does not have. e2e-report leaves the pending status the plan opener wrote when the lane was not earned, rather than falling through to its final branch, which would report "in-tree e2e did not pass (result: skipped)". That is fail-closed either way, but a red cross that nobody can explain is how a gate stops being read. The new branch sits after the docs-only and release-promotion cases on purpose: all three mean "e2e did not run" and their outcomes differ, success for the first two so the pull request can merge, pending for this one so it cannot. full-e2e becomes e2e/full and is declared for the first time, alongside e2e/run. It had been read by three workflows for months while existing in no label file, surviving only because the sync runs with delete-other-labels: false. The rename carries an alias, so the existing label is renamed rather than lost, and every consumer moves in the same commit, the way kind/backport did in #2585. The contract tests could not see either new name: gate-concurrency's label extraction matched [a-z0-9-]*, so a namespaced label was invisible and both sides of the complement simply stopped mentioning it, matching each other while covering nothing. Widening the class restores the check, and it then verified this change: the "every publishing guard names the same label set" assertion is what proves e2e/run reached all four guards. promote-gate-contract pins the same names by text in promote-rc.yaml, e2e-fork.yaml and plan's own guard, so its expectations move with them. Fork pull requests are unchanged by this commit and still run their lane per push; e2e-fork.yaml resolves through workflow_run and cannot read the heavy output, so it needs a reviews-API lookup of its own. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 4f337d6 commit b14f856

7 files changed

Lines changed: 138 additions & 46 deletions

File tree

‎.github/labels.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,26 @@
318318
color: '00ff00'
319319
description: Indicates a non-member PR is safe to run CI on
320320

321+
# ──────────────────────────────────────────────
322+
# e2e/ — manual control over the expensive E2E lane
323+
#
324+
# The lane runs on an approving review rather than on every push, so these are
325+
# how it is reached before one exists (a draft, or a change the author wants
326+
# exercised early). Both were undeclared until now: `full-e2e` has been read by
327+
# pull-requests.yaml, e2e-fork.yaml and promote-rc.yaml for months and survived
328+
# only because the sync runs with `delete-other-labels: false`, so a single
329+
# change to that flag would have deleted a label three workflows depend on.
330+
# ──────────────────────────────────────────────
331+
332+
- name: e2e/run
333+
color: '1d76db'
334+
description: Run the E2E suite now, at the scope this diff selects (one-shot; re-apply to run again)
335+
336+
- name: e2e/full
337+
color: '0e4b99'
338+
description: Run the whole E2E suite, bypassing test-impact selection
339+
aliases: ['full-e2e']
340+
321341
# ──────────────────────────────────────────────
322342
# size/ — PR size (auto-applied by .github/workflows/pr-size.yaml)
323343
# ──────────────────────────────────────────────

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ jobs:
144144
// Discarded label event: pull-requests.yaml triggers on `labeled` but
145145
// its `plan` guard discards every label a fork run could not publish
146146
// for, so such a run skips every job and builds nothing. That guard
147-
// admits full-e2e, and admits `release` for SAME-REPO PRs only —
147+
// admits e2e/full, and admits `release` for SAME-REPO PRs only —
148148
// deliberately, for this check. A `release`-labelled run reaches
149149
// `plan` but has `build`/`finalize` gated off by its own label, so it
150150
// succeeds having produced no pr-patch, and this lane would take it
@@ -293,7 +293,7 @@ jobs:
293293
// Test-Impact Analysis inputs, from the SAME trusted sources as the
294294
// docs-only decision above: the base-repo PR file list drives suite
295295
// selection (base64 so newlines survive the job output), and the
296-
// base-repo PR labels drive the full-e2e override. Deriving these
296+
// base-repo PR labels drive the e2e/full override. Deriving these
297297
// from `git diff` on the fork-controlled merge ref would hand a
298298
// fork a second way to make TIA select nothing, on top of the one
299299
// it already has — the selector runs from the fork tree, so this
@@ -322,7 +322,7 @@ jobs:
322322
.flatMap(f => f.previous_filename ? [f.filename, f.previous_filename] : [f.filename])
323323
.join('\n') + '\n';
324324
core.setOutput('changed_b64', Buffer.from(changed, 'utf8').toString('base64'));
325-
const fullE2e = (pr.labels || []).some(l => l.name === 'full-e2e');
325+
const fullE2e = (pr.labels || []).some(l => l.name === 'e2e/full');
326326
core.setOutput('full_e2e', fullE2e ? 'true' : 'false');
327327
328328
// Real run: the pending status was posted above, as soon as this
@@ -672,7 +672,7 @@ jobs:
672672
# .github/workflows/ reads as inert until it is added to the escalation
673673
# list by name, because .github/ is inert as a directory and the e2e
674674
# workflows are exempted from it individually.
675-
# Skipped (→ full suite below) on the full-e2e label.
675+
# Skipped (→ full suite below) on the e2e/full label.
676676
- name: Select E2E tests
677677
id: select
678678
if: ${{ needs.resolve.outputs.full_e2e != 'true' }}
@@ -691,7 +691,7 @@ jobs:
691691
fi
692692
# Single `chainsaw test` invocation, no per-app retry loop (e2e-testing.md
693693
# §1) — Chainsaw polls each assertion to its own timeout. Full suite only
694-
# when the full-e2e label skipped selection: if `select` ran and came back
694+
# when the e2e/full label skipped selection: if `select` ran and came back
695695
# empty it set skip=true and this step never starts.
696696
- name: Run E2E tests
697697
id: e2e_tests

‎.github/workflows/promote-rc.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1000,7 +1000,7 @@ jobs:
10001000
10011001
${E2E_NOTE}
10021002
1003-
ℹ️ E2E already ran against the immutable rc, so this promote PR does not run E2E by default. A maintainer may add the \`full-e2e\` label to run the full suite again.
1003+
ℹ️ E2E already ran against the immutable rc, so this promote PR does not run E2E by default. A maintainer may add the \`e2e/full\` label to run the full suite again.
10041004
10051005
${CHANGELOG_NOTE}
10061006
@@ -1025,7 +1025,7 @@ jobs:
10251025
echo "::warning::The open PR for ${STABLE_BRANCH} has a hand-edited body — leaving it untouched. Changelog status this run: ${CHANGELOG_PRESENT}."
10261026
fi
10271027
else
1028-
# Release-labelled promote PRs deliberately omit full-e2e: the rc
1028+
# Release-labelled promote PRs deliberately omit e2e/full: the rc
10291029
# gate above already checked the immutable candidate. Adding the
10301030
# label later is the explicit maintainer opt-in for a repeat run.
10311031
gh pr create --base "$BASE" --head "${STABLE_BRANCH}" \

0 commit comments

Comments
 (0)