Skip to content

Commit 19108d0

Browse files
ci(workflows): run the E2E suite on approval instead of on every push
The suite is 176 of a pull request's ~180 minutes and it runs on every push: 4.5 times per pull request on average, and 41 times on one renovate branch. Nothing about that buys coverage. A pull request that reaches main is approved by definition, since renovate carries automerge: false and the ruleset requires one approving review, so gating the expensive job on an approving review costs nothing and stops paying for the suite four times over. dismiss_stale_reviews is already on, so a push after approval invalidates it and the next approval earns a fresh run against the new head. Everything ahead of the suite still runs on a push, so a broken build is still reported in minutes. Only the e2e job waits. Two ways to reach the lane without an approval, which a draft or an author wanting an early run needs. Both are labels, and both are keyed on the label EVENT rather than on the label being present, which makes e2e/run a one-shot button with no code to strip it afterwards: applying it fires `labeled` and that run goes heavy, while the next push does not look at the label list. Removing and re-applying presses it again. Reading the list instead would need pull-requests: write, which a fork's read-only token does not have. e2e-report leaves the pending status the plan opener wrote when the lane was not earned, rather than falling through to its final branch, which would report "in-tree e2e did not pass (result: skipped)". That is fail-closed either way, but a red cross that nobody can explain is how a gate stops being read. The new branch sits after the docs-only and release-promotion cases on purpose: all three mean "e2e did not run" and their outcomes differ, success for the first two so the pull request can merge, pending for this one so it cannot. full-e2e becomes e2e/full and is declared for the first time, alongside e2e/run. It had been read by three workflows for months while existing in no label file, surviving only because the sync runs with delete-other-labels: false. The rename carries an alias, so the existing label is renamed rather than lost, and every consumer moves in the same commit, the way kind/backport did in #2585. The contract tests could not see either new name: gate-concurrency's label extraction matched [a-z0-9-]*, so a namespaced label was invisible and both sides of the complement simply stopped mentioning it, matching each other while covering nothing. Widening the class restores the check, and it then verified this change: the "every publishing guard names the same label set" assertion is what proves e2e/run reached all four guards. promote-gate-contract pins the same names by text in promote-rc.yaml, e2e-fork.yaml and plan's own guard, so its expectations move with them. Fork pull requests are unchanged by this commit and still run their lane per push; e2e-fork.yaml resolves through workflow_run and cannot read the heavy output, so it needs a reviews-API lookup of its own. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent ba0b1ac commit 19108d0

7 files changed

Lines changed: 139 additions & 47 deletions

File tree

‎.github/labels.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,26 @@
330330
color: '00ff00'
331331
description: Indicates a non-member PR is safe to run CI on
332332

333+
# ──────────────────────────────────────────────
334+
# e2e/ — manual control over the expensive E2E lane
335+
#
336+
# The lane runs on an approving review rather than on every push, so these are
337+
# how it is reached before one exists (a draft, or a change the author wants
338+
# exercised early). Both were undeclared until now: `full-e2e` has been read by
339+
# pull-requests.yaml, e2e-fork.yaml and promote-rc.yaml for months and survived
340+
# only because the sync runs with `delete-other-labels: false`, so a single
341+
# change to that flag would have deleted a label three workflows depend on.
342+
# ──────────────────────────────────────────────
343+
344+
- name: e2e/run
345+
color: '1d76db'
346+
description: Run the E2E suite now, at the scope this diff selects (one-shot; re-apply to run again)
347+
348+
- name: e2e/full
349+
color: '0e4b99'
350+
description: Run the whole E2E suite, bypassing test-impact selection
351+
aliases: ['full-e2e']
352+
333353
# ──────────────────────────────────────────────
334354
# size/ — PR size (auto-applied by .github/workflows/pr-size.yaml)
335355
# ──────────────────────────────────────────────

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ jobs:
144144
// Discarded label event: pull-requests.yaml triggers on `labeled` but
145145
// its `plan` guard discards every label a fork run could not publish
146146
// for, so such a run skips every job and builds nothing. That guard
147-
// admits full-e2e, and admits `release` for SAME-REPO PRs only —
147+
// admits e2e/full and e2e/run, and admits `release` for SAME-REPO PRs only —
148148
// deliberately, for this check. A `release`-labelled run reaches
149149
// `plan` but has `build`/`finalize` gated off by its own label, so it
150150
// succeeds having produced no pr-patch, and this lane would take it
@@ -293,7 +293,7 @@ jobs:
293293
// Test-Impact Analysis inputs, from the SAME trusted sources as the
294294
// docs-only decision above: the base-repo PR file list drives suite
295295
// selection (base64 so newlines survive the job output), and the
296-
// base-repo PR labels drive the full-e2e override. Deriving these
296+
// base-repo PR labels drive the e2e/full override. Deriving these
297297
// from `git diff` on the fork-controlled merge ref would hand a
298298
// fork a second way to make TIA select nothing, on top of the one
299299
// it already has — the selector runs from the fork tree, so this
@@ -322,7 +322,7 @@ jobs:
322322
.flatMap(f => f.previous_filename ? [f.filename, f.previous_filename] : [f.filename])
323323
.join('\n') + '\n';
324324
core.setOutput('changed_b64', Buffer.from(changed, 'utf8').toString('base64'));
325-
const fullE2e = (pr.labels || []).some(l => l.name === 'full-e2e');
325+
const fullE2e = (pr.labels || []).some(l => l.name === 'e2e/full');
326326
core.setOutput('full_e2e', fullE2e ? 'true' : 'false');
327327
328328
// Real run: the pending status was posted above, as soon as this
@@ -657,7 +657,7 @@ jobs:
657657
# .github/workflows/ reads as inert until it is added to the escalation
658658
# list by name, because .github/ is inert as a directory and the e2e
659659
# workflows are exempted from it individually.
660-
# Skipped (→ full suite below) on the full-e2e label.
660+
# Skipped (→ full suite below) on the e2e/full label.
661661
- name: Select E2E tests
662662
id: select
663663
if: ${{ needs.resolve.outputs.full_e2e != 'true' }}
@@ -708,7 +708,7 @@ jobs:
708708
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
709709
# Each suite runs once, one `chainsaw test` per suite directory, with no
710710
# retry loop (e2e-testing.md §1) — Chainsaw polls each assertion to its own
711-
# timeout. Full suite only when the full-e2e label skipped selection: if
711+
# timeout. Full suite only when the e2e/full label skipped selection: if
712712
# `select` ran and came back empty it set skip=true and this step never
713713
# starts.
714714
- name: Run E2E tests

‎.github/workflows/promote-rc.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1032,7 +1032,7 @@ jobs:
10321032
10331033
${E2E_NOTE}
10341034
1035-
ℹ️ E2E already ran against the immutable rc, so this promote PR does not run E2E by default. A maintainer may add the \`full-e2e\` label to run the full suite again.
1035+
ℹ️ E2E already ran against the immutable rc, so this promote PR does not run E2E by default. A maintainer may add the \`e2e/full\` label to run the full suite again.
10361036
10371037
${CHANGELOG_NOTE}
10381038
@@ -1057,7 +1057,7 @@ jobs:
10571057
echo "::warning::The open PR for ${STABLE_BRANCH} has a hand-edited body — leaving it untouched. Changelog status this run: ${CHANGELOG_PRESENT}."
10581058
fi
10591059
else
1060-
# Release-labelled promote PRs deliberately omit full-e2e: the rc
1060+
# Release-labelled promote PRs deliberately omit e2e/full: the rc
10611061
# gate above already checked the immutable candidate. Adding the
10621062
# label later is the explicit maintainer opt-in for a repeat run.
10631063
gh pr create --base "$BASE" --head "${STABLE_BRANCH}" \

0 commit comments

Comments
 (0)