You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d830e32
Browse filesBrowse the repository at this point in the historyBrowse files
ci(workflows): make the review trigger carry the context it assumed
pull_request_review does not populate github.base_ref or
github.head_ref: GitHub sets those for pull_request and
pull_request_target only. So `git diff origin/${BASE_REF}...HEAD` in
plan resolved as `origin/...HEAD`, died as an ambiguous revision under
bash -e, and every approving review failed plan and posted E2E Tests =
failure instead of starting the suite. finalize's packages-artifact pull
and the TIA selection read the same value, and verify-release-candidate
derived STABLE_VERSION from head_ref, which would have verified a
version that does not exist. All four now read the ref out of the
payload, which every event here carries.
e2e-report, verify-release-candidate and resolve_assets did not get the
review clause plan got, and none of them reaches plan through `needs`.
!cancelled() is a status-check function, so a skipped plan does not hold
e2e-report back on its own: a `commented` or `changes_requested`
submission ran it with plan skipped, read PLAN_RESULT != 'success', and
posted failure over an already-green status. That happened on this
branch (run 33750114876) rather than being read off the docs.
The concurrency key gets the complement of that clause, which the
comment above it had argued was unnecessary. A review submission is not
a `labeled` action, so without its own term a non-approving submission
landed in the main group, cancelled the live suite, and then published
nothing - the same wedge the label split exists to prevent, with a
176-minute run on the line instead of a build. The suffix is -noop
rather than -label now that two kinds of event earn it.
review.state == 'approved' does not say who approved. Anyone with read
access can submit a review on a public repository and each submission
re-fires the event, so that guard let any account start the big job on
any open same-repo pull request as often as it liked. The clause now
also requires author_association in OWNER/MEMBER/COLLABORATOR, matching
the write access the ruleset counts approvals from, and
pull_request.state == 'open', since the event fires on closed pull
requests too.
heavy stops being a pure expression. plan decides it in a step that
first reads the E2E Tests status for the head SHA, so an approving
review earns the lane only when that SHA has no green to reuse. Two
approvals are normal here and the second one cancelled the live suite
and paid for another ~176 minutes on the same answer, which is the cost
this whole change is about. Only success dedupes, so re-running a red
suite still works. That reading is published as `prior` and the opener
consults it: a run that will neither re-verify the head nor conclude the
context must not overwrite the green with a pending that e2e-report
deliberately never concludes, or the gate wedges itself on the good
path. The pending it does write now says which of the two states it is.
e2e/run reaches the release arm alongside e2e/full. Without that it set
heavy, could not reach the suite, and the reporter concluded "e2e not
repeated" - a label named "run" reporting a run that did not happen. The
reporter's release branch answers to either label for the same reason.
The contract tests could not tell the broken workflow from the fixed
one, because the whole review axis was invisible to an extraction that
matches github.event.label.name. gate-concurrency now normalises the
review predicates to the facts they assert, requires all four of plan,
verify-release-candidate, resolve_assets and e2e-report to carry them
the same way round, requires the key to carry them the other way round,
and requires every remaining job to either carry them or name plan in
its needs - which closes the enumeration hole the label loop documents.
It also pins that e2e reads plan's heavy output and that the step
computes it, so replacing the decision with a constant fails. Every
mutation above was run against the suite, including the two that used to
leave it green.
promote-gate-contract stops pinning plan's whole `if:` line. It broke on
every unrelated edit to that guard while proving nothing beyond text
identity, and the axis it was accidentally covering is now covered by
meaning next door. overlay-main-images_test pinned BASE_REF to
github.base_ref by text; it now requires the payload form and rejects
the other, since reading github.base_ref there degrades the overlay to
committed refs on every approving review - #3437's generation mismatch,
reached silently.
docs/release.md still told maintainers to add full-e2e in six places, on
the one path where skipping the suite is a deliberate release-risk call;
the label sync renames it away, so following that would have created a
fresh label no workflow reads. docs/agents/e2e-testing.md said the
e2e/full label forces the whole suite, that each lane concludes the
status when the suite is done, and that the key is the complement of
plan's guard - all three false after this change, on lines the rename
pass had already touched. It now also states the new resting state,
which is what a contributor needs from that file: E2E Tests pending
until an approval or e2e/run.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -94,7 +94,7 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
94
94
- **There is a fourth outcome, and it is a failed step rather than a selection.** It fires when the suite list itself is unavailable: `find` cannot list `hack/e2e-chainsaw/`, or that directory holds no `chainsaw-test.yaml`. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting `E2E Tests` green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under `bash -e`, so the job fails and the required check goes red. A red `Select E2E tests` step is a real signal, not a flake: read its stderr line before rerunning. A `yq` failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot.
95
95
- **Every escalation names its cause on stderr.** "Run everything" is the same list of suite names whichever rule produced it, so a reason line is the only thing that distinguishes the branches that can reach it, and without one the answer to "why did this pull request run the whole suite" has to be re-derived by hand from the diff. Four of them used to be silent — the shared `hack/e2e-chainsaw/_lib/` and `.chainsaw.yaml` case, the `full_suite_pattern` match, a `packages/` path no PackageSource claims, and the empty-selection backstop — and `full_suite_pattern` is the commonest cause of a full run by a wide margin, so the usual answer was the one the log never carried. Each line names the path or the sources that caused it, and a unit test asserts each message, because a reason that regresses to silence changes no selection and is invisible to every other test. The lines go to **stderr** and must stay there: stdout is the suite list and both lanes parse it, so a reason line on stdout would be read as a suite name.
96
96
- **The one path the fall-through cannot catch is a new e2e workflow.** `inert_config_pattern` makes `.github/` inert as a directory, and the workflows that run the suite are exempted from it by name in `full_suite_pattern`. A workflow added under `.github/workflows/` therefore reads as inert until someone adds it to that enumeration, and the fall-through never sees it. That is why the list carries a `rg -l test-chainsaw .github/workflows/` reminder: the enumeration is the guard, not a convenience.
97
-
- The `e2e/full` PR label forces the whole suite.
97
+
- The `e2e/full` PR label forces the whole suite whenever the lane runs. Scope is read from the label LIST, so it holds for every later run too, while whether the lane runs at all is a separate decision made per run (§10): applying the label earns the lane once, and a push with the label still attached selects the whole suite but does not run it until an approval or `e2e/run`.
98
98
- The selection is passed to `make test-chainsaw CHAINSAW_SUITES="<names>"`, which runs `chainsaw test <names>` from `hack/e2e-chainsaw/` (empty = the whole suite).
99
99
- The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into `packages/core/` (and other packages), which TIA escalates to the **full** suite — so the release PR exercises the whole suite against the commit the tag will point at.
100
100
@@ -147,9 +147,10 @@ A fork's `pull_request` run carries no registry credentials, so it cannot push t
147
147
148
148
Consequences to keep in mind when touching either workflow:
149
149
150
+
- **The suite runs on an approving review, not on every push.** So `E2E Tests` sits at `pending` for the whole life of an unapproved pull request, and that is its resting state rather than a fault: `plan` opens the context with `waiting for an approving review (or the e2e/run label)` and `e2e-report` deliberately posts nothing when the lane was not earned, so the `pending` stands. The one run that does NOT open it is a run that will neither re-verify the head nor conclude the context and finds a green already there — a second approving review, a reopen, a re-run of a run that passed — because writing `pending` over that green would leave the pull request unmergeable with nothing on the way to fix it. An approving review from an account with write access earns the lane and the suite runs against that head; `dismiss_stale_reviews_on_push` is on, so a push invalidates the approval and the next one earns a fresh run. Everything ahead of the suite — the build matrix, talos, finalize, the unit lane — still runs on the push, so a broken build is still reported in minutes; what waits is the 176 minutes of suite. Two labels reach the lane without an approval, which is how a draft, or an author who wants the change exercised early, gets there: `e2e/run` runs it now at the scope the diff selects, `e2e/full` runs the whole suite. Both are keyed on the label EVENT rather than on the label being present, which makes `e2e/run` a one-shot button — applying it fires `labeled` and that run goes heavy, the next push does not look at the label list, and pressing it again means removing and re-applying the label. An approving review on a head that already carries a green `E2E Tests` does not re-run the suite, so a second approval is free and leaves the green alone; a red or unset one does re-run, so re-running a flaked suite still works.
150
151
- **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`.
151
-
- **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `e2e/full` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:
152
-
- *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate.
152
+
- **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending`. Each concludes it when the suite is done — and the same-repo lane deliberately leaves it open when the lane was not earned, which is the resting state described above rather than a lane that failed to report. The opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `e2e/full` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first status write — only the step deciding whether the lane is earned precedes it, because that step reads the very status the opener is about to overwrite — so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:
153
+
- *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. That property is now load-bearing in the ordinary case and not only in the failure case: an unapproved pull request has never had the suite run against its head, and the unconcluded `pending` is exactly what keeps it unmergeable. Read a pending `E2E Tests` as "the lane has not been earned yet", not as a stuck run. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate.
153
154
- *Not a check-run:* a check-run created with the default `GITHUB_TOKEN` floats into an arbitrary GitHub Actions check-suite (in practice the earliest-registered one, a tiny labeler — that is why `E2E Tests` once rendered under "PR size label"), and an app-token check-run would need the COZYSTACK_CI app granted `checks: write`. A commit status has neither problem: branch protection matches it by context name and it carries no check-suite grouping. It is safe to post with the default token because a fork's `pull_request` token is **read-only** and cannot post a status — so only same-repo runs (`plan`'s opener and `e2e-report`, both guarded `!fork`) and the privileged `workflow_run` in `e2e-fork.yaml` can set it. Each publishing job carries `permissions: statuses: write` and calls `repos.createCommitStatus`. `plan`'s opener is additionally `continue-on-error`, since every build and test job hangs off `plan` and a transient status-API failure there must not discard the run; losing it only restores the stale-green window for that one run.
154
155
**Consequence for branch protection:** require the `E2E Tests` **status context** (no GitHub App and no app-id pin needed).
155
156
@@ -158,7 +159,7 @@ Consequences to keep in mind when touching either workflow:
158
159
159
160
- **A fork PR that adds a new image cannot go green until that image exists on the base branch.** `publish` derives the set of archive names it will accept from the base parent (`HEAD^1`), so that a fork cannot mint arbitrary repositories under the CI registry by naming an artifact. An image the PR itself introduces is therefore not in that set, and the job fails with `image '<name>' is not in the base-tree allowlist`. This is fail-closed by design and there is no label override — land the image on the base branch first, or run the suite from a same-repo branch. If a fork e2e run fails on that line, the contribution is not broken; the allowlist is doing its job.
160
161
161
-
- **`labeled` events run in their own concurrency group, and a cancelled run reports nothing.** Because the gate is a status rather than a job, a run that dies without posting leaves the head SHA at "Expected" — which blocks the merge rather than passing it. Several separate pieces implement it, and weakening any one of them reopens the gap. Each lane's terminal reporter runs under `!cancelled()` rather than `always()`, so a run that was cancelled publishes nothing at all and the `pending` it opened stands until the run that superseded it reports. Each lane's concurrency key then keeps a run that is going to publish nothing out of the group of the run that is: `pull-requests.yaml`'s appends `-label` for a `labeled` event that `plan` discards, so labelling an open PR cannot cancel a live `opened`/`synchronize` run (the replacement run would never post the status the cancelled run was going to) — the condition is the complement of `plan`'s guard, so a `e2e/full` label, which does publish, stays in the main group and supersedes the head run instead of racing it, `e2e-fork.yaml`'s concurrency key appends the triggering run's id when that run concluded `cancelled` or `skipped` so a privileged run that is going to post nothing cannot cancel the one that is going to post, and `e2e-fork.yaml`'s `resolve` returns early on a `cancelled`/`skipped` triggering run instead of posting `failure` for it. If you touch any of them, keep the invariant: exactly one run per head SHA is responsible for the status, and a run that is not that one must post nothing at all.
162
+
- **Runs that will publish nothing get their own concurrency group, and a cancelled run reports nothing.** Because the gate is a status rather than a job, a run that dies without posting leaves the head SHA at "Expected" — which blocks the merge rather than passing it. Several separate pieces implement it, and weakening any one of them reopens the gap. Each lane's terminal reporter runs under `!cancelled()` rather than `always()`, so a run that was cancelled publishes nothing at all and the `pending` it opened stands until the run that superseded it reports. Each lane's concurrency key then keeps a run that is going to publish nothing out of the group of the run that is: `pull-requests.yaml`'s appends `-noop` for either kind of event `plan` discards — a `labeled` event for a label it does not act on, and a review submission that is not an admissible approval — so neither labelling nor commenting on an open PR can cancel a live `opened`/`synchronize` run (the replacement run would never post the status the cancelled run was going to). The condition is the complement of `plan`'s guard on BOTH axes, so a `e2e/full` label or an approving review, which do publish, stay in the main group and supersede the head run instead of racing it; the review half was argued to be unnecessary once, on the grounds that a review is not a `labeled` action, and the answer was a `changes_requested` submission that cancelled the live run and posted red on a SHA nobody had touched, `e2e-fork.yaml`'s concurrency key appends the triggering run's id when that run concluded `cancelled` or `skipped` so a privileged run that is going to post nothing cannot cancel the one that is going to post, and `e2e-fork.yaml`'s `resolve` returns early on a `cancelled`/`skipped` triggering run instead of posting `failure` for it. If you touch any of them, keep the invariant: exactly one run per head SHA is responsible for the status, and a run that is not that one must post nothing at all.
162
163
163
164
**Security prerequisite (repo setting).** The split is only safe if fork runs require maintainer approval before they execute. The repository must keep **Settings → Actions → "Require approval for all outside collaborators"** enabled, so a fork's `pull_request` run — which produces the artifacts `e2e-fork.yaml` consumes — cannot run untrusted code (or feed the privileged run) without a maintainer's go-ahead.
0 commit comments