Skip to content

Commit 7dca642

Browse files
fix(ci): restore e2e for fork PRs and close the merge-gate bypass (#3262)
## What this PR does Fixes #3257 — fork PRs never ran e2e, and the skipped required check let them merge anyway. Fork PRs get no secrets on `pull_request`, so `make image` pushed anonymously and died with `denied`. That failed every build, collapsed the `build → finalize → e2e` chain, and left the required **E2E Tests** job *skipped* — which branch protection treats as satisfied, so an approved fork PR could merge with e2e never having run. This implements **Option A** from the issue (privileged `workflow_run` split), pushing fork e2e images to the existing CI registry (OCIR): **Build primitive (`hack/common-envs.mk`)** - New `OCI_EXPORT_DIR` mode: build each image to a per-image OCI archive instead of pushing, and force `PUSH`/`LOAD` off. The digest is captured via `--metadata-file` regardless of output type, so the refs baked into `pr.patch` match what the privileged run later pushes. Threaded through the `image-tags` macro every package already uses — no per-package Makefile change. **`pull-requests.yaml` (unprivileged, fork branch)** - `build` / `build-talos` / `finalize` export images to OCI archives and upload them as artifacts instead of pushing → the red `denied` build wall disappears and the chain no longer collapses. - The in-tree e2e job is renamed off `"E2E Tests"` and guarded to same-repo PRs. A new `e2e-report` job concludes the required `"E2E Tests"` context as an **explicit check-run** (success on a docs-only PR or when e2e passed, failure otherwise). A skipped job can no longer satisfy the required check — this also closes the same hole for same-repo PRs whose build fails. **`e2e-fork.yaml` (new, privileged `workflow_run` from the default branch)** - Resolves the fork PR, opens a pending `"E2E Tests"` check-run, and fails-closed if the fork build failed. - Pushes the pre-built image archives to OCIR **by digest** and flux-pushes the digest-pinned packages tree. The OCIR token is live only for trusted binaries (`git`/`skopeo`/`flux`/`yq`) over data — never for `make` or any fork-authored script. - Runs the e2e suite against the pushed closure with **no push token** and concludes the check-run on the PR head SHA. Fork build code never receives registry credentials; the fork's images and test scripts run only in the credential-less, ephemeral e2e job — a stronger posture than a `pull_request_target` run that would hand fork code the secrets directly. ### Validation Local: buildx OCI-archive output + digest capture, macro expansion for single- and multi-image packages, non-fork push path unchanged, `actionlint` clean. Needs live validation (why this is a **draft**): `workflow_run` artifact download across runs; that the `"E2E Tests"` commit status satisfies the required context and keeps a fork PR non-mergeable until the privileged run concludes it; `refs/pull/<N>/merge` fetch + `git apply --3way`; `skopeo --preserve-digests` end-to-end. `workflow_run` only runs from the default branch, so the privileged half is exercised by mirroring the workflow onto a personal fork's default branch and opening a throwaway fork PR into it — not by merging first. _Since review_: the fork `e2e` job ran the pre-#2826 BATS app loop that the merged Chainsaw migration deleted (already broken); it now runs the same Chainsaw + TIA path as the in-tree job, with suite selection and the `full-e2e` override sourced from `resolve`'s trusted base-repo file list / labels (not a merge-ref diff). Publish job hardened per the non-blocking follow-ups: base-tree image allowlist, `skopeo copy --all`, pinned flux CLI. The required `E2E Tests` gate is now published as a commit **status** (default token + `statuses:write`), not a COZYSTACK_CI app check-run — the app lacked `checks:write` (`checks.create` 403'd) and a default-token check-run floated under a labeler suite; a commit status has neither problem, and forks can't post one (their `pull_request` token is read-only). Two review-bot findings fixed too: fork `OCI_EXPORT_DIR` is now absolute (a relative path misplaced archives under `make -C <pkg>` and every fork build would fail at upload), and the "≥1 archive" guard no longer no-ops under `nullglob`. ### Repo-admin prerequisites One repository setting lives outside this diff: 1. **Branch protection: keep `E2E Tests` required.** The gate is now a commit **status** named `E2E Tests` (posted with the default `GITHUB_TOKEN` — see `docs/agents/e2e-testing.md` §10), not a job or an app check-run. Branch protection already requires `E2E Tests` (currently the e2e job's name, renamed here to `E2E (in-tree)`), so the required context now resolves to the status — **no GitHub App, no `checks:write` grant, no app-id pin needed**. Just don't pin the context to a specific app. 2. **Actions → "Require approval for all outside collaborators".** Each fork run must be a deliberate maintainer click before it executes, since the fork's unprivileged run produces the artifacts the privileged `e2e-fork.yaml` consumes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (7 files: two CI workflows, `docs/agents/e2e-testing.md`, `hack/common-envs.mk` + its bats suite, and two package `Makefile`s). The one trigger worth naming is "change developer tooling (`cozyvalues-gen`, `cozypkg`, the package `Makefile`s) → `development.md`" on the website. It does not fire here: `OCI_EXPORT_DIR` is a CI-only build mode that defaults off, and the local developer workflow is unchanged — `make image` still builds and pushes exactly as before, which the new default-path case in `hack/common-envs_test.bats` now pins. No chart values, no API surface, no app list, no platform variant, no Talos version bump and no release asset rename are touched. - [x] No downstream repository is affected by this change ### Release note ```release-note fix(ci): fork pull requests now run e2e via a privileged workflow, and a fork PR can no longer merge on a skipped (never-run) required E2E Tests check ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a dedicated fork E2E workflow that runs after the base PR and reports results via the required **E2E Tests** commit status. - Fork PRs now export build images as OCI archive artifacts for later publishing. - Added a same-repo E2E reporting job and renamed the in-tree E2E flow to **E2E (in-tree)**. - **Bug Fixes** - Added tighter, fail-closed gating to ensure merge blocking when **E2E Tests** isn’t correctly finalized. - **Documentation** - Documented fork vs same-repo E2E split, OCI export behavior, and required fork security settings. - **Tests** - Added Bats coverage validating `OCI_EXPORT_DIR` and preventing unintended registry pushes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents bc6d5e6 + 3af1ad3 commit 7dca642

8 files changed

Lines changed: 1461 additions & 24 deletions

File tree

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 735 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/workflows/pull-requests.yaml‎

Lines changed: 286 additions & 13 deletions
Large diffs are not rendered by default.

‎docs/agents/e2e-testing.md‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# E2E Testing Conventions
22

3-
Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, the E2E CI workflow (`.github/workflows/pull-requests.yaml`), or `packages/core/testing/`.
3+
Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, either E2E CI workflow (`.github/workflows/pull-requests.yaml` for same-repo pull requests, `.github/workflows/e2e-fork.yaml` for forks), or `packages/core/testing/`. §10 covers how the two split the work and which invariants hold them together.
44

55
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
66

@@ -68,7 +68,7 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
6868

6969
`hack/select-e2e.sh` walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under `hack/e2e-chainsaw/` containing a `chainsaw-test.yaml`.
7070

71-
- Conservative escalation: edits to `packages/library/`, `packages/core/`, `api/`, `cmd/`, `internal/`, top-level `hack/*.sh|*.bats` helpers, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or the E2E workflows escalate to the **full suite**. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite.
71+
- Conservative escalation: edits to `packages/library/`, `packages/core/`, `api/`, `cmd/`, `internal/`, top-level `hack/*.sh|*.bats` helpers, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or `.github/workflows/pull-requests.yaml` escalate to the **full suite**. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite. Note the exception, since "the E2E workflows" would be the natural reading: the selector's list names `pull-requests.yaml` and not `e2e-fork.yaml`, so a change touching only the fork workflow selects nothing and the Chainsaw step is skipped, while the platform install and the OpenAPI tests still run. That is tolerable rather than designed, and it is tolerable for a specific reason: a `workflow_run` always executes the default-branch copy of `e2e-fork.yaml`, so running the suite against a change to it would not exercise that change anyway. It is the same fail-open shape as #3392.
7272
- The `full-e2e` PR label forces the whole suite.
7373
- The selection is passed to `make test-chainsaw CHAINSAW_SUITES="<names>"`, which runs `chainsaw test <names>` from `hack/e2e-chainsaw/` (empty = the whole suite).
7474
- The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into `packages/core/` (and other packages), which TIA escalates to the **full** suite — so the release PR exercises the whole suite against the commit the tag will point at.
@@ -104,6 +104,30 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
104104
- **Exclude loop devices** from host LVM scanning in the Talos machine config so the host does not activate volume groups inside loop-mounted e2e disk images.
105105
- **Fail fast on node readiness** (≈5m, then bail) rather than marching into LB/NFS tests that will also fail — it saves several minutes per attempt and keeps the real failure at the top of the log.
106106

107+
### 10. The E2E workflow split: same-repo vs fork, and the "E2E Tests" commit status
108+
109+
A fork's `pull_request` run carries no registry credentials, so it cannot push the images the e2e sandbox pulls. E2E therefore runs across **two** workflows:
110+
111+
- **`.github/workflows/pull-requests.yaml`** runs e2e in-tree for **same-repo** PRs (the `e2e` job, named `E2E (in-tree)`). Fork PRs skip this job.
112+
- **`.github/workflows/e2e-fork.yaml`** runs e2e for **fork** PRs via a privileged `workflow_run` that fires after the fork's Pull Request run completes. It pushes the fork-built images to the registry and runs the suite — the same Chainsaw + TIA path as in-tree (§7) — in an ephemeral sandbox that itself has no push credentials.
113+
114+
Consequences to keep in mind when touching either workflow:
115+
116+
- **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`.
117+
- **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `full-e2e` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:
118+
- *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate.
119+
- *Not a check-run:* a check-run created with the default `GITHUB_TOKEN` floats into an arbitrary GitHub Actions check-suite (in practice the earliest-registered one, a tiny labeler — that is why `E2E Tests` once rendered under "PR size label"), and an app-token check-run would need the COZYSTACK_CI app granted `checks: write`. A commit status has neither problem: branch protection matches it by context name and it carries no check-suite grouping. It is safe to post with the default token because a fork's `pull_request` token is **read-only** and cannot post a status — so only same-repo runs (`plan`'s opener and `e2e-report`, both guarded `!fork`) and the privileged `workflow_run` in `e2e-fork.yaml` can set it. Each publishing job carries `permissions: statuses: write` and calls `repos.createCommitStatus`. `plan`'s opener is additionally `continue-on-error`, since every build and test job hangs off `plan` and a transient status-API failure there must not discard the run; losing it only restores the stale-green window for that one run.
120+
**Consequence for branch protection:** require the `E2E Tests` **status context** (no GitHub App and no app-id pin needed).
121+
122+
- **Fork TIA reads trusted inputs, but the fork still writes the selector.** `e2e-fork.yaml`'s `e2e` job runs the same `select-e2e.sh` TIA as in-tree (§7), and its inputs are hardened: the changed-file list and the `full-e2e` label come from `resolve`'s base-repo API lookups (`pulls.listFiles` / the PR's labels), the same trusted source as the docs-only decision, passed in as outputs (`changed_b64`, `full_e2e`) rather than re-derived from a `git diff` of the fork-controlled merge ref. Keep that part as it is: re-deriving the selection from the checked-out tree would add a second bypass for no gain.
123+
- **What the fork lane is, and is not.** The `e2e` job runs from the fork's tree, so `select-e2e.sh`, the suites under `hack/e2e-chainsaw/`, `hack/e2e-install-cozystack.bats` and `packages/core/testing`'s recipes are all fork-authored. A fork that wants its own suite to report green can edit any of them, and hardening the TIA inputs does not change that. Read the lane accordingly: it is a correctness gate that stops an honest contributor shipping a regression, not a boundary against a hostile one. What contains a hostile fork is that no fork run executes without maintainer approval (below), that the job holds no push credential, and that a diff editing the test harness is visible in review.
124+
125+
- **A fork PR that adds a new image cannot go green until that image exists on the base branch.** `publish` derives the set of archive names it will accept from the base parent (`HEAD^1`), so that a fork cannot mint arbitrary repositories under the CI registry by naming an artifact. An image the PR itself introduces is therefore not in that set, and the job fails with `image '<name>' is not in the base-tree allowlist`. This is fail-closed by design and there is no label override — land the image on the base branch first, or run the suite from a same-repo branch. If a fork e2e run fails on that line, the contribution is not broken; the allowlist is doing its job.
126+
127+
- **`labeled` events run in their own concurrency group, and a cancelled run reports nothing.** Because the gate is a status rather than a job, a run that dies without posting leaves the head SHA at "Expected" — which blocks the merge rather than passing it. Several separate pieces implement it, and weakening any one of them reopens the gap. Each lane's terminal reporter runs under `!cancelled()` rather than `always()`, so a run that was cancelled publishes nothing at all and the `pending` it opened stands until the run that superseded it reports. Each lane's concurrency key then keeps a run that is going to publish nothing out of the group of the run that is: `pull-requests.yaml`'s appends `-label` for a `labeled` event that `plan` discards, so labelling an open PR cannot cancel a live `opened`/`synchronize` run (the replacement run would never post the status the cancelled run was going to) — the condition is the complement of `plan`'s guard, so a `full-e2e` label, which does publish, stays in the main group and supersedes the head run instead of racing it, `e2e-fork.yaml`'s concurrency key appends the triggering run's id when that run concluded `cancelled` or `skipped` so a privileged run that is going to post nothing cannot cancel the one that is going to post, and `e2e-fork.yaml`'s `resolve` returns early on a `cancelled`/`skipped` triggering run instead of posting `failure` for it. If you touch any of them, keep the invariant: exactly one run per head SHA is responsible for the status, and a run that is not that one must post nothing at all.
128+
129+
**Security prerequisite (repo setting).** The split is only safe if fork runs require maintainer approval before they execute. The repository must keep **Settings → Actions → "Require approval for all outside collaborators"** enabled, so a fork's `pull_request` run — which produces the artifacts `e2e-fork.yaml` consumes — cannot run untrusted code (or feed the privileged run) without a maintainer's go-ahead.
130+
107131
## Chainsaw v0.2.15 gotchas
108132

109133
The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026); none of the traps below are fixed upstream yet, so the workarounds stay until a bump is possible. Each one has already cost a debugging session — this is the "no one gets it right the first time" list, worth a read before writing a new assert.

‎hack/common-envs.mk‎

Lines changed: 46 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,23 @@ WRITE_CACHE ?= 0
3636

3737
PUSH := 1
3838
LOAD := 0
39+
40+
# OCI_EXPORT_DIR: when set, build every image to a per-image OCI archive
41+
# (<dir>/<name>.oci.tar) instead of pushing or loading it. Fork PRs use this:
42+
# the unprivileged `pull_request` build carries no registry credentials, so it
43+
# exports the images as workflow artifacts and a privileged `workflow_run` later
44+
# pushes them to the registry BY DIGEST (see .github/workflows/e2e-fork.yaml).
45+
# The image digest is content-addressed and captured via --metadata-file
46+
# regardless of output type, so the values.yaml/.tag refs baked during this
47+
# build match exactly what the privileged run pushes. Setting it forces
48+
# PUSH/LOAD off so buildx emits only the archive — a `--push` alongside the OCI
49+
# --output would attempt an anonymous push and die with `denied` (#3257).
50+
OCI_EXPORT_DIR ?=
51+
ifneq ($(strip $(OCI_EXPORT_DIR)),)
52+
PUSH := 0
53+
LOAD := 0
54+
endif
55+
3956
BUILDER ?=
4057
PLATFORM ?=
4158
BUILDX_EXTRA_ARGS ?=
@@ -65,8 +82,34 @@ BUILDX_ARGS := --provenance=false --push=$(PUSH) --load=$(LOAD) \
6582
# Consumers reference images by digest, so the build-unique tag is enough
6683
# for cluster-side pulls; the versioned and floating tags exist for human
6784
# discoverability and downstream tooling on releases.
85+
# When OCI_EXPORT_DIR is set it also appends a per-image `--output type=oci`
86+
# (via the oci-output macro) so the build writes <dir>/<name>.oci.tar (the tag
87+
# becomes the archive's image name). Every package's buildx call routes its tags
88+
# through this macro, so this is the single point that turns the whole build into
89+
# an artifact-export build — except recipes that bypass image-tags, which must
90+
# call oci-output directly (see #3257).
91+
92+
# comma emits a literal comma inside the lazily-expanded macros below
93+
# (oci-output, cache-args) without make mis-reading it as an $(if …)/$(call …)
94+
# argument separator. Defined here, above its first user, so it is unambiguously
95+
# in scope wherever those macros are $(call)ed from a recipe.
96+
comma := ,
97+
98+
# oci-output <image-name> — when OCI_EXPORT_DIR is set, expands to a per-image
99+
# `--output type=oci,dest=<dir>/<name>.oci.tar`; empty otherwise. EVERY buildx
100+
# call reachable from a fork build must include this — via image-tags, or
101+
# directly for recipes that bypass image-tags (#3257). A missing archive is
102+
# caught loudly by the fork build's `if-no-files-found: error` upload.
103+
oci-output = $(if $(strip $(OCI_EXPORT_DIR)), --output type=oci$(comma)dest=$(OCI_EXPORT_DIR)/$(subst /,-,$(1)).oci.tar)
104+
105+
# image-tags <image-name> [<versioned-tag>] — the --tag/--output flags for one
106+
# image. Under OCI_EXPORT_DIR the versioned/floating tags are suppressed: a
107+
# multi-`--tag` `--output type=oci` build writes every ref into the archive's
108+
# index.json, and `skopeo copy oci-archive:<f>` then refuses it ("more than one
109+
# image in oci"). PUBLISH_* are 0 on fork PRs (the only export case today), so
110+
# this is belt-and-suspenders for that trap.
68111
define image-tags
69-
--tag $(REGISTRY)/$(1):$(IMAGE_TAG)$(if $(filter 1,$(PUBLISH_VERSIONED)),$(if $(filter-out $(IMAGE_TAG),$(strip $(2))), --tag $(REGISTRY)/$(1):$(strip $(2))))$(if $(filter 1,$(PUBLISH_FLOATING)), --tag $(REGISTRY)/$(1):latest)
112+
--tag $(REGISTRY)/$(1):$(IMAGE_TAG)$(if $(strip $(OCI_EXPORT_DIR)),,$(if $(filter 1,$(PUBLISH_VERSIONED)),$(if $(filter-out $(IMAGE_TAG),$(strip $(2))), --tag $(REGISTRY)/$(1):$(strip $(2))))$(if $(filter 1,$(PUBLISH_FLOATING)), --tag $(REGISTRY)/$(1):latest))$(call oci-output,$(1))
70113
endef
71114

72115
# cache-args <image-name> [<cache-tag>]
@@ -78,9 +121,8 @@ endef
78121
# keep the cache manifest portable across registries (OCIR/ghcr/ECR).
79122
# <cache-tag> defaults to `buildcache`; pass an explicit tag for images that build
80123
# a distinct artifact per loop iteration (e.g. ubuntu-container-disk per k8s ver).
81-
# $(comma) escapes the literal commas in the --cache-to value so make does not
82-
# mis-parse them as $(if ...) argument separators.
83-
comma := ,
124+
# $(comma) (defined above) escapes the literal commas in the --cache-to value so
125+
# make does not mis-parse them as $(if ...) argument separators.
84126
cache-args = --cache-from type=registry,ref=$(CACHE_REGISTRY)/$(1):$(if $(2),$(2),buildcache)$(if $(filter 1,$(WRITE_CACHE)), --cache-to type=registry$(comma)ref=$(CACHE_REGISTRY)/$(1):$(if $(2),$(2),buildcache)$(comma)mode=max$(comma)oci-mediatypes=true$(comma)image-manifest=true)
85127

86128
ifeq ($(COZYSTACK_VERSION),)

0 commit comments

Comments
 (0)