Skip to content

Commit 4ad2cbe

Browse files
committed
docs(agents): clarify E2E execution targets
Fork E2E workflow edits are exercised only after they reach the default branch. Document that boundary so a rerun is not mistaken for testing an unmerged workflow change. Document the shared install-readiness gate rather than a weaker single wait. Bind manual calls to the intended context and require an explicit timeout without changing the helper's convergence checks. Validation: markdownlint; 8 helper unit tests and 6 isolated command cases. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 0b720bb commit 4ad2cbe

1 file changed

Lines changed: 15 additions & 2 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,8 +81,20 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
8181

8282
The install test fails if **any** HelmRelease is not Ready. A toothless gate (backgrounded `kubectl wait` discarding child exit codes + a bare `echo` instead of `exit 1`) once shipped a permanently-failing platform HR through green CI for weeks.
8383

84-
- Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR.
85-
- Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`.
84+
Use `cozy_wait_all_helmreleases_ready` from `hack/e2e-wait-helmreleases.sh`, as `hack/e2e-install-cozystack.bats` does. Keep the helper's minimum release count, observed-generation checks, stable all-Ready set and fail-fast behavior on `Stalled=True`. A single wait and re-list does not preserve those checks.
85+
86+
For a manual readiness check from the repository root, set `KUBE_CONTEXT` to the human-approved sandbox context and `HR_READY_TIMEOUT` to a positive integer number of seconds, using [Sizing an HR-Ready budget](#sizing-an-hr-ready-budget). The subshell binds the helper's kubectl calls to that context without changing the caller's shell or the global current context:
87+
88+
```bash
89+
(
90+
: "${KUBE_CONTEXT:?}" "${HR_READY_TIMEOUT:?}"
91+
kubectl() { command kubectl --context "$KUBE_CONTEXT" "$@"; }
92+
. hack/e2e-wait-helmreleases.sh
93+
cozy_wait_all_helmreleases_ready "$HR_READY_TIMEOUT"
94+
)
95+
```
96+
97+
Keep readiness failures visible; do not mask the helper's exit status. Its diagnostics report the releases that are not Ready.
8698

8799
### 6. Assert no HelmRelease failed an action it retries in place — via `status.history`
88100

@@ -160,6 +172,7 @@ A fork's `pull_request` run carries no registry credentials, so it cannot push t
160172

161173
Consequences to keep in mind when touching either workflow:
162174

175+
- **A `workflow_run` workflow always runs the default branch's copy of itself.** An edit to `e2e-fork.yaml` therefore has no effect on any fork PR, the one carrying the edit included, until it is merged to `main`. To see it work, merge first and then push to a fork PR: re-running an existing `e2e-fork` run replays the commit and workflow file it was first triggered with.
163176
- **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`.
164177
- **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `full-e2e` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:
165178
- *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate.

0 commit comments

Comments
 (0)