Skip to content

Commit 62b0590

Browse files
committed
docs(agents): clarify E2E execution targets
Fork E2E workflow edits are exercised only after they reach the default branch. Document that boundary so a rerun is not mistaken for testing an unmerged workflow change. Manual checks need the intended context and supplied kubeconfig, or matching context names can select different clusters. Reuse the shared readiness gate and fail the standalone subshell on parser errors. Describe the polling and diagnostic limits without changing the helper. Validation: markdownlint; 8 helper unit tests and 10 isolated command cases pass, including target selection and parser failure paths. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent cf0d01a commit 62b0590

1 file changed

Lines changed: 22 additions & 2 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,8 +81,27 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
8181

8282
The install test fails if **any** HelmRelease is not Ready. A toothless gate (backgrounded `kubectl wait` discarding child exit codes + a bare `echo` instead of `exit 1`) once shipped a permanently-failing platform HR through green CI for weeks.
8383

84-
- Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR.
85-
- Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`.
84+
Use `cozy_wait_all_helmreleases_ready` from `hack/e2e-wait-helmreleases.sh`, as `hack/e2e-install-cozystack.bats` does. Keep the helper's minimum release count, observed-generation checks, stable all-Ready set and fail-fast behavior on `Stalled=True`. A single wait and re-list does not preserve those checks.
85+
86+
For a manual readiness check from the repository root, set `KUBE_CONTEXT` to the human-approved sandbox context and `HR_READY_TIMEOUT` to a positive base-10 integer number of seconds without leading zeroes, using [Sizing an HR-Ready budget](#sizing-an-hr-ready-budget). If the task supplies a kubeconfig file, set `KUBE_CONFIG` to its path; otherwise leave it unset to use kubectl's normal configuration loading. The subshell binds the helper's kubectl calls to that target without changing the caller's shell or the global current context:
87+
88+
```bash
89+
(
90+
set -e
91+
: "${KUBE_CONTEXT:?}" "${HR_READY_TIMEOUT:?}"
92+
kubeconfig_args=()
93+
if [ "${KUBE_CONFIG+x}" ]; then
94+
kubeconfig_args=(--kubeconfig "${KUBE_CONFIG:?}")
95+
fi
96+
kubectl() { command kubectl "${kubeconfig_args[@]}" --context "$KUBE_CONTEXT" "$@"; }
97+
. hack/e2e-wait-helmreleases.sh
98+
cozy_wait_all_helmreleases_ready "$HR_READY_TIMEOUT"
99+
)
100+
```
101+
102+
Run the subshell as a standalone command: using it as an `if` condition or on the left of `&&` or `||` suppresses errexit inside it. Keep readiness failures visible; do not mask the helper's exit status. Timeout diagnostics describe unready releases from the last successful poll and may be stale after a failed read. The final diagnostic list can also fail.
103+
104+
`HR_READY_TIMEOUT` is a polling budget, not a strict wall-clock limit. A pending command or the final diagnostic read can outlive it, and the current helper can accept a Ready result returned after the budget expires.
86105

87106
### 6. Assert no HelmRelease failed an action it retries in place — via `status.history`
88107

@@ -160,6 +179,7 @@ A fork's `pull_request` run carries no registry credentials, so it cannot push t
160179

161180
Consequences to keep in mind when touching either workflow:
162181

182+
- **A `workflow_run` workflow always runs the default branch's copy of itself.** An edit to `e2e-fork.yaml` therefore has no effect on any fork PR, the one carrying the edit included, until it is merged to `main`. To see it work, merge first and then push to a fork PR: re-running an existing `e2e-fork` run replays the commit and workflow file it was first triggered with.
163183
- **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`.
164184
- **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `full-e2e` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:
165185
- *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate.

0 commit comments

Comments
 (0)