Skip to content

Commit ad62a8d

Browse files
committed
test(e2e): stop blocking merges on the tenant node-join deadline
The tenant node-join wait no longer discriminates. Nested virtualisation on the shared runners degrades far enough that a guest which registers in minutes on real hardware can miss any deadline this test can afford, so a red on that one wait says as much about the machine the run landed on as about the product, and it teaches a reader to re-run the job. The suite still fails on it. That is what keeps Chainsaw's catch running: the previous-instance container logs, the host snapshot, the data-plane capture and the event dump are all keyed on the test failing, and the report stays honest. What changes is one layer up. The wait writes a marker beside its diagnostics, and a new lane-side gate reads that marker to decide whether the job blocks. It opens only when every failed test in the run carries one, and blocks on anything it cannot attribute: a report it could not read, a failed test the report does not name, a failure with no marker. Only the lanes that block a merge ask it. The two pull-request lanes do; nightly and the tag lane run the same suites and stay hard. A soft pass on the tag lane would ship a release candidate whose tenant clusters brought up no worker nodes, and nightly is the only place an e2e failure reaches anyone who did not open a pull request, since no lane in this tree notifies anywhere. A softened red finishes the job green, so the job's result stops answering whether the suite passed, and both kinds of consumer are handed the decision instead of inferring it. A step gated on the job failing would silently stop running: the SSH breakpoint is that case, and it still opens for a pull request marked for debugging. A job that reports the result would silently start reporting a pass: the required commit status now says the deadline was missed and that the merge is not blocked on it, rather than that the suite passed. The deadline goes to 29m. That is the test's own budget and nothing more: its clock starts after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, while the chart's autoscaler and health-check timers have been running since the Machine was created, so no ordering against them is claimed. The chart's 20m and 30m give a scale to pick against, and a budget over the first leaves room for a remediation cycle without promising that one fits. The operation ceiling both suites give the script rises with the deadline, and the diagnostics phase budget rises inside it, because the collectors gated ahead of the serial console cost more between them than the old budget covered. Both budget guards read their terms from the sources that set them. The e2e job cap rises with them, and its guard holds what a guard can hold: the pair's operation and catch ceilings, and one figure across every lane that runs the suite. It also records that the suites' full summed ceilings have never fit any cap in that file, so the margin above the bound is taken from a measured run instead. The prose that restates the cap elsewhere moves with it, and the sweep that holds the deadline's restatements now reads the figure spelled in words as well as in minutes. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 9c7bfb8 commit ad62a8d

22 files changed

Lines changed: 1562 additions & 195 deletions

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -540,8 +540,18 @@ jobs:
540540
name: E2E (fork run)
541541
needs: [resolve, publish]
542542
if: ${{ needs.resolve.outputs.should_run == 'true' }}
543+
# What the gate decided, promoted to a job output because the job's RESULT
544+
# can no longer answer it: a red the lane tolerated finishes `success`, so
545+
# the status this lane reports would otherwise call that run a pass.
546+
outputs:
547+
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
543548
runs-on: oracle-vm-32cpu-128gb-x86-64
544-
timeout-minutes: 180
549+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
550+
# held to one figure across every lane that runs the suite. That bound, the
551+
# measured run the rest of the margin comes from, and what neither of them
552+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
553+
# here in four workflows that would then drift apart.
554+
timeout-minutes: 215
545555
permissions:
546556
contents: read
547557
packages: read
@@ -689,6 +699,19 @@ jobs:
689699
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
690700
' || true
691701
echo "::endgroup::"
702+
# One failure the lane does not block on, and only one: the tenant
703+
# node-join deadline. The test itself stays red -- Chainsaw ran its
704+
# whole catch above and the report records the failure -- and this
705+
# decides only whether the job does. It blocks on anything it cannot
706+
# attribute to that deadline, including a report it could not read.
707+
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
708+
# Recorded because this exit makes the job succeed, and the status
709+
# this lane reports back to the pull request is derived from that
710+
# result: without this the required check would say the suite
711+
# passed on a run where it failed.
712+
echo "soft_red=true" >>"$GITHUB_OUTPUT"
713+
exit 0
714+
fi
692715
exit 1
693716
fi
694717
echo "✅ All selected E2E tests passed"
@@ -746,14 +769,22 @@ jobs:
746769
env:
747770
PUBLISH_RESULT: ${{ needs.publish.result }}
748771
E2E_RESULT: ${{ needs.e2e.result }}
772+
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
749773
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
750774
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
751775
with:
752776
script: |
753-
const { PUBLISH_RESULT, E2E_RESULT, HEAD_SHA, RUN_URL } = process.env;
777+
const { PUBLISH_RESULT, E2E_RESULT, E2E_SOFT_RED, HEAD_SHA, RUN_URL } = process.env;
754778
const { owner, repo } = context.repo;
755779
let state, description;
756-
if (E2E_RESULT === 'success') {
780+
if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
781+
// The suite did NOT pass: the tenant node-join deadline expired and
782+
// the lane decided not to block on it. `success` is the decision;
783+
// this text is the only place a reader standing in front of the
784+
// merge button learns what it was a decision about.
785+
state = 'success';
786+
description = 'fork e2e soft: node-join deadline missed, suite failed, merge not blocked';
787+
} else if (E2E_RESULT === 'success') {
757788
state = 'success';
758789
description = 'fork e2e passed (privileged run)';
759790
} else if (PUBLISH_RESULT !== 'success') {

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,12 @@ jobs:
134134
# enough host headroom for QEMU, networking, containerd, and the runner
135135
# agent while the full platform installs across all three guests.
136136
runs-on: oracle-vm-32cpu-128gb-x86-64
137-
timeout-minutes: 180
137+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
138+
# held to one figure across every lane that runs the suite. That bound, the
139+
# measured run the rest of the margin comes from, and what neither of them
140+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
141+
# here in four workflows that would then drift apart.
142+
timeout-minutes: 215
138143
permissions:
139144
contents: read
140145
checks: write # Breakpoint action updates its dedicated "Breakpoint Open" Check Run.
@@ -258,7 +263,7 @@ jobs:
258263
# collect-report packages cozyreport plus per-failed-test crust-gather
259264
# snapshots captured before Chainsaw cleanup.
260265
#
261-
# A ceiling of its own, not the job's 180 minutes. `Upload cozyreport.tgz`
266+
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
262267
# is the next step and the collector writes its tarball at the very end of
263268
# its run, so a read that never returns spends what is left of the job
264269
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/nightly.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,12 @@ jobs:
227227
# containerd and the runner agent. Enforced by
228228
# hack/sandbox-runner-headroom.bats.
229229
runs-on: oracle-vm-32cpu-128gb-x86-64
230-
timeout-minutes: 180
230+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
231+
# held to one figure across every lane that runs the suite. That bound, the
232+
# measured run the rest of the margin comes from, and what neither of them
233+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
234+
# here in four workflows that would then drift apart.
235+
timeout-minutes: 215
231236
permissions:
232237
contents: read
233238
packages: read
@@ -339,7 +344,7 @@ jobs:
339344
path: _out/chainsaw-report.xml
340345
if-no-files-found: ignore
341346

342-
# A ceiling of its own, not the job's 180 minutes. `Upload cozyreport.tgz`
347+
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
343348
# is the next step and the collector writes its tarball at the very end of
344349
# its run, so a read that never returns spends what is left of the job
345350
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/pull-requests.yaml‎

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,12 @@ jobs:
819819
# workflow_run instead.
820820
e2e:
821821
name: "E2E (in-tree)"
822+
# What the gate decided, promoted to a job output because the job's RESULT
823+
# can no longer answer it: a red the lane tolerated finishes `success`, so
824+
# every consumer downstream that reads the result alone would report this run
825+
# as one where the suite passed.
826+
outputs:
827+
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
822828
# The sandbox boots three QEMU guests (hack/e2e-prepare-cluster.bats), each
823829
# with `-smp 8 -m 24576`, so the guests alone demand 24 vCPU and 72 GiB.
824830
# On the 24-vCPU runner that is a 1:1 CPU commit with zero host headroom for
@@ -835,7 +841,12 @@ jobs:
835841
# leaving 8 vCPU (25%) and 56 GiB of host headroom, which removes the
836842
# oversubscription that starves the probes.
837843
runs-on: oracle-vm-32cpu-128gb-x86-64
838-
timeout-minutes: 180
844+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
845+
# held to one figure across every lane that runs the suite. That bound, the
846+
# measured run the rest of the margin comes from, and what neither of them
847+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
848+
# here in four workflows that would then drift apart.
849+
timeout-minutes: 215
839850
permissions:
840851
contents: read
841852
packages: read
@@ -1035,6 +1046,20 @@ jobs:
10351046
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
10361047
' || true
10371048
echo "::endgroup::"
1049+
# One failure the lane does not block on, and only one: the tenant
1050+
# node-join deadline. The test itself stays red -- Chainsaw ran its
1051+
# whole catch above and the report records the failure -- and this
1052+
# decides only whether the job does. It blocks on anything it cannot
1053+
# attribute to that deadline, including a report it could not read.
1054+
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
1055+
# The breakpoint below opens on `failure()`, which this exit is
1056+
# about to make false. Record the decision so it can open on this
1057+
# red too: the sandbox is still up at that point, and a join that
1058+
# never finished is the failure an interactive session is worth
1059+
# most on.
1060+
echo "soft_red=true" >>"$GITHUB_OUTPUT"
1061+
exit 0
1062+
fi
10381063
exit 1
10391064
fi
10401065
echo "✅ All selected E2E tests passed"
@@ -1056,7 +1081,7 @@ jobs:
10561081

10571082
# ▸ Collect debug information (always runs)
10581083
#
1059-
# A ceiling of its own, not the job's 180 minutes. `Upload
1084+
# A ceiling of its own, not the job's 215 minutes. `Upload
10601085
# cozyreport.tgz` is the next step and the collector writes its tarball
10611086
# at the very end of its run, so a read that never returns spends what
10621087
# is left of the job budget, the job is cancelled, and the upload never
@@ -1067,7 +1092,7 @@ jobs:
10671092
# 30 minutes is measured against the step: it takes 3-42s across sampled
10681093
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
10691094
# `timeout-minutes` caps a step, it does not carve budget out of the job,
1070-
# so a suite that has already burned 175 of the 180 leaves this step five
1095+
# so a suite that has already burned 210 of the 215 leaves this step five
10711096
# minutes and the upload is cancelled with everything else. What the cap
10721097
# buys is that a hung read cannot be the reason.
10731098
# `continue-on-error` because the ceiling must not change what the job
@@ -1126,6 +1151,13 @@ jobs:
11261151
# implies maintainer authorship). authorized-users is kept as a second
11271152
# defense layer controlling who may actually SSH in.
11281153
#
1154+
# A node-join red the lane decided not to block on reaches here as a
1155+
# SUCCESSFUL job, so `failure()` alone would skip the step on the one
1156+
# failure whose sandbox is worth attaching to. The E2E step records that
1157+
# decision and this reads it back. Being label-gated is what makes that
1158+
# affordable: the session opens for a PR someone deliberately marked for
1159+
# debugging, not on every softened run.
1160+
#
11291161
# Uses cozystack/breakpoint-action (fork of namespacelabs/breakpoint-action)
11301162
# pinned by SHA. The fork adds: pause-idle mode (initial grace period for
11311163
# the first SSH connection, idle-aware exit afterwards), endpoint output
@@ -1139,10 +1171,15 @@ jobs:
11391171
# forks cannot reach the self-hosted rendezvous server.
11401172
- name: Breakpoint on E2E failure
11411173
if: |
1142-
failure() &&
1174+
(failure() || steps.e2e_tests.outputs.soft_red == 'true') &&
11431175
vars.BREAKPOINT_ENDPOINT != '' &&
11441176
(contains(github.event.pull_request.labels.*.name, 'debug')
11451177
|| contains(github.event.pull_request.labels.*.name, 'release'))
1178+
# A step that fails here fails an otherwise-successful job, which on the
1179+
# softened path would let a debugging tool overturn a lane that had
1180+
# decided not to block. Same reason `Collect report` above carries it:
1181+
# what the job REPORTS must not depend on what a helper did.
1182+
continue-on-error: true
11461183
# cozystack/breakpoint-action v2-cozy.1
11471184
# mode: pause-idle defaults: grace-period=20m, idle-timeout=10m
11481185
uses: cozystack/breakpoint-action@a6f3a6f87be398ad63b6577351e3398e53f578e4
@@ -1235,14 +1272,15 @@ jobs:
12351272
PLAN_RESULT: ${{ needs.plan.result }}
12361273
CODE: ${{ needs.plan.outputs.code }}
12371274
E2E_RESULT: ${{ needs.e2e.result }}
1275+
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
12381276
CANDIDATE_RESULT: ${{ needs.verify-release-candidate.result }}
12391277
IS_RELEASE: ${{ contains(github.event.pull_request.labels.*.name, 'release') }}
12401278
IS_FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
12411279
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
12421280
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
12431281
with:
12441282
script: |
1245-
const { PLAN_RESULT, CODE, E2E_RESULT, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env;
1283+
const { PLAN_RESULT, CODE, E2E_RESULT, E2E_SOFT_RED, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env;
12461284
let state, description;
12471285
if (PLAN_RESULT !== 'success') {
12481286
state = 'failure';
@@ -1263,6 +1301,13 @@ jobs:
12631301
// full suite re-run or an admin override.
12641302
state = 'success';
12651303
description = 'release promotion — verified at rc, e2e not repeated (add full-e2e to force a run)';
1304+
} else if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
1305+
// The suite did NOT pass: the tenant node-join deadline expired and
1306+
// the lane decided not to block on it. `success` is the decision;
1307+
// this text is the only place a reader standing in front of the
1308+
// merge button learns what it was a decision about.
1309+
state = 'success';
1310+
description = 'in-tree e2e soft: node-join deadline missed, suite failed, merge not blocked';
12661311
} else if (E2E_RESULT === 'success') {
12671312
state = 'success';
12681313
description = 'in-tree e2e passed';

0 commit comments

Comments
 (0)