Skip to content

Commit 3c780ac

Browse files
authored
test(e2e): stop blocking merges on the tenant node-join deadline (#3932)
## What this PR does The tenant node-join wait in the two kubernetes e2e suites has stopped discriminating. Nested virtualisation on the shared runners degrades far enough that a worker which registers in minutes elsewhere can miss any deadline this test can afford, so a red on that one wait says as much about the machine the run landed on as about the product. What it produces in practice is a re-run, and a signal that teaches people to re-run is a signal that gets the meaningful reds re-run too. The suite still fails on it. That is the part worth reading twice, because it is what separates this from hiding a flake: the script exits non-zero, Chainsaw records a failure, and its whole catch runs (previous-instance container logs, the host snapshot, the data-plane capture, the event dump), every one of which is keyed on the test failing. The JUnit report says a test failed. What changes is one layer up. The wait writes a `SOFT-RED-node-join.txt` marker beside its diagnostics, and a new lane-side gate, `hack/e2e-node-join-soft-red.sh`, reads that marker to decide whether the job blocks. Softening inside the suite would have bought a green check by throwing all of that evidence away. The gate opens on one shape only, every failed test in the run carrying that marker, and blocks on everything else, including every way it can fail to answer the question: a report it cannot read, a failed test the report does not name, a failure with no marker, a run whose report records no failure at all. A marker is written only when `timeout` reports 124 on that one wait, so an assertion after the join, or the wait failing to run, stays an ordinary red. What the marker attributes is the test rather than the individual failure: Chainsaw writes one testcase per Test and combines that Test's step errors into its single failure element, so a second step error rides soft alongside the deadline, while a failed `catch`, `finally` or step-level `cleanup` rides soft for a different reason worth keeping apart, since at the pinned version those run with no step report and never reach the failure element at all. The convention doc states both rather than working around the granularity, since the suite stops at the wait and nothing it exists to prove runs after the marker. Only the lanes that block a merge ask the gate. `pull-requests` and `e2e-fork` do; `nightly` and `e2e-tag` run the same suites and stay hard. `e2e-tag` is the release candidate's own e2e check, where a soft pass would ship a release whose tenant clusters brought up no worker nodes. `nightly` is the only place an e2e failure reaches anyone who did not open a pull request, since nothing in this tree sends a notification anywhere, so the red job is the notification. A unit guard holds that split in both directions and derives the lane set from the workflows that run the suite, so a new lane has to be placed on one side or the other before it passes. One consequence of a softened red is worth naming rather than discovering later: the job finishes green, so its result stops answering whether the suite passed, and it has two kinds of consumer that each break their own way. A step gated on the job failing silently stops running. The SSH breakpoint in `pull-requests` is that case, and it is the only interactive way into a sandbox whose node-join never finished, so the E2E step records what the gate decided and the breakpoint's condition reads it back; the step is label-gated, so the session opens for a PR someone deliberately marked for debugging rather than on every softened run. A job that reports the result silently starts reporting a pass. That is the required `E2E Tests` commit status in both pull-request lanes, and on a softened run it now says the node-join deadline was missed, that the suite failed, and that the merge is not blocked on it, instead of saying the suite passed. The handover is four links long, and a guard reads each of them inside the block it has to live in rather than anywhere in the lane: the step carrying the gate's id writes the decision to the output sink, the job that runs the suite publishes it naming that step, the job that reports the result reads it off that job, and it branches on the value ahead of the branch that would otherwise call the run a pass. A link matched file-wide passes while it sits on a neighbour, and the value is then empty for good with nothing to say so. The first link is read tighter still, and in both directions: the write has to be an unconditional statement in the same branch as the exit that makes the job green, that branch has to fall through to a written non-zero exit, no other green exit may leave the step, the step may not be marked unable to fail, and the write may not appear anywhere else. A write under a condition of its own records the decision on some runs and not others; an exit that escapes the branch turns a red the gate REFUSED to soften into a green job with nothing recorded; and a write on the passing path would report a suite that passed as one that missed the deadline. What the guard cannot do is stated with it: it cannot tell one reporter from two, it reads the branch as text rather than evaluating it, and every block it reads is delimited by indentation. What is given up is real. A regression whose only symptom is that workers never reach Ready no longer blocks a merge, and these two suites are the only ones in the tree that boot tenant worker nodes. The mechanism sees a timer, not a cause, so a deterministic node-join regression rides soft as readily as a slow runner does, and the failed job stops being the thing that opens a debugging path on its own. That is a debt rather than a design, and the way out is written down beside the convention it bends: calibrate the runner canary against real lane data, then move the tolerance under the canary's alert so the lane softens a run measured as degraded and blocks one measured as healthy. Until that calibration exists the canary cannot carry the decision, which is why it does not carry it today. The deadline goes from 18m to 29m, and the reason is bounded on purpose. 29m is this test's own budget and nothing else: its clock starts where the wait starts, after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, each with a ceiling of its own, while the chart's timers have been running since the Machine was created. The autoscaler's `maxNodeProvisionTime` measures from that creation and the MachineHealthCheck's `nodeStartupTimeout` from the later of creation and `InfrastructureReady`, restarting at that transition, so the offset is not fixed and no ordering against either timer is claimed. What the chart's 20m and 30m give is a scale to pick against: a budget over the first leaves room for a remediation cycle to fit inside the window, which is a possibility rather than a guarantee, since a run can spend the whole window on one worker that never comes back. The operation ceiling both suites give the script rises with the deadline (50m to 67m), and the diagnostics phase budget rises inside it (7m to 8m), because the collectors gated ahead of the serial console cost more between them than the old budget covered. Both budget guards read their terms from the sources that set them rather than restating them. The e2e job cap rises from 180 to 215 minutes in all four lanes, and its guard is honest about what it can hold: it derives the pair's operation and catch ceilings from source and requires one figure across every lane that runs the suite, and it records that the suites' full summed ceilings (226 minutes counting the `finally` legs and the storageclass-fallback Test) have never fit any cap in that file, including at 180. The margin above the derivable bound is empirical instead: three nightlies that failed both bringup Tests ran their full-suite e2e job in 149, 158 and 160 minutes under the previous ceilings, a run of that shape gains at most 12 minutes per suite here, and 215 clears the resulting projection by about 30 minutes where 180 cleared its measurement by 20. The prose that restates the cap outside the workflows moves with it, in the Chainsaw config and in the two report collectors that derive their own headroom from it. Test coverage: a new unit suite for the gate pins the direction it fails in (an unreadable report, a failed testcase the report leaves unnamed or names with the empty string or with blanks only, a failure without a marker, and a report recording no failure all block), the lane split in both directions, the four-link handover of the softening decision, and the job cap. The fixtures put the unnameable case ahead of a named suite on purpose, because an awk record runs to the next testcase and a case placed last cannot exercise the leak the tally exists to catch. The existing node-join suite gains guards that the marker is written only under the status meaning the deadline expired, that it is written ahead of the diagnostics phase rather than behind it, and that the deadline quoted in prose across the tree is the one the wait gives, in words as well as in minutes. Every one of those guards was checked by mutating the thing it protects and confirming it, and only it, goes red. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): the tenant node-join deadline still fails its suite and still collects its diagnostics, but no longer blocks a merge in the pull-request lanes; nightly and the release-tag lane stay hard ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - E2E checks can now report a successful, non-blocking status when the only failure is the recognized tenant node-join deadline. - Failure diagnostics and suite results remain available for these tolerated failures. - **Bug Fixes** - Improved detection and attribution of node-join deadline failures, avoiding false-positive soft passes. - **Documentation** - Updated E2E timing, diagnostic budgets, and failure-handling guidance to reflect the longer test windows. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 9c7bfb8 + ad62a8d commit 3c780ac

22 files changed

Lines changed: 1562 additions & 195 deletions

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -540,8 +540,18 @@ jobs:
540540
name: E2E (fork run)
541541
needs: [resolve, publish]
542542
if: ${{ needs.resolve.outputs.should_run == 'true' }}
543+
# What the gate decided, promoted to a job output because the job's RESULT
544+
# can no longer answer it: a red the lane tolerated finishes `success`, so
545+
# the status this lane reports would otherwise call that run a pass.
546+
outputs:
547+
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
543548
runs-on: oracle-vm-32cpu-128gb-x86-64
544-
timeout-minutes: 180
549+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
550+
# held to one figure across every lane that runs the suite. That bound, the
551+
# measured run the rest of the margin comes from, and what neither of them
552+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
553+
# here in four workflows that would then drift apart.
554+
timeout-minutes: 215
545555
permissions:
546556
contents: read
547557
packages: read
@@ -689,6 +699,19 @@ jobs:
689699
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
690700
' || true
691701
echo "::endgroup::"
702+
# One failure the lane does not block on, and only one: the tenant
703+
# node-join deadline. The test itself stays red -- Chainsaw ran its
704+
# whole catch above and the report records the failure -- and this
705+
# decides only whether the job does. It blocks on anything it cannot
706+
# attribute to that deadline, including a report it could not read.
707+
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
708+
# Recorded because this exit makes the job succeed, and the status
709+
# this lane reports back to the pull request is derived from that
710+
# result: without this the required check would say the suite
711+
# passed on a run where it failed.
712+
echo "soft_red=true" >>"$GITHUB_OUTPUT"
713+
exit 0
714+
fi
692715
exit 1
693716
fi
694717
echo "✅ All selected E2E tests passed"
@@ -746,14 +769,22 @@ jobs:
746769
env:
747770
PUBLISH_RESULT: ${{ needs.publish.result }}
748771
E2E_RESULT: ${{ needs.e2e.result }}
772+
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
749773
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
750774
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
751775
with:
752776
script: |
753-
const { PUBLISH_RESULT, E2E_RESULT, HEAD_SHA, RUN_URL } = process.env;
777+
const { PUBLISH_RESULT, E2E_RESULT, E2E_SOFT_RED, HEAD_SHA, RUN_URL } = process.env;
754778
const { owner, repo } = context.repo;
755779
let state, description;
756-
if (E2E_RESULT === 'success') {
780+
if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
781+
// The suite did NOT pass: the tenant node-join deadline expired and
782+
// the lane decided not to block on it. `success` is the decision;
783+
// this text is the only place a reader standing in front of the
784+
// merge button learns what it was a decision about.
785+
state = 'success';
786+
description = 'fork e2e soft: node-join deadline missed, suite failed, merge not blocked';
787+
} else if (E2E_RESULT === 'success') {
757788
state = 'success';
758789
description = 'fork e2e passed (privileged run)';
759790
} else if (PUBLISH_RESULT !== 'success') {

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,12 @@ jobs:
134134
# enough host headroom for QEMU, networking, containerd, and the runner
135135
# agent while the full platform installs across all three guests.
136136
runs-on: oracle-vm-32cpu-128gb-x86-64
137-
timeout-minutes: 180
137+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
138+
# held to one figure across every lane that runs the suite. That bound, the
139+
# measured run the rest of the margin comes from, and what neither of them
140+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
141+
# here in four workflows that would then drift apart.
142+
timeout-minutes: 215
138143
permissions:
139144
contents: read
140145
checks: write # Breakpoint action updates its dedicated "Breakpoint Open" Check Run.
@@ -258,7 +263,7 @@ jobs:
258263
# collect-report packages cozyreport plus per-failed-test crust-gather
259264
# snapshots captured before Chainsaw cleanup.
260265
#
261-
# A ceiling of its own, not the job's 180 minutes. `Upload cozyreport.tgz`
266+
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
262267
# is the next step and the collector writes its tarball at the very end of
263268
# its run, so a read that never returns spends what is left of the job
264269
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/nightly.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,12 @@ jobs:
227227
# containerd and the runner agent. Enforced by
228228
# hack/sandbox-runner-headroom.bats.
229229
runs-on: oracle-vm-32cpu-128gb-x86-64
230-
timeout-minutes: 180
230+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
231+
# held to one figure across every lane that runs the suite. That bound, the
232+
# measured run the rest of the margin comes from, and what neither of them
233+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
234+
# here in four workflows that would then drift apart.
235+
timeout-minutes: 215
231236
permissions:
232237
contents: read
233238
packages: read
@@ -339,7 +344,7 @@ jobs:
339344
path: _out/chainsaw-report.xml
340345
if-no-files-found: ignore
341346

342-
# A ceiling of its own, not the job's 180 minutes. `Upload cozyreport.tgz`
347+
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
343348
# is the next step and the collector writes its tarball at the very end of
344349
# its run, so a read that never returns spends what is left of the job
345350
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/pull-requests.yaml‎

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,12 @@ jobs:
819819
# workflow_run instead.
820820
e2e:
821821
name: "E2E (in-tree)"
822+
# What the gate decided, promoted to a job output because the job's RESULT
823+
# can no longer answer it: a red the lane tolerated finishes `success`, so
824+
# every consumer downstream that reads the result alone would report this run
825+
# as one where the suite passed.
826+
outputs:
827+
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
822828
# The sandbox boots three QEMU guests (hack/e2e-prepare-cluster.bats), each
823829
# with `-smp 8 -m 24576`, so the guests alone demand 24 vCPU and 72 GiB.
824830
# On the 24-vCPU runner that is a 1:1 CPU commit with zero host headroom for
@@ -835,7 +841,12 @@ jobs:
835841
# leaving 8 vCPU (25%) and 56 GiB of host headroom, which removes the
836842
# oversubscription that starves the probes.
837843
runs-on: oracle-vm-32cpu-128gb-x86-64
838-
timeout-minutes: 180
844+
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
845+
# held to one figure across every lane that runs the suite. That bound, the
846+
# measured run the rest of the margin comes from, and what neither of them
847+
# covers are all in hack/node-join-soft-red_test.bats rather than restated
848+
# here in four workflows that would then drift apart.
849+
timeout-minutes: 215
839850
permissions:
840851
contents: read
841852
packages: read
@@ -1035,6 +1046,20 @@ jobs:
10351046
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
10361047
' || true
10371048
echo "::endgroup::"
1049+
# One failure the lane does not block on, and only one: the tenant
1050+
# node-join deadline. The test itself stays red -- Chainsaw ran its
1051+
# whole catch above and the report records the failure -- and this
1052+
# decides only whether the job does. It blocks on anything it cannot
1053+
# attribute to that deadline, including a report it could not read.
1054+
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
1055+
# The breakpoint below opens on `failure()`, which this exit is
1056+
# about to make false. Record the decision so it can open on this
1057+
# red too: the sandbox is still up at that point, and a join that
1058+
# never finished is the failure an interactive session is worth
1059+
# most on.
1060+
echo "soft_red=true" >>"$GITHUB_OUTPUT"
1061+
exit 0
1062+
fi
10381063
exit 1
10391064
fi
10401065
echo "✅ All selected E2E tests passed"
@@ -1056,7 +1081,7 @@ jobs:
10561081

10571082
# ▸ Collect debug information (always runs)
10581083
#
1059-
# A ceiling of its own, not the job's 180 minutes. `Upload
1084+
# A ceiling of its own, not the job's 215 minutes. `Upload
10601085
# cozyreport.tgz` is the next step and the collector writes its tarball
10611086
# at the very end of its run, so a read that never returns spends what
10621087
# is left of the job budget, the job is cancelled, and the upload never
@@ -1067,7 +1092,7 @@ jobs:
10671092
# 30 minutes is measured against the step: it takes 3-42s across sampled
10681093
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
10691094
# `timeout-minutes` caps a step, it does not carve budget out of the job,
1070-
# so a suite that has already burned 175 of the 180 leaves this step five
1095+
# so a suite that has already burned 210 of the 215 leaves this step five
10711096
# minutes and the upload is cancelled with everything else. What the cap
10721097
# buys is that a hung read cannot be the reason.
10731098
# `continue-on-error` because the ceiling must not change what the job
@@ -1126,6 +1151,13 @@ jobs:
11261151
# implies maintainer authorship). authorized-users is kept as a second
11271152
# defense layer controlling who may actually SSH in.
11281153
#
1154+
# A node-join red the lane decided not to block on reaches here as a
1155+
# SUCCESSFUL job, so `failure()` alone would skip the step on the one
1156+
# failure whose sandbox is worth attaching to. The E2E step records that
1157+
# decision and this reads it back. Being label-gated is what makes that
1158+
# affordable: the session opens for a PR someone deliberately marked for
1159+
# debugging, not on every softened run.
1160+
#
11291161
# Uses cozystack/breakpoint-action (fork of namespacelabs/breakpoint-action)
11301162
# pinned by SHA. The fork adds: pause-idle mode (initial grace period for
11311163
# the first SSH connection, idle-aware exit afterwards), endpoint output
@@ -1139,10 +1171,15 @@ jobs:
11391171
# forks cannot reach the self-hosted rendezvous server.
11401172
- name: Breakpoint on E2E failure
11411173
if: |
1142-
failure() &&
1174+
(failure() || steps.e2e_tests.outputs.soft_red == 'true') &&
11431175
vars.BREAKPOINT_ENDPOINT != '' &&
11441176
(contains(github.event.pull_request.labels.*.name, 'debug')
11451177
|| contains(github.event.pull_request.labels.*.name, 'release'))
1178+
# A step that fails here fails an otherwise-successful job, which on the
1179+
# softened path would let a debugging tool overturn a lane that had
1180+
# decided not to block. Same reason `Collect report` above carries it:
1181+
# what the job REPORTS must not depend on what a helper did.
1182+
continue-on-error: true
11461183
# cozystack/breakpoint-action v2-cozy.1
11471184
# mode: pause-idle defaults: grace-period=20m, idle-timeout=10m
11481185
uses: cozystack/breakpoint-action@a6f3a6f87be398ad63b6577351e3398e53f578e4
@@ -1235,14 +1272,15 @@ jobs:
12351272
PLAN_RESULT: ${{ needs.plan.result }}
12361273
CODE: ${{ needs.plan.outputs.code }}
12371274
E2E_RESULT: ${{ needs.e2e.result }}
1275+
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
12381276
CANDIDATE_RESULT: ${{ needs.verify-release-candidate.result }}
12391277
IS_RELEASE: ${{ contains(github.event.pull_request.labels.*.name, 'release') }}
12401278
IS_FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
12411279
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
12421280
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
12431281
with:
12441282
script: |
1245-
const { PLAN_RESULT, CODE, E2E_RESULT, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env;
1283+
const { PLAN_RESULT, CODE, E2E_RESULT, E2E_SOFT_RED, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env;
12461284
let state, description;
12471285
if (PLAN_RESULT !== 'success') {
12481286
state = 'failure';
@@ -1263,6 +1301,13 @@ jobs:
12631301
// full suite re-run or an admin override.
12641302
state = 'success';
12651303
description = 'release promotion — verified at rc, e2e not repeated (add full-e2e to force a run)';
1304+
} else if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
1305+
// The suite did NOT pass: the tenant node-join deadline expired and
1306+
// the lane decided not to block on it. `success` is the decision;
1307+
// this text is the only place a reader standing in front of the
1308+
// merge button learns what it was a decision about.
1309+
state = 'success';
1310+
description = 'in-tree e2e soft: node-join deadline missed, suite failed, merge not blocked';
12661311
} else if (E2E_RESULT === 'success') {
12671312
state = 'success';
12681313
description = 'in-tree e2e passed';

0 commit comments

Comments
 (0)