Skip to content

Commit bd8184a

Browse files
committed
ci(tests): run the conformance lanes through the test runner
conformance-integration in Branch E2E, Release Dev, and Release Qualification now runs the cpu-tests plan through e2e-suites.yml, one job per amd64 environment, replacing the tmachine CLI conformance run. A matrix entry can name an environment to run on its own. The separate advisory e2e-suites job is removed. Signed-off-by: Evan Lezar <[email protected]>
1 parent a29ce0f commit bd8184a

5 files changed

Lines changed: 50 additions & 55 deletions

File tree

‎.github/workflows/branch-e2e.yml‎

Lines changed: 9 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -240,25 +240,25 @@ jobs:
240240
deb-artifact-name: deb-linux-amd64
241241
rpm-artifact-name: rpm-linux-x86_64
242242

243-
# Run driver-independent conformance tests.
243+
# Run the conformance suites the cpu-tests plan selects, one job per amd64
244+
# environment.
244245
conformance-integration:
245246
needs: prepare-integration
246247
permissions:
247248
actions: read
248249
contents: read
249250
packages: read
250-
uses: ./.github/workflows/integration-runner.yml
251+
uses: ./.github/workflows/e2e-suites.yml
251252
with:
252-
category: conformance
253253
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
254254
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
255-
test-matrix: >-
255+
matrix: >-
256256
[
257-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
258-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
259-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
260-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
261-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
257+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.docker-rootful.deb"},
258+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.k3s.k3s"},
259+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.k3s.k3s-ha"},
260+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.fedora44-amd64.podman-rootful.rpm"},
261+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.fedora44-amd64.podman-rootless.rpm"}
262262
]
263263
264264
# Run feature-specific integration tests:
@@ -303,24 +303,6 @@ jobs:
303303
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
304304
]
305305
306-
# Run the SDK-driven e2e suites that each plan in tests/plans/ selects, on
307-
# each runner. Advisory until it has a passing history on amd64, so it is
308-
# not part of core-e2e-result.
309-
e2e-suites:
310-
needs: prepare-integration
311-
permissions:
312-
actions: read
313-
contents: read
314-
packages: read
315-
uses: ./.github/workflows/e2e-suites.yml
316-
with:
317-
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
318-
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
319-
matrix: >-
320-
[
321-
{"plan":"cpu-tests","runner":"ubuntu-24.04"}
322-
]
323-
324306
docker-e2e:
325307
needs: [pr_metadata, build-binaries, build-images]
326308
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'

‎.github/workflows/e2e-suites.yml‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,10 @@ on:
1616
type: string
1717
matrix:
1818
description: >-
19-
JSON array of {"plan", "runner"} entries. A plan is a file stem in
20-
tests/plans/ that selects the suites and environments to run.
19+
JSON array of {"plan", "runner"} entries, each with an optional
20+
"environment". A plan is a file stem in tests/plans/ that selects
21+
the suites and environments to run; "environment" narrows an entry
22+
to one of them, so environments run as parallel jobs.
2123
required: true
2224
type: string
2325

@@ -34,7 +36,7 @@ concurrency:
3436

3537
jobs:
3638
e2e-suites:
37-
name: E2E suites (${{ matrix.plan }}, ${{ matrix.runner }})
39+
name: E2E suites (${{ matrix.plan }}, ${{ matrix.environment || matrix.runner }})
3840
strategy:
3941
fail-fast: false
4042
matrix:
@@ -75,17 +77,22 @@ jobs:
7577
- name: Run suites
7678
env:
7779
PLAN: ${{ matrix.plan }}
80+
ENVIRONMENT: ${{ matrix.environment || '' }}
7881
run: |
7982
# Retry dependency preparation, then run the plan once. The runner
8083
# builds the suites for the guest, so it needs no prepared archives.
8184
nix build --no-link .#test-runner || nix build --no-link .#test-runner
82-
nix run .#test-runner -- test --plan "tests/plans/${PLAN}.yaml" --results artifacts/test-results
85+
args=(test --plan "tests/plans/${PLAN}.yaml" --results artifacts/test-results)
86+
if [[ -n "${ENVIRONMENT}" ]]; then
87+
args+=(--env "${ENVIRONMENT}")
88+
fi
89+
nix run .#test-runner -- "${args[@]}"
8390
8491
- name: Upload results
8592
if: always()
8693
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
8794
with:
88-
name: e2e-results-${{ matrix.plan }}-${{ matrix.runner }}
95+
name: e2e-results-${{ matrix.plan }}-${{ matrix.environment || matrix.runner }}
8996
path: artifacts/test-results
9097
if-no-files-found: ignore
9198
retention-days: 7

‎.github/workflows/release-dev.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -112,24 +112,25 @@ jobs:
112112
deb-artifact-name: deb-linux-amd64
113113
rpm-artifact-name: rpm-linux-x86_64
114114

115+
# Run the conformance suites the cpu-tests plan selects, one job per amd64
116+
# environment.
115117
conformance-integration:
116118
needs: prepare-integration
117119
permissions:
118120
actions: read
119121
contents: read
120122
packages: read
121-
uses: ./.github/workflows/integration-runner.yml
123+
uses: ./.github/workflows/e2e-suites.yml
122124
with:
123-
category: conformance
124125
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
125126
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
126-
test-matrix: >-
127+
matrix: >-
127128
[
128-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
129-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
130-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
131-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
132-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
129+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.docker-rootful.deb"},
130+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.k3s.k3s"},
131+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.k3s.k3s-ha"},
132+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.fedora44-amd64.podman-rootful.rpm"},
133+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.fedora44-amd64.podman-rootless.rpm"}
133134
]
134135
135136
feature-specific-integration:

‎.github/workflows/release-qualification.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -65,24 +65,25 @@ jobs:
6565
deb-artifact-name: deb-linux-amd64
6666
rpm-artifact-name: rpm-linux-x86_64
6767

68+
# Run the conformance suites the cpu-tests plan selects, one job per amd64
69+
# environment.
6870
conformance-integration:
6971
needs: prepare-integration
7072
permissions:
7173
actions: read
7274
contents: read
7375
packages: read
74-
uses: ./.github/workflows/integration-runner.yml
76+
uses: ./.github/workflows/e2e-suites.yml
7577
with:
76-
category: conformance
7778
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
7879
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
79-
test-matrix: >-
80+
matrix: >-
8081
[
81-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
82-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
83-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
84-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
85-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
82+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.docker-rootful.deb"},
83+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.k3s.k3s"},
84+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.ubuntu2404-amd64.k3s.k3s-ha"},
85+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.fedora44-amd64.podman-rootful.rpm"},
86+
{"plan":"cpu-tests","runner":"ubuntu-24.04","environment":"tmachine.fedora44-amd64.podman-rootless.rpm"}
8687
]
8788
8889
feature-specific-integration:

‎CI.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -156,12 +156,16 @@ retain the binary installer because their fixtures configure its system service,
156156
local HTTP gateway, and CLI path. The manual Integration Tests workflow defaults
157157
to the package installers and downloads the packages selected by its matrix.
158158

159-
Core integration qualification also runs the SDK-driven e2e suites
160-
(`e2e-suites.yml`) on the same prepared inputs. Each matrix entry pairs a plan
161-
from `tests/plans/`, which selects the suites and environments, with a runner.
162-
The test runner builds the suites for the guest itself, and each entry uploads
163-
JUnit reports and failure diagnostics as `e2e-results-<plan>-<runner>`. The job
164-
is advisory and is not part of the `OpenShell / E2E` status yet.
159+
The conformance lanes (`conformance-integration`) run the e2e suites through
160+
`e2e-suites.yml` on the same prepared inputs. Each matrix entry pairs a plan
161+
from `tests/plans/`, which selects the suites and environments, with a runner,
162+
and optionally narrows it to one environment so environments run as parallel
163+
jobs. The test runner builds the suites for the guest itself, and each entry
164+
uploads JUnit reports and failure diagnostics as
165+
`e2e-results-<plan>-<environment>`. Branch E2E, Release Dev, and Release
166+
Qualification run the `cpu-tests` plan on the five amd64 environments: Ubuntu
167+
with Docker (deb), K3s and HA K3s, and Fedora with rootful and rootless Podman
168+
(rpm).
165169

166170
Three opt-in labels enable the long-running E2E suites:
167171

@@ -561,7 +565,7 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma
561565
|---|---|
562566
| `.github/workflows/branch-checks.yml` | Required non-E2E checks. Triggers on `push: pull-request/[0-9]+` for PR mirrors and `merge_group` for queued merges. |
563567
| `.github/workflows/branch-e2e.yml` | Standard, GPU, Kubernetes HA, and Kubernetes credential-driver E2E. PR mirror pushes use `test:e2e`, `test:e2e-gpu`, and `test:e2e-kubernetes` labels; merge groups do not run E2E. |
564-
| `.github/workflows/e2e-suites.yml` | Reusable matrix job that runs the e2e suites selected by plans in `tests/plans/` on the given runners and uploads `e2e-results-<plan>-<runner>`. |
568+
| `.github/workflows/e2e-suites.yml` | Reusable matrix job that runs the e2e suites selected by plans in `tests/plans/` on the given runners, optionally one environment per job, and uploads `e2e-results-<plan>-<environment or runner>`. Used by the conformance lanes. |
565569
| `.github/workflows/build-binaries.yml`, `build-vm-driver.yml` | Shared binary matrices used by branch and release workflows. The VM driver remains separate because its build consumes the runtime binaries. |
566570
| `.github/workflows/build-images.yml` | Builds and pushes multi-platform images, then uploads the same OCI images as workflow artifacts. |
567571
| `.github/workflows/package-release-binaries.yml` | Packages raw build artifacts into release tarballs without rebuilding them. |

0 commit comments

Comments
 (0)