Skip to content

Commit ff09b71

Browse files
matthewgrossmanmatthewgrossman
committed
ci(tests): run conformance-integration through the runner
Switch the required conformance-integration job in Branch E2E, Release Dev and Release Qualification from the tmachine conformance testsuite to the E2E Suites workflow, selecting the conformance family. The job ID is unchanged. Delete the tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive: the sandbox package covers the same tests, including the ported policy-advisor ones. The Branch E2E E2E Suites job now runs the features family only, so conformance does not run twice. Point the integration runner defaults and the docs at what remains. Signed-off-by: Matthew Grossman <[email protected]>
1 parent 09a6746 commit ff09b71

13 files changed

Lines changed: 47 additions & 249 deletions

File tree

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
125125

126126
## View Job Logs
127127

128-
For tmachine Kubernetes conformance failures, inspect the `tmachine-diagnostics-*` artifact. The K3s report includes Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The conformance playbook checks reconciliation after guest boot with a temporary scheduling-gated Sandbox probe before running tests. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
128+
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-sandbox-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
129129

130130
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL after the end of the nextest log (which names the failed tests), or ERROR with the step that broke. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
131131

‎.github/workflows/branch-e2e.yml‎

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -247,19 +247,11 @@ jobs:
247247
actions: read
248248
contents: read
249249
packages: read
250-
uses: ./.github/workflows/integration-runner.yml
250+
uses: ./.github/workflows/e2e-suites.yml
251251
with:
252-
category: conformance
252+
family: conformance
253253
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
254254
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
255-
test-matrix: >-
256-
[
257-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
258-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
259-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
260-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
261-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
262-
]
263255

264256
# Run feature-specific integration tests:
265257
feature-specific-integration:
@@ -303,9 +295,9 @@ jobs:
303295
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
304296
]
305297
306-
# Run the testkit suites in tests/suites/ on tmachine. The environments and
307-
# suites are listed in e2e-suites.yml. Advisory: not part of
308-
# core-e2e-result.needs.
298+
# Run the testkit feature suites in tests/suites/ on tmachine. The
299+
# environments and suites are listed in e2e-suites.yml; the conformance rows
300+
# run in conformance-integration. Advisory: not part of core-e2e-result.needs.
309301
e2e-suites:
310302
needs: [pr_metadata, prepare-integration]
311303
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -315,6 +307,7 @@ jobs:
315307
packages: read
316308
uses: ./.github/workflows/e2e-suites.yml
317309
with:
310+
family: features
318311
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
319312
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
320313

‎.github/workflows/integration-runner.yml‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,8 @@ on:
2424
type: string
2525
default: >-
2626
[
27-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
28-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
29-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
30-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
31-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
27+
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
28+
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
3229
]
3330
3431
permissions:

‎.github/workflows/integration-test.yml‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,15 @@ on:
2020
description: Integration-test category, used for the job name and concurrency key
2121
required: true
2222
type: string
23-
default: conformance
23+
default: feature-specific
2424
test-matrix:
2525
description: JSON array of environment, installer, and testsuite combinations
2626
required: true
2727
type: string
2828
default: >-
2929
[
30-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
31-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
32-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
33-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
34-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
30+
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
31+
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
3532
]
3633
3734
permissions:

‎.github/workflows/release-dev.yml‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -118,19 +118,11 @@ jobs:
118118
actions: read
119119
contents: read
120120
packages: read
121-
uses: ./.github/workflows/integration-runner.yml
121+
uses: ./.github/workflows/e2e-suites.yml
122122
with:
123-
category: conformance
123+
family: conformance
124124
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
125125
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
126-
test-matrix: >-
127-
[
128-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
129-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
130-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
131-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
132-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
133-
]
134126

135127
feature-specific-integration:
136128
needs: prepare-integration

‎.github/workflows/release-qualification.yml‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -71,19 +71,11 @@ jobs:
7171
actions: read
7272
contents: read
7373
packages: read
74-
uses: ./.github/workflows/integration-runner.yml
74+
uses: ./.github/workflows/e2e-suites.yml
7575
with:
76-
category: conformance
76+
family: conformance
7777
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
7878
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
79-
test-matrix: >-
80-
[
81-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
82-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
83-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
84-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
85-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
86-
]
8779

8880
feature-specific-integration:
8981
needs: prepare-integration

‎CI.md‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,7 @@ push-triggered workflow jobs themselves.
174174

175175
### K3s conformance version baseline
176176

177-
The tmachine `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
177+
The `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
178178
compatibility baseline for the v1beta1 Sandbox API. They do not track the local
179179
K3s development default, currently v1.0.3. OpenShell also supports v0.4.6 through
180180
its v1alpha1 fallback, so v0.5.0 is not the overall minimum supported version.
@@ -184,26 +184,26 @@ recreating the Service requires reprovisioning the fixture. The `k3s` lane runs
184184
one plaintext gateway. The `k3s-ha` lane runs the supported HA configuration:
185185
three gateway replicas sharing PostgreSQL, serving TLS, with the CLI connecting
186186
over mTLS. Before conformance, every installer waits up to five minutes for a
187-
connected gateway; interactive shells skip the wait. K3s failures upload
188-
diagnostics as `tmachine-diagnostics-*` artifacts.
187+
connected gateway; interactive shells skip the wait. Each lane uploads its
188+
JUnit report, nextest log and, after a failure, the guest's diagnostics as an
189+
`e2e-results-*` artifact.
189190

190-
### Run only the policy advisor conformance tests
191+
### Run a subset of the conformance tests
191192

192-
Manually dispatch `Integration Tests` on the candidate branch with an
193-
`artifact-run-id` from a build of the same commit. When a `Release` or `Pre-release` run was
194-
dispatched from a different commit, also set `source-sha` to the release tag's
195-
resolved commit so runtime images and test inputs match the candidate binaries.
196-
Otherwise, `source-sha` defaults to the artifact run's head SHA. Set `category`
197-
to `policy-advisor` and `test-matrix` to:
193+
The conformance lanes run the `sandbox` package of `tests/suites/conformance`
194+
through `.github/workflows/e2e-suites.yml`, which keeps the list of
195+
environments. To run one suite in one environment from a checkout, build the
196+
candidate inputs (see [TESTING.md](TESTING.md)) and narrow the tests with a
197+
nextest filterset:
198198

199-
```json
200-
[{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
199+
```shell
200+
nix run .#openshell-test-runner -- \
201+
--environment ubuntu-docker-rootful --installer deb \
202+
--family conformance --package sandbox \
203+
--filter 'binary_id(=sandbox::policy_advisor)'
201204
```
202205

203-
This runs the `mechanistic-proposal`, `new-hostname-proposal`, and
204-
`policy-local` conformance tests in the installed-artifact suite. The artifact run must contain the candidate CLI and
205-
gateway binaries and runtime images. This manual run does not replace the
206-
required PR E2E gate.
206+
A local run does not replace the required PR E2E gate.
207207

208208
## Informational security reports
209209

‎TESTING.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -367,9 +367,8 @@ Suites:
367367
- CLI conformance (`tests/suites/conformance`) - portable Cargo tests for
368368
lifecycle, mechanistic drafts, file transfer, and the sandbox-local API,
369369
including agent-authored permission requests. Driver E2E runs the complete
370-
Cargo test package. The installed-artifact conformance suite runs the same
371-
tests from a nextest archive and offers a focused `policy-advisor` testsuite
372-
for manual integration runs.
370+
Cargo test package. The installed-artifact conformance lanes run the `sandbox`
371+
package through `openshell-test-runner` (see below).
373372
- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
374373
`e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
375374
the selected deployment.
@@ -479,7 +478,9 @@ rebuilds only the RPMs. Run conformance against the installed RPMs:
479478

480479
```shell
481480
nix run .#build-artifacts
482-
nix run .#tmachine -- test fedora-podman-rootful rpm conformance
481+
nix run .#openshell-test-runner -- \
482+
--environment fedora-podman-rootful --installer rpm \
483+
--family conformance --package sandbox
483484
```
484485

485486
Run the portable subset in a disposable rootless Podman guest:

‎e2e/policy-advisor/README.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -55,19 +55,21 @@ contents write on the repository. The test auto-resolves the token from
5555

5656
## Conformance coverage
5757

58-
The portable CLI conformance suite may include policy-related scenarios. The
58+
The portable conformance suite may include policy-related scenarios. The
5959
current coverage checks draft generation for denied IP addresses and hostnames
6060
absent from policy, plus sandbox-local policy inspection and permission
6161
requests through `policy.local`. See [TESTING.md](../../TESTING.md#rust-cli-e2e)
6262
for the supported source and installed-artifact test paths.
6363

64-
A manual `Integration Tests` workflow run can select the `policy-advisor`
65-
testsuite to run the policy-related subset against an installed candidate. Set
66-
`artifact-run-id` to the candidate build's workflow run ID and `test-matrix`
67-
to:
64+
To run only the policy-related conformance tests against an installed
65+
candidate, run the `policy-advisor` suite of the `sandbox` package through the
66+
test runner:
6867

69-
```json
70-
[{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
68+
```bash
69+
nix run .#openshell-test-runner -- \
70+
--environment ubuntu-docker-rootful --installer deb \
71+
--family conformance --package sandbox \
72+
--filter 'binary_id(=sandbox::policy_advisor)'
7173
```
7274

7375
The GitHub write test above and the regressions below still exercise distinct

‎tests/tmachine/ansible/playbooks/conformance/cli.yaml‎

Lines changed: 0 additions & 144 deletions
This file was deleted.

0 commit comments

Comments
 (0)