Skip to content

Commit b6fad3c

Browse files
refactor(conformance): make the conformance family one crate
Conformance is one crate per family, so the family directory is the crate: tests/suites/conformance/sandbox becomes the `conformance` package at tests/suites/conformance, with its suites still in tests/. The workspace manifest path, its lockfile and the scripts that run it by path are unchanged. Nextest's binaries for these tests are now `conformance::<suite>`; test names are unchanged. The tmachine runner rows, the fmt and clippy lines and the docs name the new package. Signed-off-by: Matthew Grossman <[email protected]>
1 parent 81cdd4e commit b6fad3c

17 files changed

Lines changed: 54 additions & 57 deletions

File tree

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
125125

126126
## View Job Logs
127127

128-
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-sandbox-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
128+
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-conformance-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
129129

130130
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL (exit 1) after the end of the nextest log (which names the failed tests), or ERROR (exit 2) with the step that broke. FAIL and ERROR also appear as annotations on the run's summary page. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
131131

‎.github/workflows/branch-checks.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,7 @@ jobs:
233233
cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
234234
cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check
235235
cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check
236-
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox -- --check
236+
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance -- --check
237237
cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check
238238
cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check
239239
@@ -297,7 +297,7 @@ jobs:
297297
cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
298298
cargo clippy --locked --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings
299299
cargo clippy --locked --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings
300-
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --all-targets -- -D warnings
300+
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml -p conformance --all-targets -- -D warnings
301301
cargo clippy --locked --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings
302302
cargo clippy --locked --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings
303303

‎.github/workflows/e2e-suites.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,11 @@ jobs:
6464
# installer registers with the CLI: `openshell` for the packages,
6565
# `tmachine` for the k3s installers.
6666
rows='[
67-
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
68-
{"environment":"fedora-podman-rootful","installer":"rpm","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
69-
{"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
70-
{"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"sandbox","gateway":"tmachine","runner":"ubuntu-24.04"},
71-
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"sandbox","gateway":"tmachine","runner":"ubuntu-24.04"},
67+
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
68+
{"environment":"fedora-podman-rootful","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
69+
{"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
70+
{"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
71+
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
7272
{"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}
7373
]'
7474
matrix=$(jq -c --arg family "${FAMILY}" \

‎CI.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ JUnit report, nextest log and, after a failure, the guest's diagnostics as an
204204

205205
### Run a subset of the conformance tests
206206

207-
The conformance lanes run the `sandbox` package of `tests/suites/conformance`
207+
The conformance lanes run the `conformance` crate in `tests/suites/conformance`
208208
through `.github/workflows/e2e-suites.yml`, which keeps the list of
209209
environments. To run one suite in one environment from a checkout, build the
210210
candidate inputs (see [TESTING.md](TESTING.md)) and narrow the tests with a
@@ -213,8 +213,8 @@ nextest filterset:
213213
```shell
214214
nix run .#openshell-test-runner -- \
215215
--environment ubuntu-docker-rootful --installer deb \
216-
--family conformance --package sandbox \
217-
--filter 'binary_id(=sandbox::policy_advisor)'
216+
--family conformance --package conformance \
217+
--filter 'binary_id(=conformance::policy_advisor)'
218218
```
219219

220220
A local run does not replace the required PR E2E gate.

‎TESTING.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ OpenShell that the VM's installer set up.
223223
```shell
224224
nix run .#openshell-test-runner -- \
225225
--environment ubuntu-docker-rootful --installer deb \
226-
--family conformance --package sandbox
226+
--family conformance --package conformance
227227
```
228228

229229
It needs KVM, and tmachine only boots guests of the host's architecture. The
@@ -412,10 +412,10 @@ HTTP `Host`, TLS SNI, and mTLS handling.
412412
Suites:
413413

414414
- Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
415-
- Conformance (`tests/suites/conformance`) - the testkit `sandbox` package:
415+
- Conformance (`tests/suites/conformance`) - the testkit `conformance` crate:
416416
lifecycle, the CLI, persistent workspaces, file transfer, and the policy
417417
advisor. Driver E2E runs it with nextest against the lane's gateway. The
418-
installed-artifact conformance lanes run the same package through
418+
installed-artifact conformance lanes run the same crate through
419419
`openshell-test-runner` (see above).
420420
- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
421421
`e2e-vm`) - conformance plus the common and driver-specific coverage for
@@ -503,7 +503,7 @@ rebuilds only the RPMs. Run conformance against the installed RPMs:
503503
nix run .#build-artifacts
504504
nix run .#openshell-test-runner -- \
505505
--environment fedora-podman-rootful --installer rpm \
506-
--family conformance --package sandbox
506+
--family conformance --package conformance
507507
```
508508

509509
Run the portable subset in a disposable rootless Podman guest:

‎e2e/policy-advisor/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,14 @@ requests through `policy.local`. See [TESTING.md](../../TESTING.md#rust-cli-e2e)
6262
for the supported source and installed-artifact test paths.
6363

6464
To run only the policy-related conformance tests against an installed
65-
candidate, run the `policy-advisor` suite of the `sandbox` package through the
65+
candidate, run the `policy-advisor` suite of the `conformance` crate through the
6666
test runner:
6767

6868
```bash
6969
nix run .#openshell-test-runner -- \
7070
--environment ubuntu-docker-rootful --installer deb \
71-
--family conformance --package sandbox \
72-
--filter 'binary_id(=sandbox::policy_advisor)'
71+
--family conformance --package conformance \
72+
--filter 'binary_id(=conformance::policy_advisor)'
7373
```
7474

7575
The GitHub write test above and the regressions below still exercise distinct

‎tasks/rust.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ run = [
3939
"cargo clippy --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings",
4040
"cargo clippy --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings",
4141
"cargo clippy --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings",
42-
"cargo clippy --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --all-targets -- -D warnings",
42+
"cargo clippy --manifest-path tests/suites/conformance/Cargo.toml -p conformance --all-targets -- -D warnings",
4343
"cargo clippy --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings",
4444
"cargo clippy --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings",
4545
]
@@ -57,7 +57,7 @@ run = [
5757
"cargo fmt --manifest-path tests/testkit/Cargo.toml --all",
5858
"cargo fmt --manifest-path tests/tmachine/Cargo.toml --all",
5959
"cargo fmt --manifest-path tests/runner/Cargo.toml --all",
60-
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox",
60+
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance",
6161
"cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh",
6262
"cargo fmt --manifest-path tests/support/Cargo.toml --all",
6363
]
@@ -74,7 +74,7 @@ run = [
7474
"cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check",
7575
"cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check",
7676
"cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check",
77-
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox -- --check",
77+
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance -- --check",
7878
"cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check",
7979
"cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check",
8080
]

‎tests/suites/conformance/Cargo.lock‎

Lines changed: 10 additions & 10 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/conformance/Cargo.toml‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,21 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
# SPDX-License-Identifier: Apache-2.0
33

4+
# The conformance family is one crate; each file in `tests/` is a suite.
5+
[package]
6+
name = "conformance"
7+
description = "Conformance suite: sandbox behavior on every compute driver"
8+
version = "0.0.0"
9+
edition = "2024"
10+
rust-version = "1.94"
11+
license = "Apache-2.0"
12+
repository = "https://github.com/NVIDIA/OpenShell"
13+
publish = false
14+
15+
[dependencies]
16+
openshell-testkit = { path = "../../testkit/testkit" }
17+
serde_json = "1"
18+
tempfile = "3"
19+
tokio = { version = "1", features = ["time"] }
20+
421
[workspace]
5-
resolver = "2"
6-
members = ["sandbox"]

tests/suites/conformance/sandbox/README.md renamed to tests/suites/conformance/README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
# Sandbox conformance suite
1+
# Conformance suites
22

3-
The `sandbox` crate holds testkit suites for sandbox behavior that every compute
4-
driver must provide. It is a member of the `conformance` workspace. Each file in
5-
`tests/` is one suite.
3+
The `conformance` crate holds the testkit suites for sandbox behavior that every
4+
compute driver must provide. The family is this one crate, its own workspace.
5+
Each file in `tests/` is one suite.
66

77
| Suite | Covers |
88
|---|---|
@@ -23,14 +23,14 @@ environments).
2323

2424
## Running it
2525

26-
[TESTING.md](../../../../TESTING.md) covers running it against a local Docker
26+
[TESTING.md](../../../TESTING.md) covers running it against a local Docker
2727
gateway and in a tmachine VM, as CI does.
2828

2929
## Adding a test
3030

3131
Add a `#[testkit::test]` function to a suite, or a new file in `tests/` for a new
3232
area. No playbook, test archive or workflow entry is needed: the runner builds
33-
and runs every test in the package.
33+
and runs every test in the crate.
3434

3535
## What a test may assert
3636

0 commit comments

Comments
 (0)