Skip to content

Commit 1d3ed2d

Browse files
feat(conformance): add the openshell-conformance program
The conformance crate now also builds openshell-conformance, a binary whose main calls testkit::main() and links the crate's library. It lists, waits for the gateway, and runs the conformance tests without cargo or nextest. It runs only the conformance suite and promises only the conformance contract; feature and driver suites are not part of it. The binary has test = false, so cargo test and nextest still see one conformance test binary, the library's unit tests, and the e2e lanes and the runner's filters are unchanged. The library is linked without cfg(test), so the program gets the registrations and none of the tokio test wrappers. tasks/scripts/check-conformance-program.sh checks that openshell-conformance list names the same tests as nextest; Branch Checks and mise test:rust run it. A dependency that is never named is not linked, so a suite the program forgets to name would silently go missing. testkit::main() now fails with exit 2 when a program links no tests, under every command. Signed-off-by: Matthew Grossman <[email protected]>
1 parent e6ac23b commit 1d3ed2d

11 files changed

Lines changed: 132 additions & 6 deletions

File tree

‎.github/workflows/branch-checks.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -389,6 +389,7 @@ jobs:
389389
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/testkit/Cargo.toml
390390
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/tmachine/Cargo.toml
391391
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/runner/Cargo.toml
392+
tasks/scripts/check-conformance-program.sh
392393
393394
rust-postgres-test:
394395
name: Rust PostgreSQL tests

‎TESTING.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,12 @@ by `OPENSHELL_BIN`. See
181181
[tests/testkit/README.md](tests/testkit/README.md) for how to write and run
182182
tests.
183183

184+
The conformance crate also builds `openshell-conformance`, a program that
185+
runs the conformance suite against a gateway without cargo or nextest
186+
(`openshell-conformance list`, `ready`, `run`). It covers only the
187+
conformance suite. See
188+
[tests/suites/conformance/README.md](tests/suites/conformance/README.md).
189+
184190
Suites live under `tests/suites/<family>/`. Each family is a Cargo workspace
185191
with its own lockfile. Run the `conformance` suites against a local Docker
186192
gateway with:
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/usr/bin/env bash
2+
3+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
# Check that `openshell-conformance list` names the same tests that nextest
7+
# lists for the conformance crate, so the program runs the whole suite.
8+
# Neither command needs a gateway.
9+
10+
set -euo pipefail
11+
12+
root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
13+
manifest="${root}/tests/suites/conformance/Cargo.toml"
14+
work="$(mktemp -d)"
15+
trap 'rm -rf "${work}"' EXIT
16+
17+
cargo run --quiet --locked --manifest-path "${manifest}" --bin openshell-conformance -- list \
18+
| awk '{print $1}' | sort > "${work}/program"
19+
cargo nextest list --locked --color never --manifest-path "${manifest}" --message-format oneline \
20+
| awk '{print $2}' | sort > "${work}/nextest"
21+
22+
if ! diff -u "${work}/nextest" "${work}/program"; then
23+
echo "error: openshell-conformance list differs from nextest's list (-nextest +program)" >&2
24+
exit 1
25+
fi
26+
if [[ ! -s "${work}/program" ]]; then
27+
echo "error: openshell-conformance lists no tests" >&2
28+
exit 1
29+
fi
30+
echo "openshell-conformance lists the same $(wc -l < "${work}/program" | tr -d ' ') tests as nextest"

‎tasks/test.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ run = [
127127
"cargo nextest run --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml",
128128
"cargo nextest run --config-file .config/nextest.toml --manifest-path tests/tmachine/Cargo.toml",
129129
"cargo nextest run --config-file .config/nextest.toml --manifest-path tests/runner/Cargo.toml",
130+
"tasks/scripts/check-conformance-program.sh",
130131
]
131132
run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-precommit native"
132133
hide = true

‎tests/suites/conformance/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,14 @@ license = "Apache-2.0"
1212
repository = "https://github.com/NVIDIA/OpenShell"
1313
publish = false
1414

15+
# `openshell-conformance`: the suite as one program to run against a gateway.
16+
# No test target of its own: cargo test and nextest run the tests through the
17+
# library.
18+
[[bin]]
19+
name = "openshell-conformance"
20+
path = "src/main.rs"
21+
test = false
22+
1523
[dependencies]
1624
openshell-testkit = { path = "../../testkit/testkit" }
1725
serde_json = "1"

‎tests/suites/conformance/README.md‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
The `conformance` crate holds the testkit suites for sandbox behavior that every
44
compute driver must provide. The family is this one crate, its own workspace.
55
It is a library: each module in `src/` is one suite, and `cargo test` and
6-
nextest run the tests as the library's unit tests.
6+
nextest run the tests as the library's unit tests. The crate also builds the
7+
`openshell-conformance` program, which runs the same tests.
78

89
| Suite | Covers |
910
|---|---|
@@ -27,11 +28,31 @@ environments).
2728
[TESTING.md](../../../TESTING.md) covers running it against a local Docker
2829
gateway and in a tmachine VM, as CI does.
2930

31+
### The `openshell-conformance` program
32+
33+
`openshell-conformance` runs the suite against a gateway without cargo or
34+
nextest, so a driver author or operator can check their own gateway. It finds
35+
the gateway and the CLI as the tests do. Build it and run it with:
36+
37+
```shell
38+
cargo build --locked --manifest-path tests/suites/conformance/Cargo.toml --bin openshell-conformance
39+
tests/suites/conformance/target/debug/openshell-conformance list
40+
tests/suites/conformance/target/debug/openshell-conformance ready
41+
tests/suites/conformance/target/debug/openshell-conformance run --junit conformance.xml
42+
```
43+
44+
The program runs only this suite: passing it means a gateway meets the
45+
conformance contract below. Feature and driver suites are not part of it. The
46+
[testkit README](../../testkit/README.md#test-programs) describes the commands
47+
and exit codes. `tasks/scripts/check-conformance-program.sh` checks that it
48+
lists the same tests as nextest.
49+
3050
## Adding a test
3151

3252
Add a `#[testkit::test]` function to a suite, or for a new area, a new file in
3353
`src/` and a `mod` line for it in `src/lib.rs`. No playbook, test archive or
34-
workflow entry is needed: the runner builds and runs every test in the crate.
54+
workflow entry is needed: the runner builds and runs every test in the
55+
crate, and `openshell-conformance` includes it.
3556

3657
## What a test may assert
3758

‎tests/suites/conformance/src/lib.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
//! provide. Each module is one suite.
66
//!
77
//! `cargo test` and nextest run the tests as this library's unit tests.
8-
//! `#[testkit::test]` also registers them, so a test program runs them by
9-
//! linking this crate.
8+
//! `#[testkit::test]` also registers them, so the `openshell-conformance`
9+
//! program in `src/main.rs` runs them by linking this library.
1010
1111
mod cli;
1212
mod file_transfer;
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//! `openshell-conformance`: the conformance suite as one program, to run
5+
//! against a gateway without cargo or nextest.
6+
//!
7+
//! The tests are registered by `#[testkit::test]` in this crate's library;
8+
//! see `testkit::program` for the commands.
9+
10+
use std::process::ExitCode;
11+
12+
// A dependency that is never named is not linked, and its tests would be
13+
// missing.
14+
use conformance as _;
15+
16+
fn main() -> ExitCode {
17+
testkit::main()
18+
}

‎tests/testkit/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,11 +106,20 @@ In a tmachine VM, the test runner (`tests/runner/`) does the same: it sets
106106
cargo or nextest:
107107

108108
```rust
109+
use conformance as _;
110+
109111
fn main() -> std::process::ExitCode {
110112
testkit::main()
111113
}
112114
```
113115

116+
The program must name each suite crate it runs, as `use conformance as _;` does
117+
above: Rust doesn't link a dependency that is never named, and its tests would
118+
be missing. A program that links no tests exits 2 with an error under every
119+
command. `openshell-conformance`, a binary in the conformance crate, is the
120+
program for the conformance suite; see
121+
[its README](../suites/conformance/README.md).
122+
114123
| Command | What it does |
115124
|---|---|
116125
| `list [FILTER]...` | Prints each test's name and description. Ignored tests are included and marked. |

‎tests/testkit/testkit/src/program/mod.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,15 @@ pub fn main() -> ExitCode {
6060
pub fn main_with(tests: &'static [Test]) -> ExitCode {
6161
let args: Vec<OsString> = std::env::args_os().collect();
6262
let program = program_name(&args);
63+
// A suite crate that the program never names is not linked, and its
64+
// tests are silently missing. Fail instead of passing with nothing.
65+
if tests.is_empty() {
66+
eprintln!(
67+
"{program}: no tests are linked into this program; it must name each suite crate \
68+
it runs, for example with `use <suite> as _;`"
69+
);
70+
return ExitCode::from(ERROR_EXIT);
71+
}
6372
if let Err(error) = validate(tests) {
6473
eprintln!("{program}: {error}");
6574
return ExitCode::from(ERROR_EXIT);

0 commit comments

Comments
 (0)