Skip to content

Commit e0d2385

Browse files
test(conformance): delete the old CLI conformance suite
Nothing runs tests/suites/conformance/cli now that the host lanes run the testkit suite. Remove it from the conformance workspace and its lockfile, drop the e2e:testkit exclusion for it, and point the TESTING.md manual run at the `sandbox` package. e2e:rust:push also loses its dependency on e2e:conformance:build, which no longer exists, and the unused OPENSHELL_CONFORMANCE_BIN it set. crates/openshell-conformance stays for the features and drivers suites. Signed-off-by: Matthew Grossman <[email protected]>
1 parent 3faea31 commit e0d2385

9 files changed

Lines changed: 14 additions & 195 deletions

File tree

‎TESTING.md‎

Lines changed: 10 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -397,52 +397,28 @@ Run the Docker-backed Rust CLI e2e suite:
397397
mise run e2e:docker
398398
```
399399

400-
Run the portable CLI conformance suite against the gateway selected in your
401-
OpenShell CLI configuration:
400+
Run the conformance suite against the gateway selected in your OpenShell CLI
401+
configuration:
402402

403403
```shell
404404
cargo build --package openshell-cli
405405
OPENSHELL_BIN="$PWD/target/debug/openshell" \
406-
cargo test \
406+
cargo nextest run \
407407
--locked \
408408
--manifest-path tests/suites/conformance/Cargo.toml \
409-
--package openshell-test-conformance-cli \
410-
--no-fail-fast \
411-
-- \
412-
--test-threads=1 \
413-
--nocapture
409+
--package sandbox \
410+
--test-threads 1 \
411+
--no-fail-fast
414412
```
415413

416414
The gateway must already be installed, reachable, and selected before the tests
417415
start. The test suite does not provision a gateway or select a compute driver.
418416
Set `OPENSHELL_BIN` to another executable to test a different prebuilt CLI.
417+
[tests/testkit/README.md](tests/testkit/README.md) covers gateway selection and
418+
cleanup.
419419

420-
The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without
421-
depending on how the gateway was installed or which driver is configured. It
422-
requires machine-readable gRPC status, creates a uniquely named detached
423-
sandbox with the configured default image, verifies the sandbox is `Ready` by finding its
424-
unique name in paginated JSON list output, executes `echo` with a run-specific
425-
marker, deletes the sandbox, and verifies that its name no longer appears.
426-
Driver suites enable the same profile
427-
instead of maintaining a separate smoke implementation. Sandbox lifecycle,
428-
label matrices, VM overlay, and TLS-key permission assertions remain regular
429-
E2E coverage.
430-
431-
Each invocation prints a ten-character run ID before creating resources.
432-
Conformance sandboxes use names such as `ct-<run-id>-01`. The runner tracks the
433-
exact name and uses it for cleanup; phase 1 does not add ownership labels.
434-
435-
The runner deletes owned resources after both success and failure. If the test
436-
process is interrupted before cleanup, locate leftovers without touching
437-
unrelated gateway state:
438-
439-
```shell
440-
openshell sandbox list --output json
441-
openshell sandbox delete <sandbox-name>
442-
```
443-
444-
Gateway-backed Rust E2E tasks run the CLI conformance Cargo tests against the
445-
configured gateway, then run any lane-specific Rust tests that still apply.
420+
Gateway-backed Rust E2E tasks run the conformance suite against the configured
421+
gateway, then run any lane-specific Rust tests that still apply.
446422
Run the Podman-backed Rust CLI e2e suite:
447423

448424
```shell

‎tasks/test.toml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -169,13 +169,12 @@ run = "OPENSHELL_TEST_CONTAINER_ENGINE=podman e2e/with-podman-gateway.sh cargo t
169169

170170
["e2e:testkit"]
171171
description = "Run the testkit suites in tests/suites/conformance against a Docker-backed gateway"
172-
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --exclude openshell-test-conformance-cli --test-threads 1 --no-fail-fast"
172+
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --test-threads 1 --no-fail-fast"
173173

174174
["e2e:rust:push"]
175-
description = "Run CLI conformance and policy e2e tests against a Docker gateway that pushes configuration"
176-
depends = ["e2e:conformance:build"]
175+
description = "Run conformance and policy e2e tests against a Docker gateway that pushes configuration"
177176
env = { OPENSHELL_CONFIG_DELIVERY_MODE = "push", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-config-push", OPENSHELL_E2E_DOCKER_TEST = "config_push live_policy_update policy_activation" }
178-
run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh"
177+
run = "e2e/rust/e2e-docker.sh"
179178

180179
["e2e:workload:build"]
181180
description = "Build the Docker E2E workload fixture"

‎tests/suites/conformance/Cargo.lock‎

Lines changed: 0 additions & 19 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/conformance/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33

44
[workspace]
55
resolver = "2"
6-
members = ["cli", "sandbox"]
6+
members = ["sandbox"]

‎tests/suites/conformance/cli/Cargo.toml‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/file_transfer.rs‎

Lines changed: 0 additions & 40 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/lifecycle.rs‎

Lines changed: 0 additions & 22 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/policy_advisor.rs‎

Lines changed: 0 additions & 37 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/smoke.rs‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

0 commit comments

Comments
 (0)