Skip to content

Commit 41426f7

Browse files
test(conformance): delete the old CLI conformance suite
Nothing runs tests/suites/conformance/cli now that the host lanes run the testkit suite. Remove it from the conformance workspace and its lockfile, drop the e2e:testkit exclusion for it, and point the TESTING.md manual run at the `sandbox` package. e2e:rust:push also loses its dependency on e2e:conformance:build, which no longer exists, and the unused OPENSHELL_CONFORMANCE_BIN it set. crates/openshell-conformance stays for the features and drivers suites. Signed-off-by: Matthew Grossman <[email protected]>
1 parent 26f5662 commit 41426f7

9 files changed

Lines changed: 14 additions & 195 deletions

File tree

‎TESTING.md‎

Lines changed: 10 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -454,52 +454,28 @@ Run the Docker-backed Rust CLI e2e suite:
454454
mise run e2e:docker
455455
```
456456

457-
Run the portable CLI conformance suite against the gateway selected in your
458-
OpenShell CLI configuration:
457+
Run the conformance suite against the gateway selected in your OpenShell CLI
458+
configuration:
459459

460460
```shell
461461
cargo build --package openshell-cli
462462
OPENSHELL_BIN="$PWD/target/debug/openshell" \
463-
cargo test \
463+
cargo nextest run \
464464
--locked \
465465
--manifest-path tests/suites/conformance/Cargo.toml \
466-
--package openshell-test-conformance-cli \
467-
--no-fail-fast \
468-
-- \
469-
--test-threads=1 \
470-
--nocapture
466+
--package sandbox \
467+
--test-threads 1 \
468+
--no-fail-fast
471469
```
472470

473471
The gateway must already be installed, reachable, and selected before the tests
474472
start. The test suite does not provision a gateway or select a compute driver.
475473
Set `OPENSHELL_BIN` to another executable to test a different prebuilt CLI.
474+
[tests/testkit/README.md](tests/testkit/README.md) covers gateway selection and
475+
cleanup.
476476

477-
The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without
478-
depending on how the gateway was installed or which driver is configured. It
479-
requires machine-readable gRPC status, creates a uniquely named detached
480-
sandbox with the configured default image, verifies the sandbox is `Ready` by finding its
481-
unique name in paginated JSON list output, executes `echo` with a run-specific
482-
marker, deletes the sandbox, and verifies that its name no longer appears.
483-
Driver suites enable the same profile
484-
instead of maintaining a separate smoke implementation. Sandbox lifecycle,
485-
label matrices, VM overlay, and TLS-key permission assertions remain regular
486-
E2E coverage.
487-
488-
Each invocation prints a ten-character run ID before creating resources.
489-
Conformance sandboxes use names such as `ct-<run-id>-01`. The runner tracks the
490-
exact name and uses it for cleanup; phase 1 does not add ownership labels.
491-
492-
The runner deletes owned resources after both success and failure. If the test
493-
process is interrupted before cleanup, locate leftovers without touching
494-
unrelated gateway state:
495-
496-
```shell
497-
openshell sandbox list --output json
498-
openshell sandbox delete <sandbox-name>
499-
```
500-
501-
Gateway-backed Rust E2E tasks run the CLI conformance Cargo tests against the
502-
configured gateway, then run any lane-specific Rust tests that still apply.
477+
Gateway-backed Rust E2E tasks run the conformance suite against the configured
478+
gateway, then run any lane-specific Rust tests that still apply.
503479
Run the Podman-backed Rust CLI e2e suite:
504480

505481
```shell

‎tasks/test.toml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -169,13 +169,12 @@ run = "OPENSHELL_TEST_CONTAINER_ENGINE=podman e2e/with-podman-gateway.sh cargo t
169169

170170
["e2e:testkit"]
171171
description = "Run the testkit suites in tests/suites/conformance against a Docker-backed gateway"
172-
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --exclude openshell-test-conformance-cli --test-threads 1 --no-fail-fast"
172+
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --test-threads 1 --no-fail-fast"
173173

174174
["e2e:rust:push"]
175-
description = "Run CLI conformance and policy e2e tests against a Docker gateway that pushes configuration"
176-
depends = ["e2e:conformance:build"]
175+
description = "Run conformance and policy e2e tests against a Docker gateway that pushes configuration"
177176
env = { OPENSHELL_CONFIG_DELIVERY_MODE = "push", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-config-push", OPENSHELL_E2E_DOCKER_TEST = "config_push live_policy_update policy_activation" }
178-
run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh"
177+
run = "e2e/rust/e2e-docker.sh"
179178

180179
["e2e:workload:build"]
181180
description = "Build the Docker E2E workload fixture"

‎tests/suites/conformance/Cargo.lock‎

Lines changed: 0 additions & 19 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/conformance/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33

44
[workspace]
55
resolver = "2"
6-
members = ["cli", "sandbox"]
6+
members = ["sandbox"]

‎tests/suites/conformance/cli/Cargo.toml‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/file_transfer.rs‎

Lines changed: 0 additions & 40 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/lifecycle.rs‎

Lines changed: 0 additions & 22 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/policy_advisor.rs‎

Lines changed: 0 additions & 37 deletions
This file was deleted.

‎tests/suites/conformance/cli/tests/smoke.rs‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

0 commit comments

Comments
 (0)