Repository navigation
test(e2e): port sandbox lifecycle and API tests from e2e/rust to conformance #5425
Description
User Story
As a maintainer retiring e2e/rust, I want the driver-agnostic sandbox lifecycle and API tests on testkit, so they run on every conformance row and their legacy files can be deleted.
Description
Port, merge or drop the tests below. All conformance destinations are under tests/suites/conformance/src/. Merges keep every assertion of the tests they combine.
| Legacy test | Action | Destination | Notes |
|---|---|---|---|
sandbox_lifecycle::sandbox_stop_start_preserves_workspace |
merge | persistent_workspace.rs (persistent_workspace_survives_stop_start) |
Already covered except "stop while a sandbox connect session is open", which moves to PR4's attach tests. |
sandbox_lifecycle::sandbox_can_be_deleted_while_stopped |
merge | lifecycle.rs (stopped_sandbox_can_be_deleted) |
Only the human output strings are lost. |
sandbox_lifecycle::sandbox_exec_outlives_startup_recovery_deadline |
port | lifecycle.rs |
SDK exec; about 35 s per row. |
sandbox_lifecycle::detached_canonical_main_nonzero_exit_reaches_error + detached_canonical_main_exit_zero_reaches_completed |
merge | lifecycle.rs |
One parameterized SDK test: exit 0 → Completed, exit 11 → Error with exit code 11, both after the sandbox is Ready. |
sandbox_lifecycle::canonical_main_and_exec_receive_declared_environment + canonical_tty_main_uses_sandbox_environment |
merge | lifecycle.rs (canonical_main_environment) |
SDK spec.environment and spec.tty. TTY case also checks HOME, USER, TERM. |
sandbox_lifecycle::detached_main_exit_during_provisioning_is_classified_as_workload_result |
port | lifecycle.rs |
Next to run_to_completion. |
sandbox_lifecycle::on_failure_policy_replaces_runtime_and_preserves_workspace |
stays in e2e/rust until #5507 |
lifecycle.rs after #5507 |
Ported, it failed on every VM row: the VM driver never starts the replacement (#5507). The legacy test runs only on the Docker and Kubernetes lanes, and an #[ignore] in conformance would drop that coverage. It moves to conformance once #5507 is fixed. |
ephemeral_cleanup::detached_ephemeral_{success,failure}_removes_sandbox_and_driver_resources |
split | lifecycle.rs |
One test, two sandboxes (exit 0 and 17), checks the sandbox record is gone. The "no driver resources left" half is rewritten in PR5, PR8 and PR9 under drivers/*. The legacy file stays in e2e/rust until those land. |
landlock::hard_requirement_accepts_enriched_device_path |
port | filesystem_policy.rs |
|
landlock::{best_effort,hard_requirement}_rejects_replaced_policy_{path,ancestor} (4 tests) |
merge | filesystem_policy.rs |
One table-driven test (2 compatibility modes × 2 replace modes); keep the 20 s exec timeout. |
core_dump_hardening::sandbox_processes_disable_core_dumps |
port | process.rs |
Check both ulimit -c and ulimit -Hc are 0. |
default_image::sandbox_from_default_image |
merge | lifecycle.rs (default_command_sandbox_is_listed_ready) |
Add an /etc/os-release check (ID, VERSION_ID). |
default_image::sandbox_from_explicit_nvidia_ubuntu_image |
port | image.rs |
"Explicit image is honored", using testkit::base_image(). |
sandbox_labels::sandbox_labels_are_stored_and_filterable |
port | labels.rs |
SDK labels and label_selector; 3 sandboxes; label values unique per run. |
sandbox_templates::sandbox_create_from_template_uses_reusable_workload |
port | templates.rs |
Also assert created_from_workload_template.name. |
sandbox_templates:: get-after-delete, duplicate name, missing template (3 tests) |
merge | templates.rs (template_crud_errors) |
Match SDK NotFound / AlreadyExists, not text. |
settings_management::settings_global_override_round_trip |
split | settings.rs |
(a) sandbox_setting_round_trip is ported. (b) The global override stays in e2e/rust until #5527 decides where destructive tests run: it changes a setting for every sandbox on the gateway, so it doesn't belong in conformance. |
upload_backpressure::large_upload_preserves_content_and_metadata_outside_workdir |
split | file_transfer.rs |
(a) Metadata checks (mode, symlinks, overwrite, destination outside workdir) join upload_download_round_trip; (b) large_upload_backpressure keeps count, checksum and no-retry. |
podman_oci_identity::podman_supervisor_runs_as_oci_identity_without_capabilities |
port | tests/suites/drivers/podman |
Only the part oci-image doesn't already check. |
docker_preflight::doctor_check_fails_without_docker + doctor_check_output_shows_docker_label |
merge | crates/openshell-cli/tests/doctor_integration.rs |
Hermetic: no Docker, no gateway. |
docker_preflight::doctor_check_passes_with_docker |
port | crates/openshell-cli/tests/doctor_integration.rs |
Fake docker on PATH; add a "no docker on PATH" case. |
docker_preflight::doctor_check_error_includes_guidance |
drop | — | The CLI doesn't print DOCKER_HOST when docker info fails; the legacy test passed only because its fake docker printed it. The crate test checks that the CLI shows docker's error. |
docker_corporate_proxy::docker_corporate_proxy_rejects_incoherent_configuration + vm_corporate_proxy::vm_corporate_proxy_rejects_incoherent_configuration |
drop → crate test | crates/openshell-gateway/tests/proxy_config_preflight.rs |
One hermetic test running openshell-gateway config preflight per driver. Needs approval in #5297. |
docker_corporate_proxy::proxy_settings_are_inserted_before_nested_driver_tables (unit test) |
stays in docker_corporate_proxy.rs |
— | The remaining corporate proxy tests still use the helper it covers. It goes with the file. |
edge_tunnel_e2e (2 tests) |
stays in e2e/rust |
later batch, not yet assigned | It checks the --enable-websocket-tunnel flag on real Docker and Podman gateways, which conformance can't vary. An in-process crate test checked the listener but not the flag wiring, so it was removed. |
internet_network_perf::benchmark_complete_internet_path |
drop | — | No assertions. #5472 tracks whether to bring a benchmark back. |
live_internet_traffic_perf::benchmark_live_internet_agent_traffic |
drop | — | Manual, needs the live Internet, and nothing referenced it. #5472 tracks whether to bring it back. |
Helpers this PR adds
- testkit:
CliOutput::message()(replaces the legacycreate_expect_failure). - None for global settings: conformance tests only read gateway-wide settings.
sandbox_setting_round_tripfails with a hint ifocsf_json_enabledis set globally.
Files deleted
core_dump_hardening.rs, default_image.rs, docker_preflight.rs, internet_network_perf.rs, landlock.rs, live_internet_traffic_perf.rs, podman_oci_identity.rs, sandbox_labels.rs, sandbox_templates.rs, upload_backpressure.rs.
Stay in e2e/rust: edge_tunnel_e2e.rs (later batch), ephemeral_cleanup.rs (until the driver halves in PR5, PR8 and PR9), settings_management.rs (until #5527).
Shared files: sandbox_lifecycle.rs keeps only the on-failure test after PR1 and PR4, and is deleted when that test moves to conformance after #5507; docker_corporate_proxy.rs by whichever of PR1 and PR6 lands last. vm_corporate_proxy.rs stays until PR9.
Defaults (change in review if needed)
upload_backpressureruns on every conformance row: large uploads are a public contract and the transport differs per driver. If it nears the runner's 2-minute limit, split it further.- The global settings test stays in
e2e/rustuntil test(testkit): separate destructive tests from non-destructive ones #5527 decides how destructive tests are marked and where they run. Conformance tests don't change gateway-wide state;sandbox_setting_round_triponly reads the globalocsf_json_enabled.
Context
- Part of test: retire the old e2e test system onto testkit #5281. Planned as PR1 in the test(e2e): decide which e2e/rust tests to port and in what order #5297 migration plan: test(e2e): decide which e2e/rust tests to port and in what order #5297 (comment)
- Wave 1: can start now.
- The six drops above wait on decision D in test(e2e): decide which e2e/rust tests to port and in what order #5297. Everything else can start now.
- PR2 also edits testkit's sandbox context; whichever lands second rebases.
Definition of Done
- Every test in the table is ported, merged, moved to a crate test, dropped, or kept in
e2e/rustas listed, with the drops approved in test(e2e): decide which e2e/rust tests to port and in what order #5297. - The listed files are deleted, and every reference to them in wrapper scripts,
tasks/*.tomland nextest config is removed. - Each ported test runs on at least the drivers its legacy lane covered (the conformance rows, plus the Podman row for
drivers/podman). - The PR body states which defaults above were applied.
Activity