Skip to content

Commit 27d3cb5

Browse files
committed
test(conformance): split smoke capabilities
Signed-off-by: Evan Lezar <[email protected]>
1 parent 6f00d5c commit 27d3cb5

7 files changed

Lines changed: 240 additions & 66 deletions

File tree

‎TESTING.md‎

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -181,11 +181,13 @@ lifecycle management, output parsing, and cleanup.
181181
Suites:
182182

183183
- Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
184-
- CLI conformance (`openshell-conformance`) - named scenarios for lifecycle,
185-
mechanistic drafts, and the sandbox-local API, including agent-authored
186-
permission requests. Driver E2E wrappers run every scenario. The
187-
installed-artifact conformance suite runs all scenarios and offers a focused
188-
`policy-advisor` testsuite for manual integration runs.
184+
- CLI conformance (`tests/suites/conformance`) - installed-artifact tests of
185+
portable public CLI behavior. Smoke coverage is split into independent
186+
control-plane and exec tests so each driver runs only the capabilities it
187+
implements. Scenario implementations live in `openshell-conformance`; the
188+
standalone `openshell-conformance` binary remains a compatibility runner for
189+
driver E2E wrappers. See the [suite README](tests/suites/conformance/README.md)
190+
for scope and selection guidance.
189191
- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
190192
`e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
191193
the selected deployment.
@@ -222,14 +224,14 @@ starts. The task does not provision a gateway or select a compute driver. Set
222224
`OPENSHELL_BIN` to test a prebuilt CLI; otherwise, the task builds the CLI from
223225
the current checkout.
224226

225-
The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without
226-
depending on how the gateway was installed or which driver is configured. It
227-
requires machine-readable gRPC status, creates a uniquely named detached
228-
sandbox with the configured default image, verifies the sandbox is `Ready` by finding its
229-
unique name in paginated JSON list output, executes `echo` with a run-specific
230-
marker, deletes the sandbox, and verifies that its name no longer appears.
231-
Driver suites enable the same profile
232-
instead of maintaining a separate smoke implementation. Sandbox lifecycle,
227+
The smoke contract verifies the CLI-to-gateway-to-driver path without depending
228+
on how the gateway was installed. Its control-plane test requires
229+
machine-readable status, creates a uniquely named detached sandbox, verifies it
230+
is `Ready` through get and paginated list output, deletes it, and verifies its
231+
name no longer appears. The exec test creates its own sandbox and checks
232+
`sandbox exec` with a run-specific marker. Drivers without exec support run the
233+
control-plane test alone. Runtimes that require an explicit workload set
234+
`OPENSHELL_CONFORMANCE_SMOKE_COMMAND` to a JSON string array. Sandbox lifecycle,
233235
label matrices, VM overlay, and TLS-key permission assertions remain regular
234236
E2E coverage.
235237

‎architecture/build.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -269,8 +269,11 @@ and retain that provenance with the local entry; mutable tags are used only
269269
for explicit publication.
270270

271271
CLI conformance runs after target provisioning and operates only through the
272-
configured OpenShell CLI. The smoke scenario verifies the black-box sandbox
273-
lifecycle by creating, inspecting, executing in, and deleting a sandbox.
272+
configured OpenShell CLI. The archive suite defines the installed-artifact test
273+
boundary, while `openshell-conformance` provides reusable scenarios and a
274+
standalone compatibility runner. Smoke coverage separates control-plane
275+
create/get/list/delete behavior from exec behavior so a runtime can run the
276+
portable subset matching its advertised capabilities.
274277
Feature suites use the same disposable guest but may provision isolated
275278
dependencies after installation. The Keycloak provider-refresh suite starts a
276279
guest-local Keycloak realm and verifies a successful OAuth refresh followed by

‎crates/openshell-conformance/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ use self::executor::{CliExecutionError, CliExecutor, ProcessCli};
2525

2626
pub use scenarios::{
2727
MECHANISTIC_PROPOSAL_SCENARIO, NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO,
28-
SANDBOX_LIFECYCLE_SCENARIO, SMOKE_SCENARIO,
28+
SANDBOX_LIFECYCLE_SCENARIO, SMOKE_CONTROL_PLANE_SCENARIO, SMOKE_EXEC_SCENARIO, SMOKE_SCENARIO,
2929
};
3030

3131
/// An installed conformance scenario.

‎crates/openshell-conformance/src/scenarios/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,4 @@ pub use policy_behavior::{
1111
MECHANISTIC_PROPOSAL_SCENARIO, NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO,
1212
};
1313
pub use sandbox_lifecycle::SANDBOX_LIFECYCLE_SCENARIO;
14-
pub use smoke::SMOKE_SCENARIO;
14+
pub use smoke::{SMOKE_CONTROL_PLANE_SCENARIO, SMOKE_EXEC_SCENARIO, SMOKE_SCENARIO};

‎crates/openshell-conformance/src/scenarios/smoke.rs‎

Lines changed: 159 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
// SPDX-License-Identifier: Apache-2.0
33

4-
//! Portable phase-1 CLI conformance scenario.
4+
//! Portable smoke conformance scenarios.
55
6+
use std::ffi::OsString;
67
use std::time::{Duration, Instant};
78

89
use crate::{OpenShellRunner, STATUS_TIMEOUT, Scenario, ScenarioFuture};
@@ -31,15 +32,40 @@ struct SandboxListPage {
3132
/// Certify status -> create -> list Ready -> exec -> delete -> list empty.
3233
pub const SMOKE_SCENARIO: Scenario = Scenario {
3334
name: "smoke",
34-
description: "Create, inspect, execute in, and delete a base sandbox.",
35+
description: "Run the control-plane and exec smoke scenarios.",
3536
run: run_smoke,
3637
};
3738

39+
/// Certify status -> create -> get/list Ready -> delete -> list empty.
40+
pub const SMOKE_CONTROL_PLANE_SCENARIO: Scenario = Scenario {
41+
name: "smoke-control-plane",
42+
description: "Create, inspect, and delete a sandbox without using sandbox exec.",
43+
run: run_smoke_control_plane,
44+
};
45+
46+
/// Certify create -> exec -> delete for drivers that support interactive exec.
47+
pub const SMOKE_EXEC_SCENARIO: Scenario = Scenario {
48+
name: "smoke-exec",
49+
description: "Create a sandbox, execute a command in it, and delete it.",
50+
run: run_smoke_exec,
51+
};
52+
3853
fn run_smoke(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
39-
Box::pin(async move { run_smoke_inner(runner).await })
54+
Box::pin(async move {
55+
run_smoke_control_plane_inner(runner).await?;
56+
run_smoke_exec_inner(runner).await
57+
})
58+
}
59+
60+
fn run_smoke_control_plane(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
61+
Box::pin(async move { run_smoke_control_plane_inner(runner).await })
4062
}
4163

42-
async fn run_smoke_inner(runner: &mut OpenShellRunner) -> Result<(), String> {
64+
fn run_smoke_exec(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
65+
Box::pin(async move { run_smoke_exec_inner(runner).await })
66+
}
67+
68+
async fn run_smoke_control_plane_inner(runner: &mut OpenShellRunner) -> Result<(), String> {
4369
let status = runner
4470
.step("status")
4571
.description("openshell status succeeds")
@@ -49,43 +75,18 @@ async fn run_smoke_inner(runner: &mut OpenShellRunner) -> Result<(), String> {
4975
.map_err(|error| error.to_string())?;
5076
status.require_success()?;
5177

52-
let sandbox_name = format!("ct-{}-01", runner.id());
53-
runner.track_sandbox(&sandbox_name);
54-
let create = runner
55-
.step("create")
56-
.description("sandbox creation succeeds")
57-
.with_timeout(CREATE_TIMEOUT)
58-
.run(&["sandbox", "create", "--name", &sandbox_name, "--detach"])
59-
.await
60-
.map_err(|error| error.to_string())?;
61-
create.require_success()?;
62-
63-
let get = runner
64-
.step("get-ready")
65-
.description(format!("sandbox '{sandbox_name}' can be retrieved"))
66-
.with_timeout(LIST_ATTEMPT_TIMEOUT)
67-
.run(&["sandbox", "get", &sandbox_name, "--output", "json"])
68-
.await
69-
.map_err(|error| error.to_string())?;
70-
get.require_success()?;
78+
let sandbox_name = format!("ct-{}-cp", runner.id());
79+
create_sandbox(runner, &sandbox_name, "create").await?;
80+
check_sandbox_ready(runner, &sandbox_name).await?;
81+
check_sandbox_listed(runner, &sandbox_name).await?;
7182

72-
let sandbox = get
73-
.json::<SandboxListEntry>()
74-
.map_err(|error| error.to_string())?;
75-
if sandbox.name != sandbox_name {
76-
return Err(format!(
77-
"sandbox get returned {:?}; expected sandbox '{sandbox_name}'",
78-
sandbox.name
79-
));
80-
}
81-
if sandbox.phase != "Ready" {
82-
return Err(format!(
83-
"sandbox '{sandbox_name}' is in phase {:?}; expected Ready",
84-
sandbox.phase
85-
));
86-
}
83+
delete_sandbox(runner, &sandbox_name, "delete").await
84+
}
8785

88-
check_sandbox_listed(runner, &sandbox_name).await?;
86+
async fn run_smoke_exec_inner(runner: &mut OpenShellRunner) -> Result<(), String> {
87+
let sandbox_name = format!("ct-{}-ex", runner.id());
88+
create_sandbox(runner, &sandbox_name, "create").await?;
89+
check_sandbox_ready(runner, &sandbox_name).await?;
8990

9091
let marker = format!("openshell-conformance-{}", runner.id());
9192
let exec = runner
@@ -110,17 +111,103 @@ async fn run_smoke_inner(runner: &mut OpenShellRunner) -> Result<(), String> {
110111
return Err(exec.failure_diagnostic(&format!("stdout is exactly {expected_stdout:?}")));
111112
}
112113

114+
delete_sandbox(runner, &sandbox_name, "delete").await
115+
}
116+
117+
async fn create_sandbox(
118+
runner: &mut OpenShellRunner,
119+
sandbox_name: &str,
120+
step: &str,
121+
) -> Result<(), String> {
122+
runner.track_sandbox(sandbox_name);
123+
let mut args = vec![
124+
"sandbox".to_string(),
125+
"create".to_string(),
126+
"--name".to_string(),
127+
sandbox_name.to_string(),
128+
"--detach".to_string(),
129+
];
130+
if let Some(command) = smoke_command()? {
131+
args.push("--".to_string());
132+
args.extend(command);
133+
}
134+
let args = args.iter().map(String::as_str).collect::<Vec<_>>();
135+
let create = runner
136+
.step(step)
137+
.description("sandbox creation succeeds")
138+
.with_timeout(CREATE_TIMEOUT)
139+
.run(&args)
140+
.await
141+
.map_err(|error| error.to_string())?;
142+
create.require_success()
143+
}
144+
145+
fn smoke_command() -> Result<Option<Vec<String>>, String> {
146+
parse_smoke_command(std::env::var_os("OPENSHELL_CONFORMANCE_SMOKE_COMMAND"))
147+
}
148+
149+
fn parse_smoke_command(value: Option<OsString>) -> Result<Option<Vec<String>>, String> {
150+
let Some(value) = value else {
151+
return Ok(None);
152+
};
153+
let value = value.into_string().map_err(|_| {
154+
"OPENSHELL_CONFORMANCE_SMOKE_COMMAND must contain valid Unicode JSON".to_string()
155+
})?;
156+
let command = serde_json::from_str::<Vec<String>>(&value).map_err(|error| {
157+
format!("OPENSHELL_CONFORMANCE_SMOKE_COMMAND must be a JSON string array: {error}")
158+
})?;
159+
if command.first().is_none_or(String::is_empty) {
160+
return Err(
161+
"OPENSHELL_CONFORMANCE_SMOKE_COMMAND must contain a non-empty executable".to_string(),
162+
);
163+
}
164+
Ok(Some(command))
165+
}
166+
167+
async fn delete_sandbox(
168+
runner: &mut OpenShellRunner,
169+
sandbox_name: &str,
170+
step: &str,
171+
) -> Result<(), String> {
113172
let delete = runner
114-
.step("delete")
173+
.step(step)
115174
.description("sandbox deletion succeeds")
116175
.with_timeout(DELETE_TIMEOUT)
117-
.run(&["sandbox", "delete", &sandbox_name])
176+
.run(&["sandbox", "delete", sandbox_name])
118177
.await
119178
.map_err(|error| error.to_string())?;
120179
delete.require_success()?;
121180

122-
check_empty_list(runner, &sandbox_name).await?;
123-
runner.forget_sandbox(&sandbox_name);
181+
check_empty_list(runner, sandbox_name).await?;
182+
runner.forget_sandbox(sandbox_name);
183+
Ok(())
184+
}
185+
186+
async fn check_sandbox_ready(runner: &OpenShellRunner, sandbox_name: &str) -> Result<(), String> {
187+
let get = runner
188+
.step("get-ready")
189+
.description(format!("sandbox '{sandbox_name}' can be retrieved"))
190+
.with_timeout(LIST_ATTEMPT_TIMEOUT)
191+
.run(&["sandbox", "get", sandbox_name, "--output", "json"])
192+
.await
193+
.map_err(|error| error.to_string())?;
194+
get.require_success()?;
195+
196+
let sandbox = get
197+
.json::<SandboxListEntry>()
198+
.map_err(|error| error.to_string())?;
199+
if sandbox.name != sandbox_name {
200+
return Err(format!(
201+
"sandbox get returned {:?}; expected sandbox '{sandbox_name}'",
202+
sandbox.name
203+
));
204+
}
205+
if sandbox.phase != "Ready" {
206+
return Err(format!(
207+
"sandbox '{sandbox_name}' is in phase {:?}; expected Ready",
208+
sandbox.phase
209+
));
210+
}
124211
Ok(())
125212
}
126213

@@ -204,3 +291,32 @@ async fn find_sandbox(
204291
.ok_or_else(|| "sandbox list page counter overflowed".to_string())?;
205292
}
206293
}
294+
295+
#[cfg(test)]
296+
mod tests {
297+
use super::*;
298+
299+
#[test]
300+
fn smoke_command_is_optional() {
301+
assert_eq!(parse_smoke_command(None).unwrap(), None);
302+
}
303+
304+
#[test]
305+
fn smoke_command_parses_a_json_string_array() {
306+
assert_eq!(
307+
parse_smoke_command(Some(OsString::from(r#"["cmd.exe","/c","exit","0"]"#))).unwrap(),
308+
Some(vec![
309+
"cmd.exe".into(),
310+
"/c".into(),
311+
"exit".into(),
312+
"0".into()
313+
])
314+
);
315+
}
316+
317+
#[test]
318+
fn smoke_command_rejects_an_empty_executable() {
319+
let error = parse_smoke_command(Some(OsString::from(r#"[""]"#))).unwrap_err();
320+
assert!(error.contains("non-empty executable"));
321+
}
322+
}

‎tests/suites/conformance/README.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
<!--
2+
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
-->
5+
6+
# CLI conformance suite
7+
8+
This workspace verifies that an installed `openshell` CLI and a selected gateway
9+
implement portable, public behavior. Tests treat the CLI as a black box. They do
10+
not inspect driver internals or replace driver-specific qualification.
11+
12+
The suite separates portable contracts by capability. A test environment should
13+
run every test supported by its compute driver instead of treating one broad
14+
smoke test as the portability boundary. In particular:
15+
16+
- `smoke::control_plane` covers status, create, get, list, and delete. It does
17+
not require `sandbox exec`.
18+
- `smoke::exec` covers `sandbox exec` and runs only where the driver supports
19+
that operation.
20+
- `lifecycle` and the policy advisor tests have additional runtime requirements
21+
documented in their source modules.
22+
23+
The scenario implementations live in `crates/openshell-conformance` so the
24+
archive tests and the standalone `openshell-conformance` compatibility runner
25+
share assertions and cleanup. New installed-artifact CI should use this test
26+
workspace and select tests with nextest filters.
27+
28+
Set `OPENSHELL_BIN` to the candidate CLI. The selected gateway must already be
29+
reachable. Runtimes that require an explicit sandbox workload can set
30+
`OPENSHELL_CONFORMANCE_SMOKE_COMMAND` to a JSON string array. For example:
31+
32+
```shell
33+
export OPENSHELL_CONFORMANCE_SMOKE_COMMAND='["sleep","infinity"]'
34+
cargo nextest run \
35+
--manifest-path tests/suites/conformance/Cargo.toml \
36+
-E 'binary(smoke) & test(=control_plane)'
37+
```
38+
39+
The runner owns only resources named for its generated run ID and attempts to
40+
delete them after success or failure. Tests must not depend on unrelated gateway
41+
state or delete resources they did not create.

0 commit comments

Comments
 (0)