Skip to content

Commit 20357a9

Browse files
test(tmachine): prepare runner environments for the e2e/rust migration (NVIDIA#5277)
* test(tmachine): load the Python workload image and set the k3s host gateway Every installer loads openshell/e2e-python:dev next to the runtime images, so tests can create sandboxes that run python3, curl, git or openssl. The k3s installers set the chart's server.hostGatewayIP to the node's InternalIP. Tests run on the node, so host.openshell.internal now resolves to it on k3s and k3s-ha. Refs NVIDIA#5275 Signed-off-by: Matthew Grossman <[email protected]> * test(testkit): add python_image() for the Python workload image OPENSHELL_TEST_PYTHON_IMAGE overrides the default openshell/e2e-python:dev, next to OPENSHELL_TEST_BASE_IMAGE. Refs NVIDIA#5275 Signed-off-by: Matthew Grossman <[email protected]> * test(runner): honor family nextest config and add --test-threads The runner passes its openshell profile as a nextest tool config and the bundle carries tests/suites/<family>/.config/nextest.toml into the remapped workspace, so a family can set per-test timeouts and test groups, or override the profile. --test-threads defaults to 1, which keeps the serial --no-capture run. Refs NVIDIA#5275 Signed-off-by: Matthew Grossman <[email protected]> * test(features): check host.openshell.internal from a Python sandbox The host-gateway suite starts a server on the host and checks that a sandbox from the Python workload image reaches it through host.openshell.internal. CI runs it on Docker rootful, Podman rootful and rootless, k3s and k3s-ha. Closes NVIDIA#5275 Signed-off-by: Matthew Grossman <[email protected]> * test(tmachine): use the k3s node's IPv4 address for the host gateway The node can list an IPv6 InternalIP after the IPv4 one, which made the address discovery retry until it failed. Signed-off-by: Matthew Grossman <[email protected]> * test(features): run the host-gateway client with sandbox exec On k3s-ha, attaching to a main process that already finished can reach a gateway replica without the sandbox's session and fail with "sandbox is not ready". Run the client in a Ready sandbox instead. Signed-off-by: Matthew Grossman <[email protected]> * Revert "test(runner): honor family nextest config and add --test-threads" This reverts commit 8441388. Signed-off-by: Matthew Grossman <[email protected]> * test(features): use the system Python for the host-gateway client and trim exports The client needs only the standard library, so run /usr/bin/python3 instead of the image's uv virtual environment and let the policy name its real path. Keep the new image constants private to testkit, and fail the k3s installer if the cluster ever has more than one node, since the host gateway address is the node the tests run on. Refs NVIDIA#5275 Signed-off-by: Matthew Grossman <[email protected]> * refactor(testkit): stop exporting environment variable names Tests read the image, CLI binary and gateway settings through functions such as base_image() and python_image(). Nothing outside testkit used the variable-name constants, so keep them private. Refs NVIDIA#5275 Signed-off-by: Matthew Grossman <[email protected]> --------- Signed-off-by: Matthew Grossman <[email protected]>
1 parent 11fa5f0 commit 20357a9

15 files changed

Lines changed: 341 additions & 17 deletions

File tree

‎.github/workflows/e2e-suites.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,11 @@ jobs:
7777
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
7878
{"environment":"fedora-podman-rootful","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
7979
{"environment":"fedora-podman-rootless","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
80+
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"host-gateway","gateway":"openshell","runner":"ubuntu-24.04"},
81+
{"environment":"fedora-podman-rootful","installer":"rpm","family":"features","package":"host-gateway","gateway":"openshell","runner":"ubuntu-24.04"},
82+
{"environment":"fedora-podman-rootless","installer":"rpm","family":"features","package":"host-gateway","gateway":"openshell","runner":"ubuntu-24.04"},
83+
{"environment":"ubuntu-k3s","installer":"k3s","family":"features","package":"host-gateway","gateway":"tmachine","runner":"ubuntu-24.04"},
84+
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"features","package":"host-gateway","gateway":"tmachine","runner":"ubuntu-24.04"},
8085
{"environment":"fedora-podman-rootful-keycloak","installer":"rpm","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"},
8186
{"environment":"fedora-podman-rootless-keycloak","installer":"rpm","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"},
8287
{"environment":"fedora-podman-rootful-userns-default","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"},

‎TESTING.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -586,6 +586,13 @@ mise run e2e:oci-image:podman
586586
Both tasks set `OPENSHELL_TEST_CONTAINER_ENGINE`, the command that builds test
587587
images into the gateway's image store.
588588

589+
The `host-gateway` feature suite (`tests/suites/features/host-gateway`) checks
590+
that a sandbox from the Python test image reaches a server the test runs on the
591+
host through `host.openshell.internal`. CI runs it on Docker rootful, Podman
592+
rootful and rootless, and the `k3s` and `k3s-ha` installers. The tests run on
593+
the K3s node, so the K3s installers set the chart's `server.hostGatewayIP` to
594+
the node's address.
595+
589596
Run the VM-backed Rust CLI e2e suite:
590597

591598
```shell

‎tests/suites/features/Cargo.lock‎

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/features/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33

44
[workspace]
55
resolver = "2"
6-
members = ["keycloak-provider-refresh", "kubernetes-updates", "oci-image"]
6+
members = ["host-gateway", "keycloak-provider-refresh", "kubernetes-updates", "oci-image"]
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
[package]
5+
name = "host-gateway"
6+
description = "Features suite: sandboxes reach a host server through host.openshell.internal"
7+
version = "0.0.0"
8+
edition = "2024"
9+
rust-version = "1.94"
10+
license = "Apache-2.0"
11+
repository = "https://github.com/NVIDIA/OpenShell"
12+
publish = false
13+
14+
[dependencies]
15+
openshell-testkit = { path = "../../../testkit/testkit" }
16+
tempfile = "3"
17+
tokio = { version = "1.43", features = ["io-util", "net", "rt"] }
Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//! `host.openshell.internal` names the host from inside a sandbox, on every
5+
//! compute driver.
6+
//!
7+
//! The test starts an HTTP server where the tests run, which is the host the
8+
//! gateway runs its sandboxes on, and runs a client in a sandbox from the
9+
//! Python test image ([`testkit::python_image`]) that requests a page from that
10+
//! server through the name.
11+
12+
use testkit::{Context, Error, Result, ResultExt as _};
13+
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
14+
use tokio::net::{TcpListener, TcpStream};
15+
use tokio::task::JoinHandle;
16+
17+
/// The image's system Python, which needs nothing beyond its standard library.
18+
/// The policy names its real path, `/usr/bin/python3.*`, because a rule matches
19+
/// the executable behind a symlink.
20+
const PYTHON: &str = "/usr/bin/python3";
21+
22+
/// Requests `/` from `host.openshell.internal` on the port in `argv[1]` and
23+
/// prints the response. A failed connection is retried for 20 seconds, so a
24+
/// slow start doesn't fail the test, and reported on stderr.
25+
const CLIENT: &str = r#"
26+
import socket, sys, time
27+
28+
host, port = "host.openshell.internal", int(sys.argv[1])
29+
deadline = time.monotonic() + 20
30+
while True:
31+
try:
32+
conn = socket.create_connection((host, port), timeout=10)
33+
break
34+
except OSError as error:
35+
print(f"connect {host}:{port}: {error}", file=sys.stderr, flush=True)
36+
if time.monotonic() > deadline:
37+
sys.exit(1)
38+
time.sleep(1)
39+
with conn:
40+
conn.sendall(f"GET / HTTP/1.1\r\nHost: {host}:{port}\r\nConnection: close\r\n\r\n".encode())
41+
response = b""
42+
while chunk := conn.recv(4096):
43+
response += chunk
44+
print(response.decode(errors="replace"))
45+
"#;
46+
47+
/// A sandbox from the Python test image resolves `host.openshell.internal`
48+
/// and gets a response from a server the test runs on the host. The default
49+
/// policy denies all network access, so the test passes one that lets only the
50+
/// image's Python reach the server's port.
51+
#[testkit::test]
52+
async fn python_sandbox_reaches_host_server(cx: &mut Context) -> testkit::Result {
53+
let marker = format!("host-gateway-{}", cx.run_id());
54+
let server = HostServer::start(marker.clone()).await?;
55+
let policy = PolicyFile::new(&policy(server.port))?;
56+
let port = server.port.to_string();
57+
let image = testkit::python_image();
58+
59+
// The client runs through `sandbox exec` in a Ready sandbox, not as the
60+
// main process. A gateway with several replicas (k3s-ha) can refuse to
61+
// attach to a main process that already finished with "sandbox is not
62+
// ready" when the request reaches a replica that doesn't hold the
63+
// sandbox's session.
64+
let name = cx.name("hg");
65+
cx.track(&name);
66+
cx.cli([
67+
"sandbox",
68+
"create",
69+
"--name",
70+
&name,
71+
"--from",
72+
&image,
73+
"--policy",
74+
policy.path()?,
75+
"--detach",
76+
"--no-tty",
77+
"--",
78+
"sh",
79+
"-c",
80+
"exec sleep infinity",
81+
])
82+
.await?
83+
.success()
84+
.context(format!("create a sandbox from {image}"))?;
85+
cx.adopt(&name).await?.wait_ready().await?;
86+
87+
let client = cx
88+
.cli([
89+
"sandbox", "exec", "--name", &name, "--no-tty", "--", PYTHON, "-c", CLIENT, &port,
90+
])
91+
.await?
92+
.success()
93+
.context("run the client in the sandbox")?;
94+
testkit::ensure!(
95+
client.stdout.contains(&marker),
96+
"the client did not get the host server's response:\n{}{}",
97+
client.stdout,
98+
client.stderr
99+
);
100+
Ok(())
101+
}
102+
103+
/// A sandbox policy that allows the image's system Python ([`PYTHON`]) to
104+
/// connect to `host.openshell.internal` on `port`. The endpoint names only the
105+
/// host and port, as the policy docs describe, so the driver's address for the
106+
/// name is what makes it reachable, not an `allowed_ips` range.
107+
fn policy(port: u16) -> String {
108+
format!(
109+
r#"version: 1
110+
111+
filesystem_policy:
112+
include_workdir: true
113+
read_only: [/usr, /lib, /lib64, /proc, /dev/urandom, /etc]
114+
read_write: [/sandbox, /tmp, /dev/null]
115+
landlock:
116+
compatibility: best_effort
117+
118+
process:
119+
run_as_user: sandbox
120+
run_as_group: sandbox
121+
122+
network_policies:
123+
host_server:
124+
name: host_server
125+
endpoints:
126+
- host: host.openshell.internal
127+
port: {port}
128+
binaries:
129+
- path: /usr/bin/python3.*
130+
"#
131+
)
132+
}
133+
134+
/// An HTTP server on the host that answers every request with `body`. It
135+
/// stops when dropped.
136+
struct HostServer {
137+
port: u16,
138+
task: JoinHandle<()>,
139+
}
140+
141+
impl HostServer {
142+
/// Listen on every host address, since drivers map
143+
/// `host.openshell.internal` to different ones.
144+
async fn start(body: String) -> Result<Self> {
145+
let listener = TcpListener::bind(("0.0.0.0", 0))
146+
.await
147+
.map_err(|error| Error::from_error(&error))
148+
.context("bind the host server")?;
149+
let port = listener
150+
.local_addr()
151+
.map_err(|error| Error::from_error(&error))
152+
.context("read the host server's address")?
153+
.port();
154+
let task = tokio::spawn(async move {
155+
while let Ok((stream, _)) = listener.accept().await {
156+
tokio::spawn(respond(stream, body.clone()));
157+
}
158+
});
159+
Ok(Self { port, task })
160+
}
161+
}
162+
163+
impl Drop for HostServer {
164+
fn drop(&mut self) {
165+
self.task.abort();
166+
}
167+
}
168+
169+
/// Read one request's headers from `stream` and answer with `body`.
170+
async fn respond(mut stream: TcpStream, body: String) {
171+
let _ = stream.set_nodelay(true);
172+
let mut request = Vec::new();
173+
let mut buf = [0_u8; 1024];
174+
while !request.windows(4).any(|window| window == b"\r\n\r\n") {
175+
match stream.read(&mut buf).await {
176+
Ok(read) if read > 0 && request.len() < 64 * 1024 => {
177+
request.extend_from_slice(&buf[..read]);
178+
}
179+
_ => return,
180+
}
181+
}
182+
let response = format!(
183+
"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
184+
body.len()
185+
);
186+
let _ = stream.write_all(response.as_bytes()).await;
187+
let _ = stream.shutdown().await;
188+
}
189+
190+
/// A sandbox policy written to a temporary file for `sandbox create --policy`.
191+
struct PolicyFile(tempfile::NamedTempFile);
192+
193+
impl PolicyFile {
194+
fn new(contents: &str) -> Result<Self> {
195+
let file = tempfile::NamedTempFile::new()
196+
.map_err(|error| Error::from_error(&error))
197+
.context("create policy file")?;
198+
std::fs::write(file.path(), contents)
199+
.map_err(|error| Error::from_error(&error))
200+
.context(format!("write {}", file.path().display()))?;
201+
Ok(Self(file))
202+
}
203+
204+
fn path(&self) -> Result<&str> {
205+
let path = self.0.path();
206+
path.to_str()
207+
.ok_or_else(|| Error::msg(format!("{} is not UTF-8", path.display())))
208+
}
209+
}

‎tests/testkit/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,10 @@ fails. A killed test process skips cleanup, so names are unique to the run.
7575
compute driver. Driver suites call it first.
7676
- `testkit::base_image()` for the image to create sandboxes from or build on:
7777
`OPENSHELL_TEST_BASE_IMAGE`, or OpenShell's default sandbox image.
78+
- `testkit::python_image()` for sandboxes that run `python3`, curl, git, or
79+
openssl: `OPENSHELL_TEST_PYTHON_IMAGE`, or `openshell/e2e-python:dev`
80+
(`e2e/python/Dockerfile.workload`). The tmachine installers load it into
81+
every environment; `mise run e2e:workload:build` builds it locally.
7882
- `cx.cli([...])` to run the installed CLI against the same gateway, with
7983
`success()`, `failure()`, and `json()` on the output. `cx.cli_handle()`
8084
returns that CLI for code that can't borrow `cx`, such as a cleanup hook.

‎tests/testkit/testkit/src/cli.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ use crate::gateway::{ENDPOINT_ENV, GATEWAY_ENV, Gateway, WORKSPACE_ENV};
2121
/// How long one CLI invocation may take.
2222
pub const CLI_TIMEOUT: Duration = Duration::from_mins(5);
2323
/// The CLI binary to run. Defaults to `openshell` on PATH.
24-
pub const BIN_ENV: &str = "OPENSHELL_BIN";
24+
pub(crate) const BIN_ENV: &str = "OPENSHELL_BIN";
2525

2626
/// Runs the CLI against the gateway and workspace the tests use.
2727
#[derive(Clone, Debug)]

‎tests/testkit/testkit/src/gateway.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,9 @@ use openshell_sdk::{ClientConfig, ClientIdentity, Health, OpenShellClient, Servi
2121

2222
use crate::error::{Error, Result, ResultExt};
2323

24-
pub const ENDPOINT_ENV: &str = "OPENSHELL_GATEWAY_ENDPOINT";
25-
pub const GATEWAY_ENV: &str = "OPENSHELL_GATEWAY";
26-
pub const WORKSPACE_ENV: &str = "OPENSHELL_WORKSPACE";
24+
pub(crate) const ENDPOINT_ENV: &str = "OPENSHELL_GATEWAY_ENDPOINT";
25+
pub(crate) const GATEWAY_ENV: &str = "OPENSHELL_GATEWAY";
26+
pub(crate) const WORKSPACE_ENV: &str = "OPENSHELL_WORKSPACE";
2727
const DEFAULT_WORKSPACE: &str = "default";
2828

2929
// Mirrors the SDK's endpoint settings for channels testkit builds itself.

‎tests/testkit/testkit/src/image.rs‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,35 @@
11
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
// SPDX-License-Identifier: Apache-2.0
33

4-
//! The base image tests create sandboxes from or build their images on.
4+
//! The images tests create sandboxes from or build their images on.
55
66
/// Overrides [`DEFAULT_BASE_IMAGE`], e.g. with a mirror.
7-
pub const BASE_IMAGE_ENV: &str = "OPENSHELL_TEST_BASE_IMAGE";
7+
const BASE_IMAGE_ENV: &str = "OPENSHELL_TEST_BASE_IMAGE";
88
/// OpenShell's built-in default sandbox image. Pulling it needs access to
99
/// nvcr.io.
10-
pub const DEFAULT_BASE_IMAGE: &str = "nvcr.io/nvidia/base/ubuntu:24.04";
10+
const DEFAULT_BASE_IMAGE: &str = "nvcr.io/nvidia/base/ubuntu:24.04";
11+
12+
/// Overrides [`DEFAULT_PYTHON_IMAGE`].
13+
const PYTHON_IMAGE_ENV: &str = "OPENSHELL_TEST_PYTHON_IMAGE";
14+
/// The workload image for tests that run `python3`, curl, git or openssl in a
15+
/// sandbox, built from `e2e/python/Dockerfile.workload`. It isn't published:
16+
/// the tmachine installers load it into every environment, and
17+
/// `mise run e2e:workload:build` builds it locally.
18+
const DEFAULT_PYTHON_IMAGE: &str = "openshell/e2e-python:dev";
1119

1220
/// The image named by [`BASE_IMAGE_ENV`], or [`DEFAULT_BASE_IMAGE`].
1321
pub fn base_image() -> String {
14-
std::env::var(BASE_IMAGE_ENV)
22+
image_from_env(BASE_IMAGE_ENV, DEFAULT_BASE_IMAGE)
23+
}
24+
25+
/// The image named by [`PYTHON_IMAGE_ENV`], or [`DEFAULT_PYTHON_IMAGE`].
26+
pub fn python_image() -> String {
27+
image_from_env(PYTHON_IMAGE_ENV, DEFAULT_PYTHON_IMAGE)
28+
}
29+
30+
fn image_from_env(var: &str, default: &str) -> String {
31+
std::env::var(var)
1532
.ok()
1633
.filter(|image| !image.trim().is_empty())
17-
.unwrap_or_else(|| DEFAULT_BASE_IMAGE.to_string())
34+
.unwrap_or_else(|| default.to_string())
1835
}

0 commit comments

Comments
 (0)