Skip to content

Commit c1bb9fd

Browse files
committed
fix(e2e): use a dedicated Noble fixture for Docker tests
Signed-off-by: Evan Lezar <[email protected]>
1 parent 06ea61e commit c1bb9fd

11 files changed

Lines changed: 116 additions & 23 deletions

File tree

‎TESTING.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,21 @@ requires mTLS.
106106

107107
### Python E2E (`e2e/python/`)
108108

109+
`mise run e2e:python` builds `openshell/e2e-python:dev` from
110+
`e2e/python/Dockerfile.workload` and selects it only for the test gateway.
111+
This Noble-based fixture supplies the `sandbox` user, Python tooling, Git,
112+
and a writable `/sandbox/.venv`. Its Python version comes from `.python-version`
113+
so cloudpickle code objects match the test runner. The production workload
114+
default remains the unmodified NVIDIA Ubuntu image.
115+
116+
The Rust Docker harness also selects this fixture for tests that need tools
117+
or the named user. Explicit `--from` or `--template` arguments and the default-image
118+
tests retain their own image selection. Docker sandbox and support-container
119+
fixtures use the fixed image name; no workload-image override is needed.
120+
Podman, VM, and Kubernetes retain their existing pinned, pullable fixture and
121+
image setup. Conformance-only runs do not need the Docker fixture.
122+
Build the Docker fixture separately with `mise run e2e:workload:build`.
123+
109124
Tests use the `sandbox` fixture from `conftest.py` to create real sandboxes:
110125

111126
```python

‎architecture/build.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,12 @@ with GNU libc and uses the same glibc 2.28 compatibility floor as the gateway.
102102

103103
## Container Builds
104104

105+
Docker E2E tool-dependent workloads use a dedicated Noble-based fixture,
106+
separate from the product's minimal default image. The fixture supplies the
107+
test identity and tools, with Python aligned to the host test runner for
108+
serialized callable compatibility. Default-image coverage retains the product
109+
image. Other compute-driver test lanes retain their existing workload fixtures.
110+
105111
The Docker image pipeline is a two-step flow: build the Rust binary natively
106112
for the target architecture, then assemble the container image from the
107113
prebuilt binary. The gateway, sandbox, and supervisor images use distinct

‎e2e/python/Dockerfile.workload‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
ARG WORKLOAD_BASE=nvcr.io/nvidia/base/ubuntu:24.04
5+
FROM ghcr.io/astral-sh/uv:0.12.17@sha256:10787c682e4184e4f290de1171fd4703dc63de99221f10fe1c99002ce7fa9acc AS uv
6+
FROM ${WORKLOAD_BASE}
7+
8+
COPY --from=uv /uv /uvx /usr/local/bin/
9+
# Supplied from .python-version by the build task for cloudpickle compatibility.
10+
ARG PYTHON_VERSION
11+
# curl and openssl also support the Rust Docker tests sharing this fixture.
12+
RUN apt-get update \
13+
&& apt-get install -y --no-install-recommends \
14+
bash ca-certificates curl git openssl python3 python3-pip python3-venv python-is-python3 \
15+
&& rm -rf /var/lib/apt/lists/* \
16+
&& if getent passwd ubuntu >/dev/null; then userdel ubuntu; fi \
17+
&& if getent group ubuntu >/dev/null; then groupdel ubuntu; fi \
18+
&& groupadd --gid 1000 sandbox \
19+
&& useradd --uid 1000 --gid sandbox --create-home --shell /bin/bash sandbox \
20+
&& UV_PYTHON_INSTALL_DIR=/sandbox/.uv/python uv python install "${PYTHON_VERSION}" \
21+
&& UV_PYTHON_INSTALL_DIR=/sandbox/.uv/python uv venv --python "${PYTHON_VERSION}" --seed /sandbox/.venv \
22+
&& uv pip install --python /sandbox/.venv/bin/python cloudpickle==3.1.2 \
23+
&& chown -R sandbox:sandbox /sandbox \
24+
&& uv cache clean
25+
26+
ENV VIRTUAL_ENV=/sandbox/.venv
27+
ENV PATH="/sandbox/.venv/bin:${PATH}"
28+
WORKDIR /sandbox
29+
USER sandbox:sandbox

‎e2e/python/conftest.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,7 @@ def _create(*, spec: object | None = None, delete_on_exit: bool = True) -> Sandb
9393
cluster=cluster_name,
9494
spec=spec,
9595
delete_on_exit=delete_on_exit,
96-
# The sandbox image is large (Python, Node.js, coding agents) so the
97-
# first pod in the cluster may need extra time for the image pull.
96+
# Allow time to pull an explicitly supplied workload fixture.
9897
ready_timeout_seconds=300.0,
9998
)
10099

‎e2e/rust/e2e-docker.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,5 +44,9 @@ if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then
4444
exit 0
4545
fi
4646

47+
if [ -n "${E2E_FEATURES}" ]; then
48+
CONTAINER_ENGINE=docker bash "${ROOT}/tasks/scripts/e2e-build-workload.sh"
49+
fi
50+
4751
exec "${ROOT}/e2e/with-docker-gateway.sh" \
4852
bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}"

‎e2e/rust/src/harness/container.rs‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,7 @@ use std::time::Duration;
1515
use tokio::time::{interval, timeout};
1616

1717
use super::port::find_free_port;
18-
19-
const DEFAULT_TEST_SERVER_IMAGE: &str = super::sandbox::E2E_WORKLOAD_IMAGE;
18+
use super::sandbox::E2E_WORKLOAD_IMAGE;
2019

2120
#[must_use]
2221
pub fn e2e_driver() -> Option<String> {
@@ -197,7 +196,7 @@ impl ContainerHttpServer {
197196
]);
198197
}
199198
args.extend([
200-
DEFAULT_TEST_SERVER_IMAGE.to_string(),
199+
E2E_WORKLOAD_IMAGE.to_string(),
201200
"-c".to_string(),
202201
script.to_string(),
203202
]);
@@ -325,7 +324,7 @@ impl HostSupportContainer {
325324
"python3",
326325
"-p",
327326
&format!("{port}:{container_port}"),
328-
DEFAULT_TEST_SERVER_IMAGE,
327+
E2E_WORKLOAD_IMAGE,
329328
"-c",
330329
script,
331330
])
@@ -380,7 +379,7 @@ impl HostSupportContainer {
380379
.map(|capability| format!("--cap-add={capability}")),
381380
);
382381
args.extend([
383-
DEFAULT_TEST_SERVER_IMAGE.to_string(),
382+
E2E_WORKLOAD_IMAGE.to_string(),
384383
"-c".to_string(),
385384
script.to_string(),
386385
]);
@@ -493,7 +492,7 @@ impl SupportContainer {
493492
.map(|capability| format!("--cap-add={capability}")),
494493
);
495494
args.extend([
496-
DEFAULT_TEST_SERVER_IMAGE.to_string(),
495+
E2E_WORKLOAD_IMAGE.to_string(),
497496
"-c".to_string(),
498497
script.to_string(),
499498
]);

‎e2e/rust/src/harness/sandbox.rs‎

Lines changed: 28 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,13 @@ use super::output::{extract_field, strip_ansi};
2020
/// Tool-capable workload image used by the E2E harness.
2121
///
2222
/// Product defaults remain on the minimal NVIDIA Ubuntu image. Tests that
23-
/// explicitly pass `--from` continue to exercise their requested image.
23+
/// explicitly pass `--from` or `--template` retain their requested workload.
24+
/// Docker setup builds this Noble-based fixture before running the tests.
25+
#[cfg(feature = "e2e-docker")]
26+
pub const E2E_WORKLOAD_IMAGE: &str = "openshell/e2e-python:dev";
27+
28+
/// Preserve the existing pullable fixture for non-Docker E2E lanes.
29+
#[cfg(not(feature = "e2e-docker"))]
2430
pub const E2E_WORKLOAD_IMAGE: &str = "ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim@sha256:9a59bb7206905ccaae4f7dab222fbac47c125a21e5fc16f43f427cd6c940ade3";
2531

2632
/// Extract the sandbox name from CLI create output.
@@ -44,13 +50,17 @@ fn has_explicit_sandbox_name(args: &[&str]) -> bool {
4450
.any(|arg| *arg == "--name" || arg.starts_with("--name="))
4551
}
4652

47-
fn has_explicit_sandbox_image(args: &[&str]) -> bool {
48-
args.iter()
49-
.any(|arg| *arg == "--from" || arg.starts_with("--from="))
53+
fn has_explicit_sandbox_workload(args: &[&str]) -> bool {
54+
args.iter().take_while(|arg| **arg != "--").any(|arg| {
55+
*arg == "--from"
56+
|| arg.starts_with("--from=")
57+
|| *arg == "--template"
58+
|| arg.starts_with("--template=")
59+
})
5060
}
5161

5262
fn add_test_image_if_missing(command: &mut tokio::process::Command, args: &[&str]) {
53-
if !has_explicit_sandbox_image(args) {
63+
if !has_explicit_sandbox_workload(args) {
5464
command.arg("--from").arg(E2E_WORKLOAD_IMAGE);
5565
}
5666
}
@@ -737,7 +747,7 @@ impl Drop for SandboxGuard {
737747

738748
#[cfg(test)]
739749
mod tests {
740-
use super::{has_explicit_sandbox_image, has_explicit_sandbox_name};
750+
use super::{has_explicit_sandbox_name, has_explicit_sandbox_workload};
741751

742752
#[test]
743753
fn detects_explicit_sandbox_names() {
@@ -747,9 +757,17 @@ mod tests {
747757
}
748758

749759
#[test]
750-
fn detects_explicit_sandbox_images() {
751-
assert!(has_explicit_sandbox_image(&["--from", "example:latest"]));
752-
assert!(has_explicit_sandbox_image(&["--from=example:latest"]));
753-
assert!(!has_explicit_sandbox_image(&["--policy", "policy.yaml"]));
760+
fn detects_explicit_sandbox_workloads() {
761+
assert!(has_explicit_sandbox_workload(&["--from", "example:latest"]));
762+
assert!(has_explicit_sandbox_workload(&["--from=example:latest"]));
763+
assert!(has_explicit_sandbox_workload(&["--template", "example"]));
764+
assert!(has_explicit_sandbox_workload(&["--template=example"]));
765+
assert!(!has_explicit_sandbox_workload(&["--policy", "policy.yaml"]));
766+
assert!(!has_explicit_sandbox_workload(&["--", "echo", "--from=x"]));
767+
assert!(!has_explicit_sandbox_workload(&[
768+
"--",
769+
"echo",
770+
"--template=x"
771+
]));
754772
}
755773
}

‎e2e/rust/tests/provider_auto_create.rs‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ use std::sync::Mutex;
2222

2323
use openshell_e2e::harness::binary::openshell_cmd;
2424
use openshell_e2e::harness::output::{extract_field, strip_ansi};
25+
#[cfg(feature = "e2e-docker")]
26+
use openshell_e2e::harness::sandbox::E2E_WORKLOAD_IMAGE;
2527

2628
const TEST_API_KEY: &str = "sk-e2e-auto-provider-test-key";
2729
static CLAUDE_PROVIDER_LOCK: Mutex<()> = Mutex::new(());
@@ -108,9 +110,10 @@ network_policies: {}
108110
// Create a sandbox that prints the ANTHROPIC_API_KEY env var.
109111
// --auto-providers skips the interactive prompt.
110112
let mut cmd = openshell_cmd();
111-
cmd.arg("sandbox")
112-
.arg("create")
113-
.arg("--detach")
113+
cmd.arg("sandbox").arg("create");
114+
#[cfg(feature = "e2e-docker")]
115+
cmd.arg("--from").arg(E2E_WORKLOAD_IMAGE);
116+
cmd.arg("--detach")
114117
.arg("--policy")
115118
.arg(policy.path())
116119
.arg("--provider")

‎e2e/rust/tests/provider_readiness.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1334,7 +1334,7 @@ async fn acknowledged_provider_changes_apply_to_fresh_clients_and_revoke_retaine
13341334
std::fs::write(
13351335
&dockerfile,
13361336
format!(
1337-
"FROM {E2E_WORKLOAD_IMAGE}\nUSER root\nRUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*\nCOPY client.py /opt/provider-readiness-client.py\nUSER 1000:1000\n"
1337+
"FROM {E2E_WORKLOAD_IMAGE}\nCOPY client.py /opt/provider-readiness-client.py\nUSER 1000:1000\n",
13381338
),
13391339
)
13401340
.map_err(|_| "could not write fixture Dockerfile")?;
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/env bash
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
set -euo pipefail
6+
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
7+
# shellcheck source=tasks/scripts/container-engine.sh
8+
source "${ROOT}/tasks/scripts/container-engine.sh"
9+
ce_build --load --file "${ROOT}/e2e/python/Dockerfile.workload" \
10+
--build-arg "PYTHON_VERSION=$(cat "${ROOT}/.python-version")" \
11+
--tag openshell/e2e-python:dev \
12+
"${ROOT}/e2e/python"

0 commit comments

Comments
 (0)