Skip to content

Commit 86cf640

Browse files
committed
fix(test-guest): harden runner and OCI reuse
Signed-off-by: Drew Newberry <[email protected]>
1 parent 418b260 commit 86cf640

3 files changed

Lines changed: 49 additions & 88 deletions

File tree

‎nix/test-guest/README.md‎

Lines changed: 1 addition & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -63,81 +63,6 @@ List the available distros and configurations:
6363
nix run .#test-guest -- --list
6464
```
6565

66-
## Run a host or test-guest E2E suite
67-
68-
[`e2e/run.sh`](../../e2e/run.sh) builds OpenShell from the current checkout, starts a gateway from a supplied TOML file, and runs one named host-side suite against it. This opt-in path is separate from `mise run e2e:vm`: `e2e/run.sh` selects where the gateway process runs, while the supplied gateway config selects the compute driver. It is not part of the default E2E task or CI graph.
69-
70-
Omit both `--vm` and `--with` to run the gateway on the host:
71-
72-
```shell
73-
e2e/run.sh \
74-
--gateway-config e2e/configs/gateway/docker.toml \
75-
--suite smoke
76-
```
77-
78-
Pass `--vm` or at least one `--with` to run the gateway in a disposable test guest:
79-
80-
```shell
81-
e2e/run.sh \
82-
--vm ubuntu \
83-
--with docker \
84-
--gateway-config e2e/configs/gateway/docker.toml \
85-
--suite smoke
86-
```
87-
88-
Use `--guest-setup` when a driver needs checkout-specific guest preparation
89-
before the gateway starts. For example, the Podman shutdown suite builds a
90-
supervisor image around the staged `openshell-sandbox` binary:
91-
92-
```shell
93-
e2e/run.sh \
94-
--vm ubuntu \
95-
--with podman \
96-
--guest-setup e2e/guest-setup/podman-supervisor.sh \
97-
--gateway-config e2e/configs/gateway/podman.toml \
98-
--suite podman-shutdown
99-
```
100-
101-
Set `OPENSHELL_E2E_EXPECT_SHUTDOWN_CLOSES=1` when a stacked change must
102-
classify supervisor and relay transport closures as expected teardown events.
103-
104-
Supplying `--with` without `--vm` selects Ubuntu. `--with` is repeatable and configurations run in input order. Supplying `--vm` without `--with` boots the selected base image without adding software. The wrapper validates both values against the test-guest catalogs:
105-
106-
```shell
107-
e2e/run.sh \
108-
--vm rocky \
109-
--with docker \
110-
--with selinux \
111-
--gateway-config e2e/configs/gateway/docker.toml \
112-
--suite smoke
113-
```
114-
115-
The equivalent passthrough task accepts the same arguments:
116-
117-
```shell
118-
mise run e2e:test -- \
119-
--with docker \
120-
--gateway-config e2e/configs/gateway/docker.toml \
121-
--suite smoke
122-
```
123-
124-
The wrapper always builds the tested binaries; it does not accept packages or use release artifacts. Both modes build a native host `openshell` CLI and a static Linux `openshell-sandbox` for the host architecture. Host mode also builds a native `openshell-gateway` with bundled Z3. VM mode builds a static Linux `openshell` and a Linux GNU `openshell-gateway` with bundled Z3 and a glibc 2.28 floor, then stages all three guest binaries through the VM runner's `--copy` interface.
125-
126-
The Linux targets match the host and guest architecture:
127-
128-
| Host architecture | CLI and sandbox | Gateway |
129-
| --- | --- | --- |
130-
| x86_64 | `x86_64-unknown-linux-musl` | `x86_64-unknown-linux-gnu.2.28` |
131-
| aarch64 | `aarch64-unknown-linux-musl` | `aarch64-unknown-linux-gnu.2.28` |
132-
133-
These cross-builds use the Zig and `cargo-zigbuild` versions pinned in [`mise.toml`](../../mise.toml). Install the repository's mise tools before running the wrapper. The wrapper also requires Python 3 and OpenSSL; VM mode requires `base64`. Cargo stores outputs in its normal target directory, so later runs reuse current artifacts. VM mode additionally requires the Nix, QEMU, and host-capacity prerequisites above; host mode does not invoke Nix, QEMU, or Ansible.
134-
135-
Gateway configs are ordinary, fully resolved TOML files under `e2e/configs/gateway/`. The wrapper does not inspect or rewrite them. It supplies only the bind address, gateway port, plaintext transport, and isolated XDG directories at launch. Use portable relative paths for host files. The initial Docker config points `supervisor_bin` at `.cache/openshell-e2e/bin/openshell-sandbox`; the wrapper stages that path under the repository in host mode and under `/home/openshell` in VM mode. It also generates and stages the local sandbox JWT material referenced by the initial config.
136-
137-
Suites are executable scripts under `e2e/suites/` and use lowercase letters, digits, and hyphens for their names. The wrapper exports the selected endpoint, isolated gateway registration, native host CLI path, mode metadata, source config path, and forwarded ports before executing a suite. Add new configs and suites without adding scenario branches to `e2e/run.sh`.
138-
139-
Set `OPENSHELL_E2E_KEEP=1` to retain wrapper state after a host-mode run. In VM mode the wrapper also passes `--keep` to the test-guest runner so its overlay and logs remain available.
140-
14166
## Open an interactive VM
14267

14368
Boot a base Ubuntu VM:
@@ -323,4 +248,4 @@ Use `--keep` to preserve the overlay, cloud-init seed, SSH key, and serial log f
323248
- Prepared cache entries are architecture-specific and match the exact ordered configuration list.
324249
- OCI pulls transfer a complete compressed standalone disk; incremental disk layers are not implemented.
325250
- Guest ports are reachable from the host only when explicitly exposed with loopback-only `--forward-port`.
326-
- The low-level VM runner does not build OpenShell, configure a gateway, or select an E2E test suite. Use `e2e/run.sh` for that orchestration.
251+
- The runner does not build OpenShell, configure a gateway, or select an E2E test suite.

‎nix/test-guest/cache.sh‎

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,10 @@ is_remote_miss() {
241241
}
242242

243243
pull_remote() {
244+
local requested_ref=${1:-${pull_ref}}
245+
local trusted_digest=${2:-${pull_digest}}
246+
local install_local=${3:-1}
247+
local expected_local_sha=${4:-}
244248
local pull_dir
245249
local pull_log
246250
local compressed_size
@@ -250,17 +254,17 @@ pull_remote() {
250254
pull_dir=$(mktemp -d "${cache_root}/staging/pull.${cache_key}.XXXXXX")
251255
pull_log="${pull_dir}/oras.log"
252256

253-
resolved_digest=$(oras resolve "${pull_ref}") || {
257+
resolved_digest=$(oras resolve "${requested_ref}") || {
254258
rm -rf "${pull_dir}"
255259
return 2
256260
}
257-
if [ "${resolved_digest}" != "${pull_digest}" ]; then
261+
if [ "${resolved_digest}" != "${trusted_digest}" ]; then
258262
echo "OCI cache manifest digest does not match trusted digest" >&2
259263
rm -rf "${pull_dir}"
260264
return 2
261265
fi
262266

263-
if ! oras pull --output "${pull_dir}" "${pull_ref}" >"${pull_log}" 2>&1; then
267+
if ! oras pull --output "${pull_dir}" "${requested_ref}" >"${pull_log}" 2>&1; then
264268
if is_remote_miss "${pull_log}"; then
265269
rm -rf "${pull_dir}"
266270
return 1
@@ -298,6 +302,11 @@ pull_remote() {
298302
-o "${pull_dir}/disk.qcow2"
299303
)
300304
expected_sha=$(jq -r '.disk_sha256' "${pull_dir}/metadata.json")
305+
if [ -n "${expected_local_sha}" ] && [ "${expected_sha}" != "${expected_local_sha}" ]; then
306+
echo "OCI cache disk checksum does not match the local cache entry" >&2
307+
rm -rf "${pull_dir}"
308+
return 2
309+
fi
301310
actual_sha=$(test_vm_cache_sha256 "${pull_dir}/disk.qcow2")
302311
if [ "${expected_sha}" != "${actual_sha}" ]; then
303312
echo "OCI cache disk checksum does not match metadata" >&2
@@ -310,10 +319,14 @@ pull_remote() {
310319
return 2
311320
fi
312321

313-
install_entry \
314-
"${pull_dir}/disk.qcow2" "${pull_dir}/metadata.json" "${resolved_digest}"
322+
if [ "${install_local}" -eq 1 ]; then
323+
install_entry \
324+
"${pull_dir}/disk.qcow2" "${pull_dir}/metadata.json" "${resolved_digest}"
325+
fi
315326
rm -rf "${pull_dir}"
316-
echo "==> Cache remote hit: ${pull_ref}"
327+
if [ "${install_local}" -eq 1 ]; then
328+
echo "==> Cache remote hit: ${requested_ref}"
329+
fi
317330
}
318331

319332
build_local() {
@@ -426,15 +439,24 @@ build_local() {
426439
}
427440

428441
push_remote() {
442+
local existing_ref
443+
local local_disk_sha
429444
local push_dir
430445
local manifest_log
431446
local published_digest
432447
push_dir=$(mktemp -d "${cache_root}/staging/push.${cache_key}.XXXXXX")
433448
manifest_log="${push_dir}/manifest.log"
434449

435450
if oras manifest fetch "${remote_ref}" >"${manifest_log}" 2>&1; then
436-
echo "==> Cache already published: ${remote_ref}"
437451
published_digest=$(oras resolve "${remote_ref}")
452+
existing_ref="${repository}@${published_digest}"
453+
local_disk_sha=$(jq -r '.disk_sha256' "${entry_dir}/metadata.json")
454+
if ! pull_remote "${existing_ref}" "${published_digest}" 0 "${local_disk_sha}"; then
455+
echo "existing OCI cache artifact failed validation: ${remote_ref}" >&2
456+
rm -rf "${push_dir}"
457+
return 1
458+
fi
459+
echo "==> Cache already published and validated: ${remote_ref}"
438460
echo "==> Cache immutable reference: ${repository}@${published_digest}"
439461
rm -rf "${push_dir}"
440462
return

‎nix/test-guest/run.sh‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@ if [ "${OPENSHELL_TEST_GUEST_RUNTIME:-}" != 1 ] ||
3535
exit 2
3636
fi
3737

38+
require_value() {
39+
if [ "$#" -lt 2 ] || [ -z "${2:-}" ]; then
40+
echo "$1 requires a value" >&2
41+
exit 2
42+
fi
43+
}
44+
3845
distro=
3946
requested_ssh_port=
4047
keep=0
@@ -48,23 +55,28 @@ guest_command=()
4855
while [ "$#" -gt 0 ]; do
4956
case "$1" in
5057
--distro)
51-
distro=${2:-}
58+
require_value "$@"
59+
distro=$2
5260
shift 2
5361
;;
5462
--with)
55-
configurations+=("${2:-}")
63+
require_value "$@"
64+
configurations+=("$2")
5665
shift 2
5766
;;
5867
--install)
59-
packages+=("${2:-}")
68+
require_value "$@"
69+
packages+=("$2")
6070
shift 2
6171
;;
6272
--copy)
63-
copies+=("${2:-}")
73+
require_value "$@"
74+
copies+=("$2")
6475
shift 2
6576
;;
6677
--ssh-port)
67-
requested_ssh_port=${2:-}
78+
require_value "$@"
79+
requested_ssh_port=$2
6880
shift 2
6981
;;
7082
--forward-port)
@@ -434,6 +446,7 @@ ssh_args=(
434446
-o ControlMaster=auto
435447
-o ControlPersist=60
436448
-o "ControlPath=${ssh_control_path}"
449+
-o IdentitiesOnly=yes
437450
-o LogLevel=ERROR
438451
-o StrictHostKeyChecking=no
439452
-o UserKnownHostsFile=/dev/null
@@ -447,6 +460,7 @@ scp_args=(
447460
-o ControlMaster=auto
448461
-o ControlPersist=60
449462
-o "ControlPath=${ssh_control_path}"
463+
-o IdentitiesOnly=yes
450464
-o LogLevel=ERROR
451465
-o StrictHostKeyChecking=no
452466
-o UserKnownHostsFile=/dev/null

0 commit comments

Comments
 (0)