Skip to content

Commit dc1218b

Browse files
feloyjohntmyers
authored andcommitted
feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes
The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <[email protected]>
1 parent 7b64c5c commit dc1218b

16 files changed

Lines changed: 2451 additions & 8 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -610,6 +610,49 @@ openshell status
610610
openshell logs <sandbox-name>
611611
```
612612

613+
#### Corporate upstream proxy
614+
615+
When VM sandbox egress routes through a corporate HTTP forward proxy, the
616+
operator-owned settings live under `[openshell.drivers.vm]` and the gateway
617+
forwards them to the `openshell-driver-vm` subprocess as `--https-proxy`,
618+
`--no-proxy`, `--proxy-auth-file`, `--proxy-auth-allow-insecure`,
619+
`--proxy-connect-by-hostname`, and `--proxy-ca-bundle`. Both the gateway and
620+
the driver validate them at startup, so any present-but-invalid value fails
621+
closed with an error naming the key rather than reverting to a direct dial.
622+
Confirm the configuration and the resulting driver argv first:
623+
624+
```bash
625+
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
626+
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--proxy\|--https-proxy\|--no-proxy'
627+
```
628+
629+
Reachability is the most common failure. Guest egress leaves through gvproxy,
630+
so a proxy bound to the gateway host's loopback is **not** reachable at
631+
`127.0.0.1` from inside the guest: it must be addressed as
632+
`http://host.openshell.internal:<port>`, which gvproxy NATs from
633+
`192.168.127.254` to the host's `127.0.0.1`. A `https_proxy` pointing at a
634+
loopback URL produces policy-approved CONNECT attempts that time out while
635+
public destinations still work.
636+
637+
The settings reach the supervisor through a driver-written argument file in
638+
the per-sandbox overlay, not through the guest environment. The credential and
639+
CA bundle are staged into the same overlay at fixed guest paths. Inspect the
640+
guest side from the VM console log, which records how many driver-supplied
641+
arguments the init script read:
642+
643+
```bash
644+
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
645+
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40
646+
```
647+
648+
`FATAL: supervisor argument list ... is not readable` or `FATAL: empty entry in
649+
supervisor argument list` means the overlay is broken or was tampered with, and
650+
the guest deliberately aborts rather than starting a supervisor with a
651+
truncated egress configuration. If the guest logs no driver arguments at all
652+
while `gateway.toml` sets `https_proxy`, the running driver predates the
653+
configuration — check that the gateway spawned the driver binary you expect
654+
(`[openshell.drivers.vm].driver_dir`).
655+
613656
## Common Failure Patterns
614657

615658
| Symptom | Likely cause | Check |

‎architecture/sandbox.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,30 @@ file and builds the `Proxy-Authorization: Basic` header; a credential that is
294294
empty, contains control characters, or is not in `user:pass` form is fatal on
295295
both sides.
296296

297+
The VM driver has no argv seam of its own: its guest init script runs as PID 1
298+
and execs a fixed supervisor command line, and the libkrun and QEMU launch
299+
backends both reach the supervisor through that script. Driver-owned
300+
supervisor arguments therefore travel in a per-sandbox file the driver writes
301+
into the overlay upperdir at a fixed guest path, one argument per line, which
302+
the guest reads verbatim (no word splitting or globbing) and appends to every
303+
supervisor exec. The file is written on **every** launch, including an empty
304+
file when there is nothing to pass: the upperdir copy always shadows the
305+
read-only image layer, so a sandbox image can neither supply its own
306+
supervisor arguments by baking a file at that path nor disable the operator's
307+
by omitting one. This mirrors the driver-authored `init.d` manifest, which
308+
solves the same trust problem for guest init drop-ins.
309+
310+
A microVM has no bind mounts or container secrets, so the VM driver stages the
311+
credential and the CA bundle into the per-sandbox overlay disk instead — the
312+
credential root-only, the CA world-readable, both at fixed `/opt/openshell`
313+
paths and both removed with the sandbox state directory. The consequence,
314+
which differs from the Podman secret model, is that the credential is at rest
315+
inside that overlay image on the gateway host; the per-sandbox gateway JWT
316+
already travels the same path. Guest egress leaves through gvproxy, so a proxy
317+
on the gateway host's loopback is reachable only through the host alias
318+
`host.openshell.internal`; the guest's gateway callback is unaffected and
319+
never traverses the proxy.
320+
297321
For Kubernetes sandboxes, the operator configures a Secret name and key rather
298322
than a gateway-host file path. Kubernetes projects that Secret only into the
299323
container that runs network supervision. Proxy credential Secrets require the

‎crates/openshell-core/src/container_paths.rs‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,32 @@ pub const VM_GUEST_TLS_KEY_PATH: &str = "/opt/openshell/tls/tls.key";
6565
pub const VM_GUEST_SANDBOX_TOKEN_PATH: &str = "/opt/openshell/auth/sandbox.jwt";
6666
pub const VM_GUEST_INIT_DROPIN_DIR: &str = "/opt/openshell/init.d";
6767
pub const VM_GUEST_INIT_DROPIN_MANIFEST: &str = "/opt/openshell/init.d.manifest";
68+
69+
/// Guest path for the corporate upstream-proxy credential in VM sandboxes.
70+
///
71+
/// The VM driver stages the `user:pass` credential here (mode `0600`,
72+
/// root-only) inside the per-sandbox overlay upperdir, and passes only this
73+
/// path on the supervisor's argv. A microVM has no bind mounts or container
74+
/// secrets, so this is the same delivery the per-sandbox JWT already uses.
75+
pub const VM_GUEST_UPSTREAM_PROXY_AUTH_PATH: &str = "/opt/openshell/auth/upstream-proxy";
76+
77+
/// Guest path for the corporate proxy CA bundle in VM sandboxes.
78+
///
79+
/// A CA certificate is not secret, so unlike the credential this is staged
80+
/// world-readable. The supervisor trusts it for the handshake with an
81+
/// `https://` proxy and for server certificates re-signed by a
82+
/// TLS-intercepting proxy.
83+
pub const VM_GUEST_PROXY_CA_PATH: &str = "/opt/openshell/tls/proxy-ca.pem";
84+
85+
/// Guest path for the driver-authored supervisor argument list in VM sandboxes.
86+
///
87+
/// Podman and Kubernetes build the supervisor's command line directly; the VM
88+
/// guest init script execs a fixed argv, so driver-owned arguments travel
89+
/// through this file instead. The driver writes it into the overlay upperdir
90+
/// on every launch — empty when it has no arguments to pass — so a sandbox
91+
/// image can neither forge entries nor shadow the driver's copy, and the
92+
/// guest appends exactly what it finds there and nothing else.
93+
pub const VM_GUEST_SUPERVISOR_ARGS_PATH: &str = "/opt/openshell/supervisor-args";
6894
pub const VM_UMOCI_PATH: &str = "/opt/openshell/bin/umoci";
6995
pub const VM_SANDBOX_OWNER_NORMALIZED_MARKER: &str = "/opt/openshell/.sandbox-owner-normalized";
7096

@@ -103,6 +129,9 @@ mod tests {
103129
VM_GUEST_SANDBOX_TOKEN_PATH,
104130
VM_GUEST_INIT_DROPIN_DIR,
105131
VM_GUEST_INIT_DROPIN_MANIFEST,
132+
VM_GUEST_UPSTREAM_PROXY_AUTH_PATH,
133+
VM_GUEST_PROXY_CA_PATH,
134+
VM_GUEST_SUPERVISOR_ARGS_PATH,
106135
VM_UMOCI_PATH,
107136
VM_SANDBOX_OWNER_NORMALIZED_MARKER,
108137
];

0 commit comments

Comments
 (0)