Skip to content

Commit e1fff99

Browse files
committed
feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes
The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <[email protected]>
1 parent 74960eb commit e1fff99

16 files changed

Lines changed: 2451 additions & 8 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -630,6 +630,49 @@ openshell status
630630
openshell logs <sandbox-name>
631631
```
632632

633+
#### Corporate upstream proxy
634+
635+
When VM sandbox egress routes through a corporate HTTP forward proxy, the
636+
operator-owned settings live under `[openshell.drivers.vm]` and the gateway
637+
forwards them to the `openshell-driver-vm` subprocess as `--https-proxy`,
638+
`--no-proxy`, `--proxy-auth-file`, `--proxy-auth-allow-insecure`,
639+
`--proxy-connect-by-hostname`, and `--proxy-ca-bundle`. Both the gateway and
640+
the driver validate them at startup, so any present-but-invalid value fails
641+
closed with an error naming the key rather than reverting to a direct dial.
642+
Confirm the configuration and the resulting driver argv first:
643+
644+
```bash
645+
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
646+
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--proxy\|--https-proxy\|--no-proxy'
647+
```
648+
649+
Reachability is the most common failure. Guest egress leaves through gvproxy,
650+
so a proxy bound to the gateway host's loopback is **not** reachable at
651+
`127.0.0.1` from inside the guest: it must be addressed as
652+
`http://host.openshell.internal:<port>`, which gvproxy NATs from
653+
`192.168.127.254` to the host's `127.0.0.1`. A `https_proxy` pointing at a
654+
loopback URL produces policy-approved CONNECT attempts that time out while
655+
public destinations still work.
656+
657+
The settings reach the supervisor through a driver-written argument file in
658+
the per-sandbox overlay, not through the guest environment. The credential and
659+
CA bundle are staged into the same overlay at fixed guest paths. Inspect the
660+
guest side from the VM console log, which records how many driver-supplied
661+
arguments the init script read:
662+
663+
```bash
664+
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
665+
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40
666+
```
667+
668+
`FATAL: supervisor argument list ... is not readable` or `FATAL: empty entry in
669+
supervisor argument list` means the overlay is broken or was tampered with, and
670+
the guest deliberately aborts rather than starting a supervisor with a
671+
truncated egress configuration. If the guest logs no driver arguments at all
672+
while `gateway.toml` sets `https_proxy`, the running driver predates the
673+
configuration — check that the gateway spawned the driver binary you expect
674+
(`[openshell.drivers.vm].driver_dir`).
675+
633676
## Common Failure Patterns
634677

635678
| Symptom | Likely cause | Check |

‎architecture/sandbox.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,30 @@ file and builds the `Proxy-Authorization: Basic` header; a credential that is
300300
empty, contains control characters, or is not in `user:pass` form is fatal on
301301
both sides.
302302

303+
The VM driver has no argv seam of its own: its guest init script runs as PID 1
304+
and execs a fixed supervisor command line, and the libkrun and QEMU launch
305+
backends both reach the supervisor through that script. Driver-owned
306+
supervisor arguments therefore travel in a per-sandbox file the driver writes
307+
into the overlay upperdir at a fixed guest path, one argument per line, which
308+
the guest reads verbatim (no word splitting or globbing) and appends to every
309+
supervisor exec. The file is written on **every** launch, including an empty
310+
file when there is nothing to pass: the upperdir copy always shadows the
311+
read-only image layer, so a sandbox image can neither supply its own
312+
supervisor arguments by baking a file at that path nor disable the operator's
313+
by omitting one. This mirrors the driver-authored `init.d` manifest, which
314+
solves the same trust problem for guest init drop-ins.
315+
316+
A microVM has no bind mounts or container secrets, so the VM driver stages the
317+
credential and the CA bundle into the per-sandbox overlay disk instead — the
318+
credential root-only, the CA world-readable, both at fixed `/opt/openshell`
319+
paths and both removed with the sandbox state directory. The consequence,
320+
which differs from the Podman secret model, is that the credential is at rest
321+
inside that overlay image on the gateway host; the per-sandbox gateway JWT
322+
already travels the same path. Guest egress leaves through gvproxy, so a proxy
323+
on the gateway host's loopback is reachable only through the host alias
324+
`host.openshell.internal`; the guest's gateway callback is unaffected and
325+
never traverses the proxy.
326+
303327
For Kubernetes sandboxes, the operator configures a Secret name and key rather
304328
than a gateway-host file path. Kubernetes projects that Secret only into the
305329
container that runs network supervision. Proxy credential Secrets require the

‎crates/openshell-core/src/container_paths.rs‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,32 @@ pub const VM_GUEST_TLS_KEY_PATH: &str = "/opt/openshell/tls/tls.key";
6565
pub const VM_GUEST_SANDBOX_TOKEN_PATH: &str = "/opt/openshell/auth/sandbox.jwt";
6666
pub const VM_GUEST_INIT_DROPIN_DIR: &str = "/opt/openshell/init.d";
6767
pub const VM_GUEST_INIT_DROPIN_MANIFEST: &str = "/opt/openshell/init.d.manifest";
68+
69+
/// Guest path for the corporate upstream-proxy credential in VM sandboxes.
70+
///
71+
/// The VM driver stages the `user:pass` credential here (mode `0600`,
72+
/// root-only) inside the per-sandbox overlay upperdir, and passes only this
73+
/// path on the supervisor's argv. A microVM has no bind mounts or container
74+
/// secrets, so this is the same delivery the per-sandbox JWT already uses.
75+
pub const VM_GUEST_UPSTREAM_PROXY_AUTH_PATH: &str = "/opt/openshell/auth/upstream-proxy";
76+
77+
/// Guest path for the corporate proxy CA bundle in VM sandboxes.
78+
///
79+
/// A CA certificate is not secret, so unlike the credential this is staged
80+
/// world-readable. The supervisor trusts it for the handshake with an
81+
/// `https://` proxy and for server certificates re-signed by a
82+
/// TLS-intercepting proxy.
83+
pub const VM_GUEST_PROXY_CA_PATH: &str = "/opt/openshell/tls/proxy-ca.pem";
84+
85+
/// Guest path for the driver-authored supervisor argument list in VM sandboxes.
86+
///
87+
/// Podman and Kubernetes build the supervisor's command line directly; the VM
88+
/// guest init script execs a fixed argv, so driver-owned arguments travel
89+
/// through this file instead. The driver writes it into the overlay upperdir
90+
/// on every launch — empty when it has no arguments to pass — so a sandbox
91+
/// image can neither forge entries nor shadow the driver's copy, and the
92+
/// guest appends exactly what it finds there and nothing else.
93+
pub const VM_GUEST_SUPERVISOR_ARGS_PATH: &str = "/opt/openshell/supervisor-args";
6894
pub const VM_UMOCI_PATH: &str = "/opt/openshell/bin/umoci";
6995
pub const VM_SANDBOX_OWNER_NORMALIZED_MARKER: &str = "/opt/openshell/.sandbox-owner-normalized";
7096

@@ -103,6 +129,9 @@ mod tests {
103129
VM_GUEST_SANDBOX_TOKEN_PATH,
104130
VM_GUEST_INIT_DROPIN_DIR,
105131
VM_GUEST_INIT_DROPIN_MANIFEST,
132+
VM_GUEST_UPSTREAM_PROXY_AUTH_PATH,
133+
VM_GUEST_PROXY_CA_PATH,
134+
VM_GUEST_SUPERVISOR_ARGS_PATH,
106135
VM_UMOCI_PATH,
107136
VM_SANDBOX_OWNER_NORMALIZED_MARKER,
108137
];

0 commit comments

Comments
 (0)