Skip to content

Commit 710ce26

Browse files
feloypurp
authored andcommitted
feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes (NVIDIA#3090)
* feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <[email protected]> * fix(vm): bound the proxy CA read and scope the host-loopback recipe Two review findings on the corporate forward proxy support for microVM sandboxes. The driver read the operator's proxy_ca_bundle with an unbounded fs::read and accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A special file such as /dev/zero therefore grew driver memory without bound on every authorized sandbox create, and a PEM block holding invalid DER passed the host check but contributes no trust anchor in the guest, so every supervisor would fail after boot with an error attributed to the sandbox rather than to the setting. Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it reuses the credential reader's bounded-read path (non-regular files rejected on fstat, size capped, read bounded even if the file grows), then requires at least one anchor that RootCertStore::add_parsable_certificates accepts. The supervisor's own reader now delegates to it, so host acceptance and guest acceptance are the same function and cannot drift. The published host-loopback recipe was written for libkrun only. gvproxy NATs host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run on the QEMU/TAP backend where that name resolves to the TAP host address and the driver's own nftables input chain accepts only the gateway port from the guest — no proxy on the gateway host is reachable there at any bind address, so an operator following the generic recipe lost all proxy-required egress while configuration validation succeeded. Scope the recipe to libkrun in every reference and reject a gateway-host proxy URL when a launch plan resolves to QEMU, naming the reason, instead of booting a sandbox whose policy-approved CONNECTs all time out. Signed-off-by: Philippe Martin <[email protected]> * fix(vm): match the QEMU proxy preflight to the selected TAP host The gateway-host proxy guard added for the QEMU/TAP backend classified the wrong set of addresses in both directions. It ran at the top of configure_qemu_launch_plan, before the subnet allocation that settles plan.host_ip, so it could not compare against the address the guest actually reaches the host on. An operator pointing https_proxy at the sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the driver's nftables input chain — which accepts only the gateway port from the guest — then dropped every policy-approved CONNECT, which is exactly the silent timeout the guard exists to prevent. In the other direction it rejected 192.168.127.254 unconditionally. That address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary address that may be routable through the guest's masqueraded egress, so the guard refused a working configuration. Run the check after the launch plan's network allocation, on both the freshly-allocated and already-complete paths, and compare IP literals with that sandbox's selected TAP host. Loopback literals, localhost, and the documented host aliases that write_host_gateway_aliases seeds to the TAP host still classify as the gateway host, and the failure names the address. The gvproxy host-loopback constant returns to being a documentation anchor. Signed-off-by: Philippe Martin <[email protected]> --------- Signed-off-by: Philippe Martin <[email protected]>
1 parent d5a04ad commit 710ce26

20 files changed

Lines changed: 2923 additions & 72 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -630,6 +630,58 @@ openshell status
630630
openshell logs <sandbox-name>
631631
```
632632

633+
#### Corporate upstream proxy
634+
635+
When VM sandbox egress routes through a corporate HTTP forward proxy, the
636+
operator-owned settings live under `[openshell.drivers.vm]` and the gateway
637+
forwards them to the `openshell-driver-vm` subprocess as `--https-proxy`,
638+
`--no-proxy`, `--proxy-auth-file`, `--proxy-auth-allow-insecure`,
639+
`--proxy-connect-by-hostname`, and `--proxy-ca-bundle`. Both the gateway and
640+
the driver validate them at startup, so any present-but-invalid value fails
641+
closed with an error naming the key rather than reverting to a direct dial.
642+
Confirm the configuration and the resulting driver argv first:
643+
644+
```bash
645+
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
646+
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--proxy\|--https-proxy\|--no-proxy'
647+
```
648+
649+
Reachability is the most common failure, and it depends on the VM backend.
650+
On libkrun (non-GPU sandboxes) guest egress leaves through gvproxy, so a proxy
651+
bound to the gateway host's loopback is **not** reachable at `127.0.0.1` from
652+
inside the guest: it must be addressed as
653+
`http://host.openshell.internal:<port>`, which gvproxy NATs from
654+
`192.168.127.254` to the host's `127.0.0.1`. A `https_proxy` pointing at a
655+
loopback URL produces policy-approved CONNECT attempts that time out while
656+
public destinations still work.
657+
658+
GPU sandboxes run on QEMU/TAP, where no gateway-host proxy is reachable at
659+
all: `host.openshell.internal` resolves to the TAP host address, and the
660+
driver's nftables `input` chain accepts only the gateway port from the guest.
661+
The driver rejects such a configuration at launch — a create failing with
662+
`https_proxy ... addresses the gateway host, which a QEMU/TAP sandbox ...
663+
cannot reach` means the proxy must move to an address routable from the
664+
guest's masqueraded egress (or the sandbox must run without a GPU).
665+
666+
The settings reach the supervisor through a driver-written argument file in
667+
the per-sandbox overlay, not through the guest environment. The credential and
668+
CA bundle are staged into the same overlay at fixed guest paths. Inspect the
669+
guest side from the VM console log, which records how many driver-supplied
670+
arguments the init script read:
671+
672+
```bash
673+
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
674+
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40
675+
```
676+
677+
`FATAL: supervisor argument list ... is not readable` or `FATAL: empty entry in
678+
supervisor argument list` means the overlay is broken or was tampered with, and
679+
the guest deliberately aborts rather than starting a supervisor with a
680+
truncated egress configuration. If the guest logs no driver arguments at all
681+
while `gateway.toml` sets `https_proxy`, the running driver predates the
682+
configuration — check that the gateway spawned the driver binary you expect
683+
(`[openshell.drivers.vm].driver_dir`).
684+
633685
## Common Failure Patterns
634686

635687
| Symptom | Likely cause | Check |

‎Cargo.lock‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/sandbox.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,35 @@ file and builds the `Proxy-Authorization: Basic` header; a credential that is
300300
empty, contains control characters, or is not in `user:pass` form is fatal on
301301
both sides.
302302

303+
The VM driver has no argv seam of its own: its guest init script runs as PID 1
304+
and execs a fixed supervisor command line, and the libkrun and QEMU launch
305+
backends both reach the supervisor through that script. Driver-owned
306+
supervisor arguments therefore travel in a per-sandbox file the driver writes
307+
into the overlay upperdir at a fixed guest path, one argument per line, which
308+
the guest reads verbatim (no word splitting or globbing) and appends to every
309+
supervisor exec. The file is written on **every** launch, including an empty
310+
file when there is nothing to pass: the upperdir copy always shadows the
311+
read-only image layer, so a sandbox image can neither supply its own
312+
supervisor arguments by baking a file at that path nor disable the operator's
313+
by omitting one. This mirrors the driver-authored `init.d` manifest, which
314+
solves the same trust problem for guest init drop-ins.
315+
316+
A microVM has no bind mounts or container secrets, so the VM driver stages the
317+
credential and the CA bundle into the per-sandbox overlay disk instead — the
318+
credential root-only, the CA world-readable, both at fixed `/opt/openshell`
319+
paths and both removed with the sandbox state directory. The consequence,
320+
which differs from the Podman secret model, is that the credential is at rest
321+
inside that overlay image on the gateway host; the per-sandbox gateway JWT
322+
already travels the same path. Proxy reachability differs by VM backend. libkrun-backed
323+
sandboxes egress through gvproxy, so a proxy on the gateway host's loopback is
324+
reachable through the host alias `host.openshell.internal`, which gvproxy NATs
325+
to the host's `127.0.0.1`. QEMU/TAP sandboxes (GPU) have no equivalent: that
326+
alias resolves to the TAP host address, and the driver's nftables `input`
327+
chain accepts only the gateway port from the guest, so no gateway-host proxy
328+
is reachable. The driver rejects a gateway-host proxy URL on the QEMU path at
329+
launch rather than producing CONNECT timeouts. The guest's gateway callback is
330+
unaffected in both backends and never traverses the proxy.
331+
303332
For Kubernetes sandboxes, the operator configures a Secret name and key rather
304333
than a gateway-host file path. Kubernetes projects that Secret only into the
305334
container that runs network supervision. Proxy credential Secrets require the

‎crates/openshell-core/Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ serde_json = { workspace = true }
2727
tracing = { workspace = true }
2828
url = { workspace = true }
2929
ipnet = "2"
30+
rustls = { workspace = true }
31+
rustls-pemfile = { workspace = true }
3032
base64 = { workspace = true }
3133
chrono = { version = "0.4", default-features = false, features = ["clock", "std"], optional = true }
3234
reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots"], optional = true }
@@ -53,6 +55,7 @@ protoc-bin-vendored = { workspace = true }
5355

5456
[dev-dependencies]
5557
tempfile = "3"
58+
rcgen = { workspace = true }
5659

5760
[lints]
5861
workspace = true

‎crates/openshell-core/src/container_paths.rs‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,32 @@ pub const VM_GUEST_TLS_KEY_PATH: &str = "/opt/openshell/tls/tls.key";
6565
pub const VM_GUEST_SANDBOX_TOKEN_PATH: &str = "/opt/openshell/auth/sandbox.jwt";
6666
pub const VM_GUEST_INIT_DROPIN_DIR: &str = "/opt/openshell/init.d";
6767
pub const VM_GUEST_INIT_DROPIN_MANIFEST: &str = "/opt/openshell/init.d.manifest";
68+
69+
/// Guest path for the corporate upstream-proxy credential in VM sandboxes.
70+
///
71+
/// The VM driver stages the `user:pass` credential here (mode `0600`,
72+
/// root-only) inside the per-sandbox overlay upperdir, and passes only this
73+
/// path on the supervisor's argv. A microVM has no bind mounts or container
74+
/// secrets, so this is the same delivery the per-sandbox JWT already uses.
75+
pub const VM_GUEST_UPSTREAM_PROXY_AUTH_PATH: &str = "/opt/openshell/auth/upstream-proxy";
76+
77+
/// Guest path for the corporate proxy CA bundle in VM sandboxes.
78+
///
79+
/// A CA certificate is not secret, so unlike the credential this is staged
80+
/// world-readable. The supervisor trusts it for the handshake with an
81+
/// `https://` proxy and for server certificates re-signed by a
82+
/// TLS-intercepting proxy.
83+
pub const VM_GUEST_PROXY_CA_PATH: &str = "/opt/openshell/tls/proxy-ca.pem";
84+
85+
/// Guest path for the driver-authored supervisor argument list in VM sandboxes.
86+
///
87+
/// Podman and Kubernetes build the supervisor's command line directly; the VM
88+
/// guest init script execs a fixed argv, so driver-owned arguments travel
89+
/// through this file instead. The driver writes it into the overlay upperdir
90+
/// on every launch — empty when it has no arguments to pass — so a sandbox
91+
/// image can neither forge entries nor shadow the driver's copy, and the
92+
/// guest appends exactly what it finds there and nothing else.
93+
pub const VM_GUEST_SUPERVISOR_ARGS_PATH: &str = "/opt/openshell/supervisor-args";
6894
pub const VM_UMOCI_PATH: &str = "/opt/openshell/bin/umoci";
6995
pub const VM_SANDBOX_OWNER_NORMALIZED_MARKER: &str = "/opt/openshell/.sandbox-owner-normalized";
7096

@@ -103,6 +129,9 @@ mod tests {
103129
VM_GUEST_SANDBOX_TOKEN_PATH,
104130
VM_GUEST_INIT_DROPIN_DIR,
105131
VM_GUEST_INIT_DROPIN_MANIFEST,
132+
VM_GUEST_UPSTREAM_PROXY_AUTH_PATH,
133+
VM_GUEST_PROXY_CA_PATH,
134+
VM_GUEST_SUPERVISOR_ARGS_PATH,
106135
VM_UMOCI_PATH,
107136
VM_SANDBOX_OWNER_NORMALIZED_MARKER,
108137
];

0 commit comments

Comments
 (0)