Skip to content

Commit abeee90

Browse files
mtsmfmclaude
andcommitted
Build OpenShell's supervisor with HTTP/2 for opted-in endpoints
The gateway image builds openshell-supervisor from OpenShell v0.1.5's source with h2-l7.patch (OPENSHELL_PATCH=h2-l7, the default) and carries it at /openshell-supervisor, so the gateway image is also the sandboxes' supervisor image: the WSLc driver is given it, and the Docker driver gets it through SUPERVISOR_IMAGE. The patch offers h2 only to the host:port destinations named in OPENSHELL_HTTP2_ENDPOINTS at build time; every other endpoint keeps http/1.1 only. The build is --locked, and the unpatched source is built in a layer of its own so a patch change rebuilds only the crates it touches. ADR 0020 records the decision (upstream: NVIDIA/OpenShell#2426). Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 32b4ff6 commit abeee90

10 files changed

Lines changed: 1722 additions & 15 deletions

File tree

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# 0021. A patched supervisor for HTTP/2 on opted-in endpoints
2+
3+
- Status: Accepted
4+
- Date: 2026-10-26
5+
6+
## Context
7+
8+
OpenShell v0.1.5's L7 proxy terminates TLS with ALPN `http/1.1` only, so clients that speak only HTTP/2 (gRPC: Modal's `modal` CLI and Python client) cannot use an inspected endpoint, and without inspection a provider's credential is not substituted. Upstream tracks HTTP/2 in NVIDIA/OpenShell#2426, not yet released.
9+
10+
Alternatives: wait for upstream (no Modal until then); `tls: skip` (no credential substitution, and an uninspected credentialed path); offering h2 wherever the prototype could enforce it (every credentialed REST endpoint, GitHub and Anthropic included, would change protocol at once); an `http2` field in the profile and policy schema (the gateway is the unpatched release, and protobuf drops fields it does not know, so the field would never reach the supervisor without patching and building the gateway too).
11+
12+
## Decision
13+
14+
- The gateway image builds `openshell-supervisor` from v0.1.5's source with `internal/openshell/gateway/h2-l7.patch` and serves as the supervisor image for both drivers. The gateway, CLI and `openshell-sandbox` stay the release's
15+
- The patch offers `h2` only to the `host:port` destinations named in `OPENSHELL_HTTP2_ENDPOINTS` when the supervisor is built, and only where every endpoint of the route is inspected REST without SigV4 or body rewriting, the tunnel carries credentials and no middleware applies. Each stream is admitted like an HTTP/1 request
16+
- Only `iwato-modal`'s endpoints are listed (a test keeps the list equal to the profile). Everything else keeps `http/1.1` only
17+
18+
## Consequences
19+
20+
- Modal works with credential substitution; other endpoints behave as upstream
21+
- The gateway image build compiles Rust (the dependencies in a layer the patch does not invalidate) and the patch must be re-applied and re-checked on each OpenShell bump, until upstream ships HTTP/2 and this ADR is superseded
22+
- Adding an HTTP/2 endpoint means a new supervisor build, not a profile change
23+
- A sandbox with the Modal provider can run code on Modal, which has open internet access and the workspace's Modal Secrets: the profile's note says so

‎docs/dev/architecture.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ The draft keeps growing without deleting chunks, so each read fetches only pendi
7070

7171
- `engine.go`, `images.go`: calling the engine's CLI (wslc.exe on Windows; on macOS and Linux the `docker` CLI of a Docker-compatible engine, chosen when `iwato.json` has no `wslc`; [ADR 0002](../adr/0002-macos-docker-compatible-runtime.md)). `internal/dockercli` turns wslc's commands into docker's (only `inspect --format json` differs). On Docker, volumes must be `local` without options and carry the labels the Docker driver's resource admission requires (`attachLabels`); `checks_docker.go` checks the engine (below). Also pulling and building the standard image (`sandboximage/`). What goes in it: [ADR 0004](../adr/0004-standard-image-contents.md). A default development build pulls ghcr.io's `edge` (published by `.github/workflows/image.yml` on pushes to main; PRs that touch the images only build them) only when it has none locally, and updates only on request (`imageMoves`, `UpdateImage`; `GET /api/images` and `POST /api/images/update` in `image_update.go`; in the UI, the settings' **Update the standard image** and the recreate dialog's **Pull the latest image first**). `scripts/build-exe.sh --local-images` builds locally; `--images` (releases, and the published edge exes with `--image-tag edge`) also pulls the gateway image (`pullsGatewayImage`), pinned by digest (`imageDigests`), so it does not move
7272
- `internal/openshell/wslc`: OpenShell's WSLc compute driver (a port of the Docker driver; [ADR 0007](../adr/0007-in-house-wslc-compute-driver.md)). The image convention ([ADR 0005](../adr/0005-sandbox-image-convention.md)) carries the daemon and the Policy Advisor guide, so the driver does not place them at creation (only the daemon for old sandboxes, `-legacy-term` below). The difference that a writable volume may be the working directory itself is kept for images whose WORKDIR is the home (the earlier standard image). `e2e.sh` tests it with a fake wslc (`fakewslc`) together with the official gateway and sandbox
73-
- `internal/openshell/gateway`: the gateway's image and startup. One image for both engines: on WSLc the entrypoint relays the driver socket to Iwato's WSLc driver; on a Docker engine (`Options.DockerSocket`, `COMPUTE_DRIVER=docker`) the gateway runs upstream's Docker driver with `gateway-docker.toml`, on the engine's socket mounted into the container (`/var/run/docker.sock` in the engine's VM; rootless Docker's own socket on Linux, `dockercli.Socket`). The supervisors use the engine's host network and reach the gateway on 127.0.0.1:17670, where its port is published; the driver keeps each sandbox's state in the state volume (`XDG_STATE_HOME`). Sandboxes get `driver_config.docker` with the home volume and a 1 GiB `/dev/shm` tmpfs (`OpenShell.driverConfig`). e2e on Docker: `openshell_docker_e2e_test.go` (`IWATO_E2E_DOCKER=1`, the `docker` job of `e2e.yml`). The Dockerfile checks the OpenShell release tarballs against SHA-256 pinned per release and architecture before extracting them, and both drivers use the official runtime images pinned by digest (`wslc.DefaultSupervisorImage`, `DefaultSandboxImage`; the same refs in `gateway-docker.toml`); [bumping them](development.md#bumping-pinned-versions)
73+
- `internal/openshell/gateway`: the gateway's image and startup. One image for both engines: on WSLc the entrypoint relays the driver socket to Iwato's WSLc driver; on a Docker engine (`Options.DockerSocket`, `COMPUTE_DRIVER=docker`) the gateway runs upstream's Docker driver with `gateway-docker.toml`, on the engine's socket mounted into the container (`/var/run/docker.sock` in the engine's VM; rootless Docker's own socket on Linux, `dockercli.Socket`). The supervisors use the engine's host network and reach the gateway on 127.0.0.1:17670, where its port is published; the driver keeps each sandbox's state in the state volume (`XDG_STATE_HOME`). Sandboxes get `driver_config.docker` with the home volume and a 1 GiB `/dev/shm` tmpfs (`OpenShell.driverConfig`). e2e on Docker: `openshell_docker_e2e_test.go` (`IWATO_E2E_DOCKER=1`, the `docker` job of `e2e.yml`). The Dockerfile checks the OpenShell release tarballs against SHA-256 pinned per release and architecture before extracting them, and both drivers use the official sandbox image pinned by digest (`wslc.DefaultSandboxImage`; the same ref in `gateway-docker.toml`); [bumping them](development.md#bumping-pinned-versions). The supervisor is not the release's: the Dockerfile builds it from v0.1.5's source plus `h2-l7.patch` (HTTP/2 for the `host:port` endpoints opted in with `OPENSHELL_HTTP2_ENDPOINTS`, only `iwato-modal`'s; [openshell.md](openshell.md#the-patched-supervisor-http2-for-opted-in-endpoints-nvidiaopenshell2426), ADR 0021) and the gateway image, which carries it at `/openshell-supervisor`, is the supervisor image for both drivers (`gateway.PatchedSupervisor`; the Docker driver gets it through `SUPERVISOR_IMAGE`). The patch is re-applied on each OpenShell bump
7474

7575
## The in-sandbox daemon
7676

‎docs/dev/development.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,8 @@ Everything the images and the gateway download is pinned; `scripts/check-pins.sh
5151
1. the release tarballs: `ARG OPENSHELL_TAG` in `internal/openshell/gateway/Dockerfile` and its `case` entry, the SHA-256 of the 4 tarballs for x86_64 and aarch64, copied from the release's `*-checksums-sha256.txt` (or `gh api repos/NVIDIA/OpenShell/releases/tags/vX.Y.Z --jq '.assets[] | [.name, .digest] | @tsv'`). A tag with no entry fails the build; `dev` is checked only against the checksums published with it
5252
2. the runtime images: `supervisorImageDigest` and `sandboxImageDigest` in `driver.go`, and the same refs in `gateway-docker.toml` (the Docker driver's). The digest is the tag's index (multi-architecture), the `Docker-Content-Digest` of `HEAD /v2/nvidia/openshell/NAME/manifests/X.Y.Z` on ghcr.io, as `scripts/check-pins.sh` prints on a mismatch (or `docker buildx imagetools inspect ghcr.io/nvidia/openshell/NAME:X.Y.Z`)
5353
3. the Go SDK at the release's commit (`go get github.com/NVIDIA/OpenShell/sdk/go@COMMIT`), and the compute driver protos (`internal/openshell/wslc/README.md`)
54-
4. `go test ./internal/openshell/...` (`TestDockerfilePinsOpenShell`, `TestImageTag`) and `scripts/check-pins.sh`
54+
4. the supervisor patch: rebase `h2-l7.patch` onto the release's commit (the commit hash in the Dockerfile's `supervisor-h2-l7` stage), run OpenShell's `cargo test -p openshell-supervisor-network`, and regenerate it with `git format-patch` ([openshell.md](openshell.md#the-patched-supervisor-http2-for-opted-in-endpoints-nvidiaopenshell2426))
55+
5. `go test ./internal/openshell/...` (`TestDockerfilePinsOpenShell`, `TestImageTag`) and `scripts/check-pins.sh`
5556
- `ubuntu:24.04` (`FROM` in `sandboximage/Dockerfile` and the gateway's): pinned by the digest of the tag's index. Dependabot (`.github/dependabot.yml`) proposes the new digest weekly; by hand, `scripts/check-pins.sh` prints the current one
5657
- Selkies (`sandboximage/Dockerfile`): `SELKIES_VERSION` and the `case` entry per architecture, from the release's asset digests (`gh api repos/selkies-project/selkies/releases/tags/VERSION --jq '.assets[] | [.name, .digest] | @tsv'`). A version or architecture with no hash, or a mismatch, fails the build; `--build-arg SELKIES_DEB_SHA256=` overrides the hash for one build
5758
- CodeMirror and marked (`web/vendor/codemirror.js`, `markdown.js`): change `scripts/codemirror/package.json`, update the lock file (`npm install --package-lock-only` there), run `scripts/build-codemirror.sh` and put the printed hashes in `web/vendor/manifest.json`. The `codemirror` job of `build.yml` rebuilds them from the lock file (with its integrity hashes) and fails unless they are the committed files with the manifest's hashes

‎docs/dev/openshell.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,10 @@ v0.1.5 renames the bootstrap's and descriptor's `outer_fence` to `provisioning_e
3939
## Which runtime images a sandbox runs
4040

4141
Upstream's Docker driver starts every supervisor from the gateway's current `supervisor_image` (`spawn_docker_control_process`; a new gateway removes the old supervisors at startup) and stages its current `openshell-sandbox` into a stopped workload on each start, so a restart is enough. The WSLc driver keeps both per sandbox until it is recreated: the supervisor image's ID is in the sandbox's state (`supervisor-image`, used on every start) and `openshell-sandbox` is copied into the workload only at creation (`stageWorkload`; `rotate` does not restage it). So with a new supervisor image (an OpenShell update, or a gateway image that carries a patched supervisor) existing WSLc sandboxes keep the old one. The state also records the sandbox runtime image's ID (`sandbox-image`); `openshell_runtime.go` compares both with the IDs of the driver's current refs and the card shows **Recreate to update**. Refreshing them on a stopped sandbox's start would be possible (restage the binary, rewrite `supervisor-image`), but only when the stored bootstrap and descriptor are in the new version's format, so the driver does not.
42+
43+
## The patched supervisor: HTTP/2 for opted-in endpoints (NVIDIA/OpenShell#2426)
44+
45+
Upstream's L7 proxy offers only `http/1.1` in ALPN, so gRPC clients (Modal) cannot use an inspected, credentialed endpoint. Iwato builds `openshell-supervisor` from v0.1.5's source plus `internal/openshell/gateway/h2-l7.patch` (ADR 0021). The patch offers `h2` only to the `host:port` destinations listed in `OPENSHELL_HTTP2_ENDPOINTS` when the supervisor is built (the gateway's Dockerfile: `iwato-modal`'s endpoints), and only when every endpoint of the route is inspected REST without SigV4 or body rewriting, the tunnel carries credentials and no middleware applies. Every other destination keeps `http/1.1` only, as upstream. Each h2 stream goes through the same admission as an HTTP/1 request (policy, placeholder rewriting in headers, path and query); a denied stream gets 403 and a `:path` that is not origin-form 400, without affecting the others; a placeholder in a request body or trailer resets the stream.
46+
47+
The list is a build setting, not a profile or policy field, because the gateway is the unpatched release and drops fields it does not know. The patch is the `git format-patch` of one commit on v0.1.5 (`bdb956f2`) and must be re-applied, and re-checked, on each OpenShell bump.
48+

‎internal/openshell/gateway/Dockerfile‎

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,14 @@
88
# build of main. For dev the sandbox and supervisor images are built here
99
# too (--target sandbox / supervisor), as the official images of main are
1010
# not tagged.
11+
#
12+
# OPENSHELL_PATCH picks the supervisor: release (the release's binary), or
13+
# h2-l7 (v0.1.5's source with h2-l7.patch, which adds HTTP/2 for opted-in
14+
# endpoints, built here with Rust; only for v0.1.5). The gateway image
15+
# carries the same supervisor at /openshell-supervisor, so it serves as the
16+
# sandboxes' supervisor image too (gateway.PatchedSupervisor).
1117
ARG OPENSHELL_TAG=v0.1.5
18+
ARG OPENSHELL_PATCH=h2-l7
1219

1320
FROM ubuntu:24.04@sha256:68ee22004a9109b44973313b02ff7b1de250f1de0f5ecb628d73f579aaff33ac AS base
1421
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl socat openssh-client \
@@ -48,18 +55,60 @@ RUN set -e; base=https://github.com/NVIDIA/OpenShell/releases/download/${OPENSHE
4855
tar xzf "$f.tar.gz" -C /out; done; \
4956
cd /; rm -rf /dl; /out/openshell-gateway --version; /out/openshell --version
5057

58+
FROM fetch AS supervisor-release
59+
60+
# v0.1.5's source (the tag's commit) with the patch applied, built as the
61+
# release builds openshell-supervisor (crates' version set to the release's,
62+
# default features, glibc) with the repository's toolchain, 1.95.0. rustup-init
63+
# is pinned by SHA-256 per architecture; rustup checks the toolchain itself.
64+
FROM base AS supervisor-h2-l7
65+
ARG OPENSHELL_TAG
66+
RUN apt-get update && apt-get install -y --no-install-recommends git build-essential clang cmake pkg-config \
67+
&& rm -rf /var/lib/apt/lists/*
68+
RUN set -e; [ "$OPENSHELL_TAG" = v0.1.5 ] || { echo "h2-l7.patch is for OpenShell v0.1.5, not $OPENSHELL_TAG" >&2; exit 1; }; \
69+
a=$(uname -m); case "$a" in \
70+
x86_64) want=dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 ;; \
71+
aarch64) want=15f6e4ce9f583b929c996c91562bad6d4454f3281de858b02cdfdef615fac433 ;; \
72+
*) echo "unsupported architecture $a" >&2; exit 1 ;; \
73+
esac; \
74+
curl -fsSL -o /tmp/rustup-init "https://static.rust-lang.org/rustup/archive/1.29.1/$a-unknown-linux-gnu/rustup-init"; \
75+
echo "$want /tmp/rustup-init" | sha256sum --check --strict -; \
76+
chmod +x /tmp/rustup-init; /tmp/rustup-init -y --no-modify-path --profile minimal --default-toolchain 1.95.0; rm /tmp/rustup-init
77+
ENV PATH=/root/.cargo/bin:$PATH RUSTUP_TOOLCHAIN=1.95.0
78+
WORKDIR /src
79+
# The version is set in Cargo.lock too (only the workspace's crates are at
80+
# 0.0.0), so --locked holds.
81+
RUN git init -q . && git fetch -q --depth 1 https://github.com/NVIDIA/OpenShell.git bdb956f2cd65a635f1294fccf1d138d263bf650f \
82+
&& git checkout -q FETCH_HEAD \
83+
&& sed -i 's/^version = "0\.0\.0"$/version = "0.1.5"/' Cargo.toml Cargo.lock \
84+
&& cargo fetch --locked
85+
# The unpatched build, in a layer the patch does not invalidate: a change to
86+
# the patch rebuilds only the crates it touches and those depending on them.
87+
RUN cargo build --release --locked --offline -p openshell-supervisor --bin openshell-supervisor
88+
COPY h2-l7.patch /tmp/
89+
# OPENSHELL_HTTP2_ENDPOINTS: the destinations offered HTTP/2, iwato-modal's
90+
# endpoints (TestHTTP2EndpointsAreModals).
91+
RUN git apply /tmp/h2-l7.patch \
92+
&& OPENSHELL_HTTP2_ENDPOINTS=api.modal.com:443,api.modal2.com:443 \
93+
cargo build --release --locked --offline -p openshell-supervisor --bin openshell-supervisor \
94+
&& mkdir /out && cp target/release/openshell-supervisor /out/ \
95+
&& rm -rf target && /out/openshell-supervisor --version
96+
97+
FROM supervisor-${OPENSHELL_PATCH} AS supervisor-bin
98+
5199
# As deploy/docker/Dockerfile.sandbox and Dockerfile.supervisor upstream.
52100
FROM scratch AS sandbox
53101
COPY --from=fetch --chmod=0555 /out/openshell-sandbox /openshell-sandbox
54102
USER 65532:65532
55103
ENTRYPOINT ["/openshell-sandbox"]
56104

57105
FROM ubuntu:24.04@sha256:68ee22004a9109b44973313b02ff7b1de250f1de0f5ecb628d73f579aaff33ac AS supervisor
58-
COPY --from=fetch --chmod=0555 /out/openshell-supervisor /openshell-supervisor
106+
COPY --from=supervisor-bin --chmod=0555 /out/openshell-supervisor /openshell-supervisor
59107
ENTRYPOINT ["/openshell-supervisor"]
60108

61109
FROM base AS gateway
62110
COPY --from=fetch /out/openshell-gateway /out/openshell /usr/local/bin/
111+
COPY --from=supervisor-bin --chmod=0555 /out/openshell-supervisor /openshell-supervisor
63112
COPY gateway.toml gateway-docker.toml entrypoint.sh /etc/openshell/
64113
RUN chmod 0755 /etc/openshell/entrypoint.sh
65114
# The CLI inside this container (wslc container exec ... openshell ...)

‎internal/openshell/gateway/entrypoint.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,12 @@ if [ "${COMPUTE_DRIVER:-wslc}" = docker ]; then
3030
export XDG_STATE_HOME=/var/lib/openshell/state
3131
case "${SANDBOX_LABEL:-}" in ''|*[!a-z0-9-]*) echo "invalid SANDBOX_LABEL" >&2; exit 1 ;; esac
3232
sed "s/SANDBOX_LABEL/$SANDBOX_LABEL/" /etc/openshell/gateway-docker.toml >$config
33+
# The supervisor image, when Iwato names one: this image, whose
34+
# supervisor is built with a patch (gateway.PatchedSupervisor).
35+
if [ -n "${SUPERVISOR_IMAGE:-}" ]; then
36+
case "$SUPERVISOR_IMAGE" in *[!A-Za-z0-9._/:@-]*) echo "invalid SUPERVISOR_IMAGE" >&2; exit 1 ;; esac
37+
sed -i "s|^supervisor_image .*|supervisor_image = \"$SUPERVISOR_IMAGE\"|" $config
38+
fi
3339
else
3440
d=/etc/openshell/driver-tls
3541
rm -f /run/openshell/wslc.sock

0 commit comments

Comments
 (0)