Skip to content

Latest commit

 

History

History
467 lines (406 loc) · 36.1 KB

FRICTION_LOG.md

File metadata and controls

467 lines (406 loc) · 36.1 KB

OpenShell friction log

A factual record of every step, error, unclear doc and workaround met while installing and running NVIDIA OpenShell on a Mac, as preparation for the Paved Gate supervisor-middleware integration. Entries are in time order (UTC). Each entry notes whether a claim was verified (observed or read in a primary source, with its URL) or is an assumption.

Environment

Item Value
Host macOS 26.6.2, Apple Silicon (arm64), 16 GB RAM
Package manager Homebrew 6.0.17
Container runtime at start none: no Docker Desktop, Podman or Colima installed
OpenShell docs version read v0.1.2 ("latest")

Log

Addresses in this log: 192.168.65.x is Docker Desktop's built-in internal subnet (host.docker.internal = 192.168.65.254), the same on every install; 127.0.0.1 is loopback. No machine-specific addresses, usernames, paths or credentials are recorded.

2026-10-09T13:14Z · Discovery (documentation only)

  1. The architecture page doesn't mention the extension points. The architecture overview describes the gateway, sandbox, supervisor and compute runtime, but says nothing about supervisor middleware or gateway interceptors. These are documented only under Extensibility, which I found through the docs index (llms.txt), not by navigating from the architecture page. Verified.

    • Impact: someone evaluating whether OpenShell supports third-party content inspection could conclude from the architecture page that it doesn't.
  2. A docs link returns 404. https://docs.nvidia.com/openshell/sandboxes/inference-routing (returned by web search as "Inference Routing") gives HTTP 404, rechecked at 2026-10-09T13:14Z. The current page is how-it-works/inference. Verified.

    • Workaround: use the llms.txt index to find current page paths.
  3. The "privacy router" in marketing doesn't match the docs. The OpenShell product page says the privacy router "keeps sensitive context on-device with local open models and routes to frontier models … only when policy allows". The inference docs and NemoClaw inference options describe a route chosen in configuration (in NemoClaw, during onboarding), with no per-request, sensitivity-based routing and no inspection of prompt content. The issue that planned sensitivity routing, #1043 "Privacy Guard", was closed as not planned on 2026-09-21. Verified (docs and issue); the gap between them is my reading.

  4. The middleware API is explicitly unstable. The supervisor middleware page says "The middleware API is still evolving." The proto package is openshell.middleware.v1, but no compatibility promise or deprecation policy is documented. Verified.

    • Workaround for this integration: vendor the .proto pinned to a release tag.
  5. The only reference middleware is labelled unsafe. NVIDIA's example examples/supervisor-middleware-content-guard (Rust) matches literal strings and "must not be used as a security control". There is no Python example, and no example of a model-backed evaluator. Verified.

2026-10-09T13:14Z – 13:16Z · Install Docker Desktop (prerequisite, not OpenShell itself)

OpenShell on macOS requires Docker Desktop or Engine 28.0 or later (support matrix). This Mac had no container runtime.

  1. brew install --cask docker resolves to the cask docker-desktop 4.88.1,237512. The old name is now an alias. Verified (brew info --cask docker-desktop).
  2. A non-interactive install fails because it needs sudo. The cask moves Docker.app into place, then runs sudo mkdir -p /usr/local/bin (that directory doesn't exist on a fresh Apple Silicon Mac, where Homebrew lives in /opt/homebrew). In a shell with no terminal this fails with sudo: a terminal is required to read the password, and Homebrew rolls back the whole install. Verified (exit 1, full log kept locally).
    • Not an OpenShell bug, but it affects anyone scripting OpenShell setup on a new Mac: the documented prerequisite can't be installed unattended.
    • Workaround: run brew install --cask docker-desktop in an interactive terminal, enter the admin password, then open Docker.app once to accept the Docker Subscription Service Agreement.
    • Also note: the default Homebrew output ends with a long list of Removing: cache-cleanup lines that pushes the error off-screen. Setting HOMEBREW_NO_INSTALL_CLEANUP=1 surfaced it.

2026-10-09T13:23Z · Docker Desktop installed (by the user, interactively)

  1. The user ran brew install --cask docker-desktop in Terminal, entered the admin password and accepted the Docker agreement. Result: Docker 29.8.2 (client and server), Linux VM linux/arm64, 10 CPUs, about 7.7 GiB memory. This meets the 28.0 minimum. Verified (docker version, docker info).

2026-10-09T13:24Z · Install OpenShell v0.1.2 (about 15 s)

  1. Pinned install works. I reviewed the installer before running it (install.sh from main, sha256 15fa05f2ccdb3f51…, 1,632 lines). It supports OPENSHELL_VERSION=<tag> and checks a published sha256 checksum file. Ran OPENSHELL_VERSION=v0.1.2 sh install.sh: exit 0, 13:23:59Z → 13:24:14Z. Verified.

    • Doc gap: the README shows only the unpinned curl … main/install.sh | sh. Pinning with OPENSHELL_VERSION is documented only in the script's own --help text.
  2. On Apple Silicon the installer uses Homebrew, not a standalone binary. It downloads the release formula openshell.rb, creates a local tap nvidia/openshell with brew tap-new, installs openshell 0.1.2 (161.9 MB, 10 files) and starts a brew services LaunchAgent (sh.brew.openshell) that starts at login. Verified.

    • The background-service-at-login behaviour isn't mentioned in the README quick start. Users may not expect a persistent service; stop it with brew services stop openshell.
  3. Install-time warnings (none fatal):

    • Warning: tap-new is a developer command… (the installer's local-tap approach)
    • Warning: Calling post_install is deprecated! Use post_install_steps instead. (from the OpenShell formula)
    • This is a Tier 2 configuration (Homebrew's notice, triggered by an outdated Xcode on this machine, so environmental, not OpenShell)

    Verified (install log). The formula's deprecated post_install will need updating before Homebrew drops it.

  4. The installer registered the local gateway automatically (openshell gateway add https://localhost:17670 --local). openshell status reports Connected, Authenticated (mTLS transport), Version 0.1.2. Verified.

2026-10-09T13:24Z · First sandbox fails to start (ControlSupervisorStartFailed)

  1. The first openshell sandbox create on a fresh macOS install fails. openshell status had just reported Connected. 13 s after openshell sandbox create --name pg-smoke --no-keep -- sh -c '…':
    × sandbox entered error phase while provisioning:
      ControlSupervisorStartFailed: Docker supervisor exited before becoming ready
      … Startup configuration fetch failed after 5 attempts: failed to connect to OpenShell server
    
    Verified. Cause: the Homebrew gateway binds 127.0.0.1:17670. The Docker driver's default grpc_endpoint is https://127.0.0.1:<port> and the supervisor runs with network_mode: host, which under Docker Desktop is the Linux VM, not the Mac. This is the open, triaged bug NVIDIA/OpenShell#3880. It was filed for WSL 2, and a comment from 2026-10-01 reproduces it on macOS 26.6.2 with OpenShell 0.1.2, the same as this machine.
    • Nothing in the docs mentions it: the support matrix lists macOS with Docker Desktop as Supported, and the docs index has no troubleshooting page for this error.
    • Workaround applied (from #3880): add to /opt/homebrew/var/openshell/gateway.toml
      [openshell.drivers.docker]
      grpc_endpoint = "https://host.docker.internal:17670"
      then brew services restart openshell. Sandboxes then start in under 1 s. Verified (13:26Z).
    • I first misread a curl test from a container (exit 56) as "unreachable". It was actually the gateway rejecting a connection without a client certificate, as mutual TLS requires. The TLS handshake to host.docker.internal succeeded.

2026-10-09T13:26Z · Default image and default egress behaviour

  1. The default sandbox image has no curl or python3. It is nvcr.io/nvidia/base/ubuntu:24.04, so even a first egress check needs a custom image (--from <image>). openshell sandbox template list returns "No sandbox templates found". Verified. Workaround: a two-line python:3.12-slim Dockerfile.
  2. A denied connection looks like an OS error, not a policy decision. Under the default policy, Python's urllib gets URLError: [Errno 13] Permission denied for both http:// and https://example.com. The network rules docs say "an HTTP client receives an OpenShell policy_denied response". That appears to apply only to request-stage (L7) denials, while a connection-stage denial surfaces as EACCES. Verified (observed); the interpretation is mine. An agent, or a developer, seeing "Permission denied" has no hint that network policy was the cause.
  3. TLS trust works without changes to the image. The supervisor sets SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, GIT_SSL_CAINFO and NODE_EXTRA_CA_CERTS to its generated CA bundle, so Python's standard library trusts OpenShell's TLS interception. Verified.

2026-10-09T13:27Z – 13:29Z · Reaching a service on the Mac from a sandbox (not yet working)

  1. host.openshell.internal does not resolve inside the sandbox ([Errno -5] No address associated with hostname), even with a policy allowing it. The same happens with host.docker.internal, and with allowed_ips: [192.168.65.0/24 plus the matching Docker Desktop IPv6 range] added for Docker Desktop's private addresses (private destinations are blocked unless listed in allowed_ips, per the network rules docs). Verified.
  2. The real reason is only in the sandbox log. openshell logs <sandbox> shows NET:REFUSE [MED] DENIED host.docker.internal [reason:policy_dns_trusted_gateway_unavailable]. That reason code isn't documented. The source explains it: policy_dns/mod.rs refuses the reserved host-gateway aliases unless a trusted host gateway exists, and proxy.rs detect_trusted_host_gateway() reads it from a host.openshell.internal line in the supervisor container's /etc/hosts. Verified (v0.1.2 source).
  3. The #3880 workaround causes this. openshell-driver-docker/src/lib.rs (docker_supervisor_host_address, around lines 4999–5009) injects the host.openshell.internal and host.docker.internal extra-host entries only when grpc_endpoint is an IPv4 literal or localhost. With grpc_endpoint = "https://host.docker.internal:17670" it returns None, the supervisor gets ExtraHosts=[] (confirmed with docker inspect), and every host alias is refused. Using the VM's host address (https://192.168.65.254:17670) instead would inject the aliases, but the gateway's generated server certificate has no IP SAN for it (SANs include host.docker.internal, host.openshell.internal, 127.0.0.1 and ::1), so TLS verification would fail. Verified (source, docker inspect, openssl x509).
    • Net effect on macOS with Docker Desktop: either sandboxes don't start (default), or they start but can't reach any service on the host (#3880 workaround). That's the setup the docs' own middleware example uses (grpc_endpoint = "http://host.openshell.internal:50051").
    • Next step: enable Docker Desktop host networking (Settings → Resources → Network) and revert grpc_endpoint to the default. A 2026-10-02 comment on #3880 reports that host networking alone makes sandboxes start on macOS.

2026-10-09T13:40Z · Fix: enable Docker Desktop host networking

  1. Host networking fixes both problems, and needs no gateway changes. The user enabled Docker Desktop Settings → Resources → Network → Enable host networking and restarted Docker (settings-store.json: "HostNetworkingEnabled": true). I then removed the #3880 workaround, restoring the default gateway.toml, and restarted the gateway. Verified (13:40Z):

    • a host-network container reaches the gateway on 127.0.0.1:17670
    • a sandbox starts in 0.66 s
    • with a policy allowing host.openshell.internal:8099, the sandbox gets HTTP 200 from a test server on the Mac (also visible in the server's log)
    • https://example.com is still blocked (EACCES)

    This matches the 2026-10-02 comment on #3880. The support matrix doesn't mention that host networking is effectively required on macOS.

2026-10-09T13:41Z – 13:42Z · Supervisor middleware smoke test (Python, gRPC)

NVIDIA's reference middleware is Rust and needs cargo (plus zig for Linux cross-builds). Neither was installed, so I wrote a minimal throwaway Python grpc.aio middleware against the v0.1.2 protos instead (scratch only, not committed). It denies any request body containing a marker string.

  1. Generating stubs works: proto/supervisor_middleware.proto plus its import proto/extension.proto at tag v0.1.2, compiled with grpcio-tools 1.76.0. Verified.

  2. The capability name isn't in the v0.1.2 proto or docs; the protocol version is. MiddlewareManifest.extension (PeerMetadata) is "Required for negotiation". The proto gives no values. The v0.1.2 0.1.0 upgrade guide says to "use protocol 1.0, and advertise their family base capability", but never names that capability. The name is in openshell-core/src/extension_protocol.rs: protocol_version {major: 1, minor: 0} and the capability openshell.supervisor-middleware.contract (both supported and required). Verified (source and docs at v0.1.2).

    Correction (2026-10-09): an earlier version of this entry said neither value was documented. That was wrong. Also, the capability name was documented for a few days: #3352 (merged 2026-09-21) added an extension negotiation page with a base capability table, and a line in the middleware guide naming openshell.supervisor-middleware.contract. Neither is in the v0.1.2 docs. The middleware docs reorganization (#3636, merged 2026-09-24) replaced the middleware guide's sentence with a link to the negotiation page, and a docs refresh (#3705, commit 2026-09-25) removed that page. Verified (commit history). Whether the table was meant to move somewhere else is unknown.

    Update (2026-10-12): reported as #4378 and fixed by #4405 (merged 2026-10-12, 6b7a00e). On main, the Protocol Negotiation section now has a base capability table for all four extension families with an example Describe manifest, each extension field in the protos names its expected values, and the missing metadata and missing version errors name the expected protocol and capability. Verified (docs and proto on main, 2026-10-12). Not yet in a release as of this update.

  3. Registration and policy work as documented. I added to gateway.toml:

    [[openshell.supervisor.middleware]]
    name = "pg-smoke"
    grpc_endpoint = "http://127.0.0.1:50051"
    allow_insecure_transport = true
    max_payload_bytes = 262144
    timeout = "1500ms"

    and to the sandbox policy: network_middlewares: {smoke: {middleware: pg-smoke, endpoints: {include: ["host.openshell.internal"]}}}. After the gateway restart it called Describe, and ValidateConfig was called when the sandbox was created. Verified. 127.0.0.1 works for both the gateway and the supervisors because of host networking (entry 20).

  4. End-to-end allow and deny both work. From the sandbox:

    • A clean POST was allowed and reached upstream (the server logged POST /upload), 12 ms round trip.
    • A POST containing the marker got HTTP 403 and never reached upstream. The agent receives this JSON:
      {"binary":"/usr/local/bin/python3.12","detail":"Request rejected by configured middleware",
       "error":"middleware_denied","host":"host.openshell.internal","layer":"l7","method":"POST",
       "middleware":"smoke","path":"/upload","policy":"fixture_api","port":8099,"reason_code":"pg_smoke_marker"}
      The middleware's free-text reason isn't relayed; the reason_code is. Verified. This is a clear, structured denial, much more helpful than the connection-stage EACCES (entry 15).
  5. RequestContext.originating_process was empty in every EvaluateHttpRequest the middleware received (binary=""), although the gateway knew the binary: it appears in the 403 body above, and the policy matched on binaries. The proto says the field is set "when available". Verified (observed); why it's empty is unknown. A middleware can't make decisions based on the process.

  6. Clean-up: the smoke middleware and test server are stopped, gateway.toml is back to the install default, and there are no sandboxes. Docker Desktop host networking stays on, which the integration needs.

2026-10-09T13:44Z · Isolation check with Docker Desktop host networking enabled

  1. Host networking does not weaken sandbox isolation. It is a Docker-wide setting, so after enabling it I checked that the sandbox boundary is unchanged. One sandbox ran a policy allowing only host.openshell.internal:8099, with two servers on the Mac: :8099 (allowed) and :8098 (not allowed). Probes from inside the sandbox (Python urllib and raw socket):

    Probe from inside the sandbox Result
    http://host.openshell.internal:8099/ (allowed) reached, HTTP 200
    http://host.openshell.internal:8098/ (port not in policy) blocked, EACCES
    host.openshell.internal:17670 (the gateway's own port) blocked, EACCES
    https://example.com/ (public host, not in policy) blocked, EACCES
    1.1.1.1:443 (public IP, raw TCP) blocked, EACCES
    127.0.0.1:8098 and 127.0.0.1:17670 (raw loopback) ECONNREFUSED: the sandbox's own loopback, not the Mac's
    192.168.65.254:8098 (Docker Desktop host IP, raw TCP) blocked, EACCES
    host.docker.internal:8098 does not resolve

    The servers' access logs agree: :8099 logged the sandbox's request, while :8098 logged only my own check from the Mac. Verified (13:44Z).

    Why: docker inspect shows the workload container with NetworkMode=none, Privileged=false, no added capabilities and no published ports. Its only interface is lo plus the kernel's default unconfigured tunnel devices, with no eth0. Only the trusted supervisor container uses NetworkMode=host, and every workload connection goes through it and its policy check. This matches the driver source, which requires the workload's "outer fence" to be network_mode=none (openshell-driver-docker/src/lib.rs, around lines 3008–3015). Verified.

    Host-level side effect, outside OpenShell: with host networking on, any container started with --network host on this Mac (not only OpenShell's supervisor) can reach services the Mac binds to 127.0.0.1. That widens trust in the Docker VM as a whole, not the sandbox boundary. Only run host-network containers you trust; OpenShell sandbox workloads are not host-network containers. Assumption based on Docker Desktop's documented behaviour, consistent with the probe results above.

  2. Default deny holds: every destination not named in the policy was refused (public hostnames, public IPs, other host ports, Docker's host IP and alias), and the connection-stage denial stays EACCES (entry 15). Verified.

  3. Minor CLI friction: openshell sandbox create --upload <file> -- <command> errors with "the argument '--upload ' cannot be used with '[COMMAND]...'". You can't upload a script and run it in one step. Workaround: pass the script inline (python3 -c "$(cat probe.py)") or upload into a kept sandbox and then use sandbox exec. Verified.

2026-10-09T13:52Z – 13:54Z · Phase 2: the Paved Gate adapter against the real gateway

  1. The production adapter works with OpenShell 0.1.2 end to end. I registered paved_gate.integrations.openshell (a Python grpc.aio service implementing SupervisorMiddleware and HttpResponsePreReturn) as paved-gate in gateway.toml, and ran a sandbox through it (network_middlewares selecting host.openshell.internal). The gateway negotiated Describe (protocol 1.0) and called ValidateConfig. Request evaluation (PRE_CREDENTIALS) and the streaming response hook (PRE_RETURN, WHOLE_BODY_BYTES) both worked as the proto describes. request_id and sandbox_id arrive on both sides, so a request can be linked to its response. Verified (13:53Z): a synthetic-PHI POST and an injection-driven follow-up POST got HTTP 403 middleware_denied with our reason_code, and the upstream server received only the clean POST.
  2. The plaintext-mode warning is clear and specific (positive): on startup the gateway logs WARN … extension authentication is DISABLED for this registration by allow_insecure_transport; OpenShell attaches no caller credential and the service cannot distinguish OpenShell from any other network client. Verified. (For production: register with TLS and JWT audience checks; this integration binds to loopback only.)
  3. Cleanup: the adapter and test API are stopped, gateway.toml is back to the install default, and there are no sandboxes.

2026-10-09 · How Paved Gate hooks into OpenShell: documented or inferred?

  1. Documented and supported, with one undocumented detail. Paved Gate plugs into OpenShell through supervisor middleware, a documented extension point:

    • The Extensibility overview lists it as one of four extension points and assigns it "application traffic processing": it "inspects, transforms, or denies allowed HTTP and WebSocket traffic. It runs after policy evaluation and before OpenShell injects provider credentials, so deployments can add content controls and auditing without exposing managed secrets."
    • The Supervisor middleware page says "A middleware service is a gRPC server that implements the services in proto/supervisor_middleware.proto", and documents what it can see, its limits and its fail-closed default.
    • The Configure page documents the [[openshell.supervisor.middleware]] gateway registration and the network_middlewares policy section that this integration uses.
    • The service contract is the proto itself: proto/supervisor_middleware.proto @ v0.1.2.

    What was taken from source rather than the v0.1.2 docs: only the capability that Describe must return, openshell.supervisor-middleware.contract, taken from openshell-core/src/extension_protocol.rs @ v0.1.2. The protocol version (1.0) is in the v0.1.2 upgrade guide. See entry 22, including its correction. Everything else follows the documentation and the proto's own comments. Since #4405 (2026-10-12) the capability is also documented on main (entry 22, update).

    Support level: documented and present in the released v0.1.2 binaries (verified working, entries 23–24 and 30), but explicitly unstable: "The middleware API is still evolving. Future versions will change it…" (source). That's why the protos are pinned to v0.1.2 in this repo.

2026-10-09T14:32Z · Phase 3: the demo in a real sandbox

  1. The default filesystem policy blocks reading files baked into the image. A demo image with COPY agent.py /app/agent.py failed at start with python3: can't open file '/app/agent.py': [Errno 13] Permission denied. Without an explicit filesystem_policy, Landlock allows only OpenShell's baseline paths (the sandbox log says "Enriched policy with baseline filesystem paths"), and /app isn't one of them. The error is a bare EACCES, with nothing naming the filesystem policy. Verified. Workaround: add filesystem_policy: {include_workdir: true, read_only: [/app], read_write: [/tmp]} to the sandbox policy. The default policy page documents the baseline paths, but a first-time user meets this as an unexplained permission error. Assumption: the baseline contents are as documented; I didn't diff them against the source.

  2. The demo works end to end (examples/openshell/run.sh, mock evaluator, 14:32Z). The five requests behaved as designed:

    • the unlisted host was blocked by OpenShell
    • the clean request passed both layers
    • synthetic PHI was allowed by OpenShell and denied by Paved Gate (local detectors, under 1 ms)
    • the injected tool result was flagged at PRE_RETURN
    • the follow-through request was denied

    Only the clean request reached the upstream API. On exit the demo restored gateway.toml byte for byte and restarted the gateway, leaving no sandbox and no open listeners. Verified.

  3. Minor: registering, then unregistering, the middleware costs two gateway restarts per demo run, because "registration changes require a gateway restart" (configure docs). Each takes about 2–3 s here. Registering a middleware dynamically, or per sandbox, would make integrations like this less intrusive. Verified (behaviour), suggestion (fix).

2026-10-09 to 2026-10-11 · Phase 4: the benchmark (evals/openshell/)

  1. Sandbox names are capped at 19 characters, and only the server says so. openshell sandbox create --name pg-bench-u-<unix time> (21 characters) failed with Client specified an invalid argument … name exceeds maximum length (21 > 19). openshell sandbox create --help describes --name only as "Optional sandbox name (auto-generated when omitted)". The benchmark now uses names like pgb-u-573059. Verified. Stating the limit in the help text, or checking it in the CLI before the request, would save a round trip.

  2. A few requests to the allowed host fail in transit, with or without the middleware. Each benchmark arm sends 430 sequential HTTP requests from one sandbox to the one allowed host port. A handful failed inside the sandbox with RemoteDisconnected: Remote end closed connection without response or a 20 s client timeout, then succeeded on retry:

    Run OpenShell only (no middleware selected) + local detectors + local detectors + Jev
    Mock, 2026-10-09 2 of 430 3 of 430 3 of 430
    Live, 2026-10-11 0 of 430 1 of 430 5 of 430

    It happens in the arm with no middleware selected, so it is not the middleware. In one live case the abandoned attempt still reached the middleware later, after the agent had moved on to later requests, so the middleware's log was out of order with the agent's. Verified (counts in the results JSON). Unknown: the cause. The upstream is a local Python ThreadingHTTPServer, which could also be the cause; I did not isolate it. The benchmark retries a failed step up to twice and reports the counts instead of hiding them.

  3. PRE_RETURN runs on every response OpenShell forwards from a selected endpoint, not only tool calls. With network_middlewares selecting host.openshell.internal, every allowed POST's reply ({"stored": true}) went through HttpResponsePreReturn.Evaluate, so the gate scored it too. I had expected the response hook to matter only for tool calls. It matters for cost and latency, because each forwarded response is one more evaluator call (530 Jev calls for 410 requests in the live arm). Verified (audit log). A middleware that only cares about tool output can skip short or known-safe bodies itself, or use a narrower endpoints.include.

  4. Results. The live benchmark (2026-10-11, Jev 1.13.0, synthetic data only) is in RESULTS.md, with the dashboard at index.html. With local detectors and Jev, 2 of 110 attack requests reached the upstream API (both synthetic PHI without identifier patterns), against 90 of 110 with OpenShell alone. The added latency was 112 ms at p50. The paired taint test is in the results: 15 of 30 follow-through requests passed when the sandbox was untainted, and all 15 were denied after an injection was flagged. Verified (results JSON).

2026-10-11T20:08Z – 20:15Z · Testing the fix for #3880 / #4403 (PR #4404)

NVIDIA opened #4403 from the #3880 comment, with the fix #4404 ("fix(docker): pin Docker Desktop host gateway", head dca78bd, one commit on main at eeba0e7). The PR changes only the Docker driver (crates/openshell-driver-docker/src/lib.rs): when grpc_endpoint uses host.docker.internal, the gateway starts a short-lived, confined resolver container (no network, all capabilities dropped, read-only root, the supervisor image) with extra_hosts host.openshell.internal:host-gateway, reads the address Docker wrote into its /etc/hosts, rejects missing, ambiguous, unspecified, multicast, broadcast and metadata addresses, and passes the one address into the existing host.openshell.internal trust pin. Verified (diff).

  1. The fix works on this Mac, with host networking off. Setup, all isolated from the Homebrew install:

    • gateway and CLI built from dca78bd (debug build; --features openshell-server/prebuilt-z3, because the default build links a system libz3 and this Mac has none), with the default supervisor and sandbox images pinned to eeba0e7 through OPENSHELL_IMAGE_TAG (published images for that exact commit exist on ghcr.io; the PR changes no image code)
    • run as a second gateway on 127.0.0.1:17680 with its own XDG_STATE_HOME, PKI and sandbox_label, and [openshell.drivers.docker] grpc_endpoint = "https://host.docker.internal:17680"
    • Docker Desktop host networking off (verified: a --network host container could not reach the Mac's loopback)
    • a policy allowing only host.openshell.internal:18099; two loopback services on the Mac, :18099 and :18098
    • the same setup with a gateway built from the base commit eeba0e7 as the control
    From inside the sandbox Base eeba0e7 PR dca78bd
    Sandbox starts (create to first output) yes, 0.56 s yes, 0.70 s and 0.65 s (two runs)
    host.openshell.internal resolves no (No address associated with hostname) yes
    host.openshell.internal:18099 (allowed) refused, policy_dns_trusted_gateway_unavailable HTTP 200
    host.openshell.internal:18098 (port not in policy) refused (no DNS) blocked, EACCES
    host.openshell.internal:17680 and :17670 (gateway ports) refused (no DNS) blocked, EACCES
    https://example.com, 1.1.1.1:443 blocked, EACCES blocked, EACCES
    192.168.65.254:18098 (Docker Desktop host IP), raw TCP blocked blocked
    198.18.0.2:18098 (the synthetic address the name resolves to), raw TCP not applicable blocked
    host.docker.internal unresolved unresolved (policy_dns_ineligible)
    127.0.0.1:18098 refused (the sandbox's own loopback) refused (the sandbox's own loopback)

    With the PR, the sandbox log shows the pin: Policy DNS mapped host.openshell.internal resolved=192.168.65.254 synthetic=198.18.0.2 ports=18099. The Mac's services logged exactly one request per PR run, the allowed one. docker events (second PR run) shows the resolver container created, started, exited and removed within the same second, and no containers or volumes were left after deleting the sandboxes. The workload container ran with NetworkMode=none, the supervisor with no added capabilities and Privileged=false. Verified (2026-10-11T20:10Z to 20:13Z). Not tested: the resolver's failure paths, Docker Engine on Linux, an IPv6 only engine, a release build.

  2. openshell-gateway generate-certs --output-dir also overwrites the CLI's default gateway credentials. Generating a PKI for the second gateway replaced ~/.config/openshell/gateways/openshell/mtls/{ca.crt,tls.crt,tls.key} with the new client bundle, so the CLI could no longer talk to the Homebrew gateway (invalid peer certificate: BadSignature). The source documents it (certgen.rs: local mode "also copies client materials to $XDG_CONFIG_HOME/openshell/gateways/openshell/mtls/"), but --help describes --output-dir only as "Write PEMs to a filesystem directory instead of Kubernetes Secrets". My own mistake was not isolating XDG_CONFIG_HOME for that one command; I restored the three files from a snapshot taken before the test. Verified. Suggestion: mention the copy in --help, or skip it when the target already holds a different CA.

Status at end of Phase 1 (2026-10-09T13:42Z)

Component Version / state
Docker Desktop 4.88.1 (engine 29.8.2), host networking enabled
OpenShell 0.1.2 via the Homebrew local tap, LaunchAgent sh.brew.openshell, gateway https://localhost:17670 (mTLS)
Sandbox start works (under 1 s) with host networking and the default gateway config
Sandbox → host service works via host.openshell.internal, only on ports the policy allows (entry 27)
Default-deny egress holds under host networking (entries 27–28)
Supervisor middleware (Python gRPC) works: Describe, ValidateConfig and EvaluateHttpRequest with allow and deny