Skip to content

Commit 6c2cafa

Browse files
committed
feat(relay): route forwarding through ForwardTcp
1 parent 2646b8c commit 6c2cafa

42 files changed

Lines changed: 2081 additions & 1200 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,7 +182,7 @@ Expected port: `30051/tcp` (mapped to configurable host port, default 8080; set
182182

183183
Component images (server, sandbox) can reach kubelet via two paths:
184184

185-
**Local/external pull mode** (default local via `mise run cluster`): Local images are tagged to the configured local registry base (default `127.0.0.1:5000/openshell/*`), pushed to that registry, and pulled by k3s via `registries.yaml` mirror endpoint (typically `host.docker.internal:5000`). The `cluster` task pushes prebuilt local tags (`openshell/*:dev`, falling back to `localhost:5000/openshell/*:dev` or `127.0.0.1:5000/openshell/*:dev`).
185+
**Local/external pull mode** (default local via `mise run cluster`): Local images are tagged to the configured local registry base (default `127.0.0.1:5000/openshell/*`), pushed to that registry, and pulled by k3s via `registries.yaml` mirror endpoint (typically `host.docker.internal:5000`). The `cluster` task rebuilds the local gateway image before tagging and pushing it, so a fresh bootstrap should not reuse stale `openshell/gateway:dev` bits from a previous source revision.
186186

187187
Gateway image builds now stage a partial Rust workspace from `deploy/docker/Dockerfile.images`. If cargo fails with a missing manifest under `/build/crates/...`, or an imported symbol exists locally but is missing in the image build, verify that every current gateway dependency crate (including `openshell-driver-kubernetes` and `openshell-ocsf`) is copied into the staged workspace there.
188188

‎architecture/README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ flowchart TB
3838
end
3939
4040
CLI -- "gRPC / HTTPS" --> SERVER
41-
CLI -- "SSH over HTTP CONNECT" --> SERVER
41+
CLI -- "SSH over gRPC ForwardTcp" --> SERVER
4242
SERVER -- "CRUD + Watch" --> DB
4343
SERVER -- "Create / Delete Pods" --> SBX
4444
SUPERVISOR -- "Fetch Policy + Credentials + Inference Bundle" --> SERVER
@@ -129,17 +129,17 @@ The first command installs the CLI. The second command bootstraps the cluster on
129129

130130
For more detail, see [Cluster Bootstrap Architecture](cluster-single-node.md).
131131

132-
### Sandbox Connect (SSH Tunneling)
132+
### Sandbox Connect (SSH Forwarding)
133133

134134
Users can open interactive terminal sessions into running sandboxes. SSH traffic is tunneled through the gateway rather than exposing sandbox pods directly on the network.
135135

136136
The connection flow works as follows:
137137

138138
1. The CLI requests a session token from the gateway.
139-
2. The CLI opens an HTTP CONNECT tunnel to the gateway's SSH tunnel endpoint, passing the token and sandbox identifier.
140-
3. The gateway validates the token, confirms the sandbox is running, resolves the pod's network address, and establishes a TCP connection to the sandbox's embedded SSH server.
141-
4. A cryptographic handshake (HMAC-verified) confirms the gateway's identity to the sandbox.
142-
5. The CLI and sandbox exchange SSH traffic bidirectionally through the tunnel.
139+
2. The CLI opens a bidirectional gRPC `ForwardTcp` stream with `target.ssh`, passing the token and sandbox identifier.
140+
3. The gateway validates the token, confirms the sandbox is ready, and asks the already-connected supervisor to open an SSH-targeted relay.
141+
4. The supervisor connects the relay to the sandbox's embedded SSH server over the local Unix socket.
142+
5. The CLI and sandbox exchange SSH traffic bidirectionally through the gRPC stream and supervisor relay.
143143

144144
This design provides several benefits:
145145

‎architecture/gateway-security.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -234,16 +234,17 @@ The sandbox calls two RPCs over this authenticated channel:
234234
- `GetSandboxSettings` -- fetches the YAML policy that governs the sandbox's behavior.
235235
- `GetSandboxProviderEnvironment` -- fetches provider credentials as environment variables.
236236

237-
## SSH Tunnel Authentication
237+
## SSH Forward Authentication
238238

239-
SSH connections into sandboxes pass through the gateway's HTTP CONNECT tunnel at `/connect/ssh`. This adds a second authentication layer on top of mTLS.
239+
SSH connections into sandboxes pass through the gateway's bidirectional gRPC `ForwardTcp` stream with `target.ssh`. This adds a second authorization layer on top of gateway mTLS.
240240

241-
### Request Headers
241+
### Forward Initialization
242242

243-
| Header | Purpose |
243+
| Field | Purpose |
244244
|---|---|
245-
| `x-sandbox-id` | Identifies the target sandbox |
246-
| `x-sandbox-token` | Session token (created via `CreateSshSession` RPC) |
245+
| `sandbox_id` | Identifies the target sandbox |
246+
| `target.ssh` | Requests the built-in SSH Unix-socket target |
247+
| `authorization_token` | Session token created via `CreateSshSession` |
247248

248249
The gateway validates the token against the stored `SshSession` record and checks:
249250

@@ -269,16 +270,16 @@ The gateway enforces two concurrent connection limits to bound the impact of cre
269270
| Per-token | 10 concurrent tunnels | Limits damage from a single leaked token |
270271
| Per-sandbox | 20 concurrent tunnels | Prevents bypass via creating many tokens for one sandbox |
271272

272-
These limits are tracked in-memory and decremented when tunnels close. Exceeding either limit returns HTTP 429 (Too Many Requests).
273+
These limits are tracked in-memory and decremented when streams close. Exceeding either limit returns gRPC `ResourceExhausted`.
273274

274275
### Supervisor-Initiated Relay Model
275276

276-
The gateway never dials the sandbox. Instead, the sandbox supervisor opens an outbound `ConnectSupervisor` bidirectional gRPC stream to the gateway on startup and keeps it alive for the sandbox lifetime. SSH traffic for `/connect/ssh` (and exec traffic for `ExecSandbox`) rides this same TCP+TLS+HTTP/2 connection as separate multiplexed HTTP/2 streams. The gateway-side registry and `RelayStream` handler live in `crates/openshell-server/src/supervisor_session.rs`; the supervisor-side bridge lives in `crates/openshell-sandbox/src/supervisor_session.rs`.
277+
The gateway never dials the sandbox. Instead, the sandbox supervisor opens an outbound `ConnectSupervisor` bidirectional gRPC stream to the gateway on startup and keeps it alive for the sandbox lifetime. SSH traffic for `ForwardTcp(target.ssh)` and exec traffic for `ExecSandbox` ride this same TCP+TLS+HTTP/2 connection as separate multiplexed HTTP/2 streams. The gateway-side registry and `RelayStream` handler live in `crates/openshell-server/src/supervisor_session.rs`; the supervisor-side bridge lives in `crates/openshell-sandbox/src/supervisor_session.rs`.
277278

278279
Per-connection flow:
279280

280-
1. CLI presents `x-sandbox-id` + `x-sandbox-token` at `/connect/ssh` and passes gateway token validation.
281-
2. Gateway calls `SupervisorSessionRegistry::open_relay(sandbox_id, ...)`, which allocates a `channel_id` (UUID) and sends a `RelayOpen` message to the supervisor over the already-established `ConnectSupervisor` stream. If no session is registered yet, it polls with exponential backoff up to a bounded timeout (30 s for `/connect/ssh`, 15 s for `ExecSandbox`).
281+
1. CLI opens `ForwardTcp` with `TcpForwardInit { sandbox_id, target.ssh, authorization_token }` and passes gateway token validation.
282+
2. Gateway calls `SupervisorSessionRegistry::open_relay_with_target(sandbox_id, SshRelayTarget, ...)`, which allocates a `channel_id` (UUID) and sends a `RelayOpen` message to the supervisor over the already-established `ConnectSupervisor` stream. If no session is registered yet, it polls with exponential backoff up to a bounded timeout.
282283
3. The supervisor opens a new `RelayStream` RPC on the same `Channel` — a new HTTP/2 stream, no new TCP connection and no new TLS handshake. The first `RelayFrame` is a `RelayInit { channel_id }` that claims the pending slot on the gateway.
283284
4. `claim_relay` pairs the gateway-side waiter with the supervisor-side RPC via a `tokio::io::duplex(64 KiB)` pair. Subsequent `RelayFrame::data` frames carry raw SSH bytes in both directions. The supervisor is a dumb byte bridge: it has no protocol awareness of the SSH bytes flowing through.
284285
5. Inside the sandbox pod, the supervisor connects the relay to sshd over a Unix domain socket at `/run/openshell/ssh.sock` (see `crates/openshell-driver-kubernetes/src/main.rs`).

0 commit comments

Comments
 (0)