Skip to content

Commit 11fd7ef

Browse files
committed
fix(sandbox): enforce network namespace and proxy policy in SSH sessions (!17)
Closes #24 ## Problem SSH sessions into the sandbox were **not entering the network namespace** or receiving proxy environment variables. This meant every command run via SSH (the only user-facing path) had unrestricted internet access, completely bypassing OPA network policy enforcement. The root cause: the SSH server was started **before** the network namespace and proxy were created in `lib.rs`, so it never received the netns fd or proxy URL. Additionally, **gRPC inference from within the sandbox did not work at all** — even after fixing the netns, multiple issues prevented the Python SDK from reaching the navigator server through the CONNECT proxy. ## Changes ### Sandbox binary **Core fix — reorder startup + thread netns through SSH:** - `lib.rs`: Move netns + proxy creation before SSH server start. Compute `ssh_netns_fd` and `ssh_proxy_url`, pass them to `run_ssh_server()`. - `ssh.rs`: Thread `netns_fd` and `proxy_url` through the full SSH call chain into `spawn_pty_shell()`. Set proxy env vars on the shell command. Call `setns(fd, CLONE_NEWNET)` in `install_pre_exec()`. **Proxy — control plane allowlist + IPv6 socket lookup:** - `proxy.rs`: Connections to the navigator endpoint (derived from `NAVIGATOR_ENDPOINT`) are always allowed without OPA evaluation, logged with `engine=control_plane`. This is infrastructure the sandbox needs to function, not a user-configurable policy. - `procfs.rs`: Extended `parse_proc_net_tcp` to also check `/proc/<pid>/net/tcp6`. gRPC C-core uses `AF_INET6` sockets even for IPv4 connections, so its TCP entries were invisible to the proxy's identity resolver. Also fixed port parsing to use `rsplit_once(':')` for correct IPv6 address handling. **Proxy env vars — lowercase variants for gRPC C-core:** - `process.rs` + `ssh.rs`: Added lowercase `http_proxy`, `https_proxy`, `grpc_proxy` alongside uppercase. gRPC C-core (libgrpc) checks lowercase first and was ignoring the uppercase-only vars. ### Server - `sandbox/mod.rs`: Added `CAP_SYS_PTRACE` to sandbox pod security context. Required for the proxy (root) to read `/proc/<pid>/fd/` of sandbox-user processes for binary identity resolution. ### Python SDK - `inference.py`: Strip `http://`/`https://` scheme from endpoint before passing to `grpc.insecure_channel()`, which expects `host:port`. Default `endpoint` and `sandbox_id` from `NAVIGATOR_ENDPOINT` / `NAVIGATOR_SANDBOX_ID` env vars so `Inference()` works inside sandboxes with no arguments. ### Build / infra - `ci.toml`: Added `--cap-add=SYS_PTRACE` to `mise run sandbox` to mirror the k8s pod capabilities. ### Documentation - `architecture/sandbox.md`: Documented `CAP_SYS_PTRACE` requirement and the full set of proxy env vars (uppercase + lowercase). ## Testing All 29 sandbox unit tests pass. `mise run pre-commit` passes (fmt, clippy, all workspace tests, python lint). E2E verified on a live cluster: | Test | Result | |------|--------| | Proxy env vars (6 vars, upper+lowercase) | PASS | | Blocked endpoints (google, anthropic via curl) | PASS — all denied | | **gRPC inference from SSH session** (`Inference()` with no args, env var defaults) | **PASS** | | Proxy log: navigator requests show `engine=control_plane` | PASS | | Proxy log: blocked requests show `engine=opa` with correct deny reasons | PASS |
1 parent 8f1d85d commit 11fd7ef

10 files changed

Lines changed: 295 additions & 71 deletions

File tree

‎architecture/sandbox.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,12 @@ The implementation lives in `crates/navigator-sandbox/src/sandbox/linux/netns.rs
108108

109109
- `CAP_SYS_ADMIN`: Required for creating namespaces and mount operations
110110
- `CAP_NET_ADMIN`: Required for creating veth pairs and configuring interfaces
111+
- `CAP_SYS_PTRACE`: Required for the CONNECT proxy to read `/proc/<pid>/fd/` and
112+
`/proc/<pid>/exe` of sandbox-user processes. The proxy resolves binary identity by mapping
113+
TCP socket inodes (from `/proc/<entrypoint>/net/tcp{,6}`) to process file descriptors,
114+
then reading the executable symlink. Without this capability, the kernel blocks access to
115+
`/proc/<pid>/fd/` for processes running as a different user (the proxy runs as root,
116+
sandboxed processes run as the `sandbox` user).
111117
- `iproute2` package: Provides the `ip` command for namespace/interface setup
112118

113119
If namespace creation fails (e.g., missing capabilities), the sandbox logs a warning and continues
@@ -118,8 +124,11 @@ production deployments should ensure capabilities are granted.
118124

119125
The sandbox exports proxy configuration to the child process:
120126

121-
- `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY`: Set to `http://10.200.0.1:3128` (with netns) or
122-
`http://127.0.0.1:3128` (without netns)
127+
- `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY`: Uppercase variants for curl, wget, and similar tools
128+
- `http_proxy` / `https_proxy` / `grpc_proxy`: Lowercase variants for gRPC C-core (libgrpc),
129+
which checks lowercase first
130+
131+
All set to `http://10.200.0.1:3128` (with netns) or `http://127.0.0.1:3128` (without netns).
123132

124133
## Process Privileges
125134

‎build/ci.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ fi
2323
docker run ${TTY_FLAG} \
2424
--cap-add=SYS_ADMIN \
2525
--cap-add=NET_ADMIN \
26+
--cap-add=SYS_PTRACE \
2627
-v ${PWD}/dev-sandbox-policy.yaml:/var/navigator/policy.yaml:ro \
2728
-v ${PWD}/tmp:/sandbox/tmp \
2829
-e HOME=/sandbox \

‎crates/navigator-sandbox/src/lib.rs‎

Lines changed: 74 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ pub async fn run_sandbox(
5454
.ok_or_else(|| miette::miette!("No command specified"))?;
5555

5656
// Load policy and initialize OPA engine
57+
let navigator_endpoint_for_proxy = navigator_endpoint.clone();
5758
let (policy, opa_engine) =
5859
load_policy(sandbox_id, navigator_endpoint, rego_policy, rego_data).await?;
5960

@@ -65,28 +66,9 @@ pub async fn run_sandbox(
6566
// Prepare filesystem: create and chown read_write directories
6667
prepare_filesystem(&policy)?;
6768

68-
if let Some(listen_addr) = ssh_listen_addr {
69-
let addr: SocketAddr = listen_addr.parse().into_diagnostic()?;
70-
let policy_clone = policy.clone();
71-
let workdir_clone = workdir.clone();
72-
let secret = ssh_handshake_secret.unwrap_or_default();
73-
tokio::spawn(async move {
74-
if let Err(err) = ssh::run_ssh_server(
75-
addr,
76-
policy_clone,
77-
workdir_clone,
78-
secret,
79-
ssh_handshake_skew_secs,
80-
)
81-
.await
82-
{
83-
tracing::error!(error = %err, "SSH server failed");
84-
}
85-
});
86-
}
87-
8869
// Create network namespace for proxy mode (Linux only)
89-
// This must be created before the proxy so the proxy can bind to the veth IP
70+
// This must be created before the proxy AND SSH server so that SSH
71+
// sessions can enter the namespace for network isolation.
9072
#[cfg(target_os = "linux")]
9173
let netns = if matches!(policy.network.mode, NetworkMode::Proxy) {
9274
match NetworkNamespace::create() {
@@ -137,20 +119,91 @@ pub async fn run_sandbox(
137119
#[cfg(not(target_os = "linux"))]
138120
let bind_addr: Option<SocketAddr> = None;
139121

122+
// Build the control plane allowlist: the navigator endpoint is always
123+
// allowed so sandbox processes can reach the server for inference.
124+
let control_plane_endpoints = navigator_endpoint_for_proxy
125+
.as_deref()
126+
.and_then(proxy::parse_endpoint_url)
127+
.into_iter()
128+
.collect::<Vec<_>>();
129+
140130
Some(
141131
ProxyHandle::start_with_bind_addr(
142132
proxy_policy,
143133
bind_addr,
144134
engine,
145135
cache,
146136
entrypoint_pid.clone(),
137+
control_plane_endpoints,
147138
)
148139
.await?,
149140
)
150141
} else {
151142
None
152143
};
153144

145+
// Compute the proxy URL and netns fd for SSH sessions.
146+
// SSH shell processes need both to enforce network policy:
147+
// - netns_fd: enter the network namespace via setns() so all traffic
148+
// goes through the veth pair (hard enforcement, non-bypassable)
149+
// - proxy_url: set HTTP_PROXY/HTTPS_PROXY/ALL_PROXY env vars so
150+
// cooperative tools (curl, etc.) route through the CONNECT proxy
151+
#[cfg(target_os = "linux")]
152+
let ssh_netns_fd = netns.as_ref().and_then(|ns| ns.ns_fd());
153+
154+
#[cfg(not(target_os = "linux"))]
155+
let ssh_netns_fd: Option<i32> = None;
156+
157+
let ssh_proxy_url = if matches!(policy.network.mode, NetworkMode::Proxy) {
158+
#[cfg(target_os = "linux")]
159+
{
160+
netns.as_ref().map(|ns| {
161+
let port = policy
162+
.network
163+
.proxy
164+
.as_ref()
165+
.and_then(|p| p.http_addr)
166+
.map_or(3128, |addr| addr.port());
167+
format!("http://{}:{port}", ns.host_ip())
168+
})
169+
}
170+
#[cfg(not(target_os = "linux"))]
171+
{
172+
policy
173+
.network
174+
.proxy
175+
.as_ref()
176+
.and_then(|p| p.http_addr)
177+
.map(|addr| format!("http://{addr}"))
178+
}
179+
} else {
180+
None
181+
};
182+
183+
if let Some(listen_addr) = ssh_listen_addr {
184+
let addr: SocketAddr = listen_addr.parse().into_diagnostic()?;
185+
let policy_clone = policy.clone();
186+
let workdir_clone = workdir.clone();
187+
let secret = ssh_handshake_secret.unwrap_or_default();
188+
let proxy_url = ssh_proxy_url;
189+
let netns_fd = ssh_netns_fd;
190+
tokio::spawn(async move {
191+
if let Err(err) = ssh::run_ssh_server(
192+
addr,
193+
policy_clone,
194+
workdir_clone,
195+
secret,
196+
ssh_handshake_skew_secs,
197+
netns_fd,
198+
proxy_url,
199+
)
200+
.await
201+
{
202+
tracing::error!(error = %err, "SSH server failed");
203+
}
204+
});
205+
}
206+
154207
#[cfg(target_os = "linux")]
155208
let mut handle = ProcessHandle::spawn(
156209
program,

‎crates/navigator-sandbox/src/process.rs‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,14 +93,22 @@ impl ProcessHandle {
9393
// The proxy is on 10.200.0.1:3128 (or configured port)
9494
let port = proxy.http_addr.map_or(3128, |addr| addr.port());
9595
let proxy_url = format!("http://10.200.0.1:{port}");
96+
// Both uppercase and lowercase variants: curl/wget use uppercase,
97+
// gRPC C-core (libgrpc) checks lowercase http_proxy/https_proxy.
9698
cmd.env("ALL_PROXY", &proxy_url)
9799
.env("HTTP_PROXY", &proxy_url)
98-
.env("HTTPS_PROXY", &proxy_url);
100+
.env("HTTPS_PROXY", &proxy_url)
101+
.env("http_proxy", &proxy_url)
102+
.env("https_proxy", &proxy_url)
103+
.env("grpc_proxy", &proxy_url);
99104
} else if let Some(http_addr) = proxy.http_addr {
100105
let proxy_url = format!("http://{http_addr}");
101106
cmd.env("ALL_PROXY", &proxy_url)
102107
.env("HTTP_PROXY", &proxy_url)
103-
.env("HTTPS_PROXY", &proxy_url);
108+
.env("HTTPS_PROXY", &proxy_url)
109+
.env("http_proxy", &proxy_url)
110+
.env("https_proxy", &proxy_url)
111+
.env("grpc_proxy", &proxy_url);
104112
}
105113
}
106114

‎crates/navigator-sandbox/src/procfs.rs‎

Lines changed: 39 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,11 @@ pub fn collect_cmdline_paths(pid: u32, stop_pid: u32, exclude: &[PathBuf]) -> Ve
152152
paths
153153
}
154154

155-
/// Parse `/proc/<pid>/net/tcp` to find the socket inode for a given local port.
155+
/// Parse `/proc/<pid>/net/tcp` (and `/proc/<pid>/net/tcp6`) to find the socket
156+
/// inode for a given local port.
157+
///
158+
/// Checks both IPv4 and IPv6 tables because some clients (notably gRPC C-core)
159+
/// use AF_INET6 sockets with IPv4-mapped addresses even for IPv4 connections.
156160
///
157161
/// Format of `/proc/net/tcp`:
158162
/// ```text
@@ -164,41 +168,49 @@ pub fn collect_cmdline_paths(pid: u32, stop_pid: u32, exclude: &[PathBuf]) -> Ve
164168
/// - Inode is field index 9 (0-indexed)
165169
#[cfg(target_os = "linux")]
166170
fn parse_proc_net_tcp(pid: u32, peer_port: u16) -> Result<u64> {
167-
let path = format!("/proc/{pid}/net/tcp");
168-
let content = std::fs::read_to_string(&path).into_diagnostic()?;
169-
170-
for line in content.lines().skip(1) {
171-
let fields: Vec<&str> = line.split_whitespace().collect();
172-
if fields.len() < 10 {
173-
continue;
174-
}
175-
176-
// Parse local_address to extract port
177-
let local_addr = fields[1];
178-
let local_port = match local_addr.split_once(':') {
179-
Some((_, port_hex)) => u16::from_str_radix(port_hex, 16).unwrap_or(0),
180-
None => continue,
171+
// Check IPv4 first (most common), then IPv6.
172+
for suffix in &["tcp", "tcp6"] {
173+
let path = format!("/proc/{pid}/net/{suffix}");
174+
let content = match std::fs::read_to_string(&path) {
175+
Ok(c) => c,
176+
Err(_) => continue,
181177
};
182178

183-
// Check state is ESTABLISHED (01)
184-
let state = fields[3];
185-
if state != "01" {
186-
continue;
187-
}
179+
for line in content.lines().skip(1) {
180+
let fields: Vec<&str> = line.split_whitespace().collect();
181+
if fields.len() < 10 {
182+
continue;
183+
}
184+
185+
// Parse local_address to extract port.
186+
// IPv4 format: AABBCCDD:PORT
187+
// IPv6 format: 00000000000000000000000000000000:PORT
188+
let local_addr = fields[1];
189+
let local_port = match local_addr.rsplit_once(':') {
190+
Some((_, port_hex)) => u16::from_str_radix(port_hex, 16).unwrap_or(0),
191+
None => continue,
192+
};
188193

189-
if local_port == peer_port {
190-
let inode: u64 = fields[9]
191-
.parse()
192-
.map_err(|_| miette::miette!("Failed to parse inode from {}", fields[9]))?;
193-
if inode == 0 {
194+
// Check state is ESTABLISHED (01)
195+
let state = fields[3];
196+
if state != "01" {
194197
continue;
195198
}
196-
return Ok(inode);
199+
200+
if local_port == peer_port {
201+
let inode: u64 = fields[9]
202+
.parse()
203+
.map_err(|_| miette::miette!("Failed to parse inode from {}", fields[9]))?;
204+
if inode == 0 {
205+
continue;
206+
}
207+
return Ok(inode);
208+
}
197209
}
198210
}
199211

200212
Err(miette::miette!(
201-
"No ESTABLISHED TCP connection found for port {} in /proc/{}/net/tcp",
213+
"No ESTABLISHED TCP connection found for port {} in /proc/{}/net/tcp{{,6}}",
202214
peer_port,
203215
pid
204216
))

0 commit comments

Comments
 (0)