Skip to content

Commit 490055b

Browse files
authored
fix(sandbox): keep boundary connection live under stalled relays (NVIDIA#3642)
* fix(sandbox): keep boundary connection live under stalled relays Stalled relay streams could exhaust the shared HTTP/2 connection window and freeze DNS, exec, and control traffic. Size flow control so the connection window exceeds all streams' windows plus a reserve, and cap relay streams. Recovery no longer replaces a live connection after a single stream failure, force-closes a dead transport before reattaching, and retries while the sandbox still reports the old same-epoch connection as active. The sandbox now reports attach/confirm rejections instead of closing the stream. Closes NVIDIA#3396 Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): warn only on sustained relay stream waits Package managers routinely hold more than the relay budget's worth of connections for short bursts; report only waits that last longer than five seconds. Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): keep aborted channel cached until reattach completes Clearing the channel cache during the reattach retry window let concurrent requests cache and use a connection that never attached or confirmed. Keep the aborted generation installed so they fail fast and wait on the in-progress recovery. Signed-off-by: Drew Newberry <[email protected]> --------- Signed-off-by: Drew Newberry <[email protected]>
1 parent bfd1268 commit 490055b

7 files changed

Lines changed: 675 additions & 67 deletions

File tree

‎architecture/sandbox.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,12 @@ TCP Service, or VM vsock channel. Independent bidirectional `Exchange` RPCs
2727
carry lifecycle, exec, TCP, and forwarding traffic, while one persistent
2828
bidirectional `Mediate` RPC carries multiplexed DNS traffic. General application
2929
UDP is unsupported; UDP DNS remains mediated by the supervisor.
30+
Both ends size HTTP/2 flow control so the connection window exceeds the
31+
per-stream window times the concurrent-stream limit plus a reserve. Relays whose
32+
workload stops reading therefore stall only their own streams and cannot starve
33+
DNS, exec, or control traffic of connection-level credit. The supervisor caps
34+
concurrent TCP relay and pending-accept streams below the stream limit, so new
35+
workload connections queue before control exchanges lose stream slots.
3036
The sandbox probes HTTP/2 connection liveness every five seconds and closes
3137
connections that miss a ten-second acknowledgement deadline. Closing a
3238
connection freezes the owned workload process tree and cancels its stream
@@ -39,6 +45,12 @@ credentials to claim the existing runtime generation. Confirmation resumes the
3945
workload; expiration terminates it. A credential replacement does not displace
4046
the active connection until the new connection is confirmed. Idle healthy
4147
connections remain usable.
48+
A failed stream alone does not trigger reconnection. The supervisor first
49+
reconfirms on the current connection and keeps it if the boundary answers.
50+
Otherwise it closes that transport before replaying attach, and retries while
51+
the sandbox still reports the equal-epoch connection as active, until the
52+
boundary observes the disconnect. The sandbox reports attach and confirm
53+
rejections as typed errors rather than closing the stream.
4254
TCP mediation accepts use the same authenticated transport recovery as process waits. A healthy idle accept has no timeout. An interrupted pending open fails closed, while a replacement accept waits for new workload traffic; decisions and established byte streams are not replayed. Boundary rejections and failed recovery remain terminal to the proxy.
4355

4456
A renewed Sandbox Protocol bearer is authenticated even when its credential epoch is unchanged. The supervisor confirms that bearer on the active physical connection and records its fingerprint only after confirmation succeeds, preserving pending streams and the mediation session. Changing the credential epoch still requires an authenticated replacement connection.

‎crates/openshell-sandbox-backend/src/boundary_protocol.rs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,20 @@ pub const STREAM_STDIN_CLOSED: u8 = 4;
4343
pub const STREAM_NETWORK_DECISION: u8 = 5;
4444
pub const MAX_STREAM_FRAME_BYTES: usize = 64 * 1024;
4545

46+
// Every relay, exec, and control exchange shares one HTTP/2 connection. The
47+
// connection window must exceed what all streams can hold unread, otherwise
48+
// stalled relays starve DNS and control traffic of connection-level credit.
49+
// h2 keeps at least two thirds of `connection - in-flight` advertised, so the
50+
// reserve stays usable even with every stream stalled at its window.
51+
pub const BOUNDARY_MAX_CONCURRENT_STREAMS: u32 = 128;
52+
pub const BOUNDARY_STREAM_WINDOW_BYTES: u32 = 256 * 1024;
53+
pub const BOUNDARY_CONNECTION_WINDOW_RESERVE_BYTES: u32 = 16 * 1024 * 1024;
54+
pub const BOUNDARY_CONNECTION_WINDOW_BYTES: u32 = BOUNDARY_MAX_CONCURRENT_STREAMS
55+
* BOUNDARY_STREAM_WINDOW_BYTES
56+
+ BOUNDARY_CONNECTION_WINDOW_RESERVE_BYTES;
57+
// HTTP/2 caps any flow-control window at 2^31 - 1.
58+
const _: () = assert!(BOUNDARY_CONNECTION_WINDOW_BYTES <= i32::MAX as u32);
59+
4660
/// Capability masks measured from `/proc/<pid>/status` by the `OpenShell`
4761
/// co-located runtime.
4862
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]

0 commit comments

Comments
 (0)