Skip to content

Commit 1ea895f

Browse files
n1hilityclaude
andcommitted
perf(server): drop per-connection session tokens for TCP forwards
`openshell forward service` minted an SSH session token before every forwarded TCP connection and revoked it afterwards: two store commits per connection. The token added nothing on that path. `ForwardTcp` already authenticates the caller and authorizes it against the sandbox's workspace on every stream before it looks at the token, the relay to the supervisor is opened with the sandbox id and target only, and the token is never forwarded, audited, or visible to the target service. The mechanism exists for `openshell sandbox ssh`, where the process that opens the stream is an ssh ProxyCommand holding nothing but the token. Reusing it per TCP connection put a store write on the connect path and serialized concurrent forwards on commit latency: NVIDIA#3494 measured the symptom, and NVIDIA#3543 made the commits cheaper, but each one still holds SQLite's writer lock for an fsync, so connection setup under a burst stayed linear in the number of concurrent connections. Let `target.tcp` streams omit `authorization_token`. The gateway admits them on the already-authorized principal, counts them against the same per-sandbox connection cap, and touches no store. `target.ssh` streams keep requiring the token. A token supplied with a TCP target is still validated and counted per token, so an older CLI against a new gateway is unchanged. The CLI stops minting and revoking a session per forwarded connection; against a gateway that predates this change it recognizes the `authorization_token is required` rejection once and falls back to per-connection tokens for the rest of that forward. Tests cover token-less TCP admission and slot release, SSH targets still rejected without a token, a supplied token still validated, and the per-sandbox cap for token-less forwards. CLI integration tests run `service_forward_tcp` against a mock gateway: token-less inits echo data with no CreateSshSession or RevokeSshSession call, and a gateway that rejects the empty token is detected once, after which every connection in that forward mints and revokes its own token. Architecture and security docs describe which targets carry a token, and the per-token connection limit now reads 3, matching the gateway. Signed-off-by: Jason T. Greene <[email protected]> (cherry picked from commit f4a67c6) Carried onto v0.0.116-rhaiv.0 from NVIDIA#3734 (f4a67c6, open upstream). Downstream adaptations only: `TcpForwardInit` here still has `sandbox_id` rather than `sandbox`/`workspace`, `RevokeSshSessionRequest` has no `allow_missing`, and the CLI test module imports match this tree. Co-Authored-By: Claude Opus 5 <[email protected]> Signed-off-by: Jason T. Greene <[email protected]>
1 parent b65f116 commit 1ea895f

5 files changed

Lines changed: 543 additions & 72 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -978,10 +978,15 @@ status, the gateway also bounds how long it waits for the trailing channel close
978978

979979
`ForwardTcp` is the client-facing byte stream for SSH and service forwarding.
980980
The first frame is a `TcpForwardInit` that carries the workspace-scoped sandbox
981-
name, an authorization token from `CreateSshSession`, and an explicit target:
982-
`target.ssh` for the sandbox SSH socket or `target.tcp` for a loopback service
983-
inside the sandbox. The gateway validates the token and sandbox readiness,
984-
sends a targeted `RelayOpen` to the supervisor, then bridges
981+
name and an explicit target: `target.ssh` for the sandbox SSH socket or
982+
`target.tcp` for a loopback service inside the sandbox. The gateway authorizes
983+
the caller's principal against the sandbox's workspace on every stream. SSH
984+
targets additionally require the `authorization_token` issued by
985+
`CreateSshSession`, because the process that opens them is an ssh
986+
`ProxyCommand` that holds nothing else. TCP targets carry no token, so a
987+
service forward costs no store access per connection and only the in-memory
988+
per-sandbox connection cap applies. The gateway then checks sandbox readiness,
989+
sends a targeted `RelayOpen` to the supervisor, and bridges
985990
`TcpForwardFrame::Data` to `RelayFrame::Data` until either side closes.
986991

987992
Browser service URLs use the same supervisor relay path after host-based

‎crates/openshell-cli/src/run.rs‎

Lines changed: 131 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1707,6 +1707,9 @@ pub async fn service_forward_tcp(
17071707

17081708
let sandbox_id = sandbox.object_id().to_string();
17091709
let (fatal_tx, mut fatal_rx) = tokio::sync::mpsc::channel::<String>(1);
1710+
// Set once this forward learns that the gateway predates principal-authorized
1711+
// TCP forwards and still requires a `CreateSshSession` token per connection.
1712+
let legacy_session_tokens = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
17101713
let mut health_check = tokio::time::interval(Duration::from_secs(2));
17111714
health_check.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
17121715
loop {
@@ -1729,25 +1732,16 @@ pub async fn service_forward_tcp(
17291732
let target_host = target_host.to_string();
17301733
let service_id = format!("service-forward:{name}:{target_host}:{target_port}");
17311734
let fatal_tx = fatal_tx.clone();
1735+
let legacy_session_tokens = legacy_session_tokens.clone();
17321736
tokio::spawn(async move {
1733-
let token = match create_forward_session_token(&mut client, &sandbox_id).await {
1734-
Ok(token) => token,
1735-
Err(err) => {
1736-
tracing::warn!(peer = %peer, error = %err, "service forward session creation failed");
1737-
if err.fatal {
1738-
let _ = fatal_tx.send(err.message).await;
1739-
}
1740-
return;
1741-
}
1742-
};
17431737
if let Err(err) = forward_one_tcp_connection(
17441738
&mut client,
17451739
socket,
17461740
sandbox_id,
17471741
target_host,
17481742
target_port,
17491743
service_id,
1750-
token.clone(),
1744+
legacy_session_tokens,
17511745
)
17521746
.await
17531747
{
@@ -1756,9 +1750,6 @@ pub async fn service_forward_tcp(
17561750
let _ = fatal_tx.send(err.message).await;
17571751
}
17581752
}
1759-
let _ = client
1760-
.revoke_ssh_session(RevokeSshSessionRequest { token })
1761-
.await;
17621753
});
17631754
}
17641755
}
@@ -1778,6 +1769,14 @@ async fn create_forward_session_token(
17781769
Ok(response.into_inner().token)
17791770
}
17801771

1772+
/// Older gateways reject a token-less `ForwardTcp` init with this
1773+
/// `Unauthenticated` status; newer ones authorize TCP targets on the caller's
1774+
/// principal and only demand a token for SSH targets.
1775+
fn forward_requires_session_token(status: &Status) -> bool {
1776+
status.code() == Code::Unauthenticated
1777+
&& status.message().contains("authorization_token is required")
1778+
}
1779+
17811780
async fn fetch_ready_sandbox_for_forward(
17821781
client: &mut crate::tls::GrpcClient,
17831782
name: &str,
@@ -1875,36 +1874,115 @@ async fn forward_one_tcp_connection(
18751874
target_host: String,
18761875
target_port: u16,
18771876
service_id: String,
1878-
authorization_token: String,
1877+
legacy_session_tokens: std::sync::Arc<std::sync::atomic::AtomicBool>,
18791878
) -> std::result::Result<(), ForwardTcpConnectionError> {
1880-
use tokio::io::{AsyncReadExt, AsyncWriteExt};
1879+
let mut init = TcpForwardInit {
1880+
sandbox_id: sandbox_id.clone(),
1881+
service_id,
1882+
target: Some(tcp_forward_init::Target::Tcp(TcpRelayTarget {
1883+
host: target_host,
1884+
port: u32::from(target_port),
1885+
})),
1886+
// The gateway authorizes TCP forwards on this client's credentials for
1887+
// every stream, so no per-connection session token is minted unless the
1888+
// gateway turns out to predate that (`forward_requires_session_token`),
1889+
// which this forward remembers in `legacy_session_tokens`.
1890+
authorization_token: String::new(),
1891+
};
1892+
1893+
let mut session_token = None;
1894+
if legacy_session_tokens.load(std::sync::atomic::Ordering::Relaxed) {
1895+
match create_forward_session_token(client, &sandbox_id).await {
1896+
Ok(token) => {
1897+
init.authorization_token.clone_from(&token);
1898+
session_token = Some(token);
1899+
}
1900+
Err(err) => {
1901+
drain_and_shutdown_local_socket(socket).await;
1902+
return Err(err);
1903+
}
1904+
}
1905+
}
1906+
1907+
let opened = match open_forward_tcp_stream(client, init.clone()).await {
1908+
Ok(opened) => opened,
1909+
Err(status) if session_token.is_none() && forward_requires_session_token(&status) => {
1910+
tracing::info!(
1911+
"gateway requires an SSH session token per forwarded connection; \
1912+
minting one per connection for the rest of this forward"
1913+
);
1914+
legacy_session_tokens.store(true, std::sync::atomic::Ordering::Relaxed);
1915+
let token = match create_forward_session_token(client, &sandbox_id).await {
1916+
Ok(token) => token,
1917+
Err(err) => {
1918+
drain_and_shutdown_local_socket(socket).await;
1919+
return Err(err);
1920+
}
1921+
};
1922+
init.authorization_token.clone_from(&token);
1923+
session_token = Some(token);
1924+
match open_forward_tcp_stream(client, init).await {
1925+
Ok(opened) => opened,
1926+
Err(status) => {
1927+
drain_and_shutdown_local_socket(socket).await;
1928+
revoke_forward_session_token(client, session_token).await;
1929+
return Err(ForwardTcpConnectionError::from_status(status));
1930+
}
1931+
}
1932+
}
1933+
Err(status) => {
1934+
drain_and_shutdown_local_socket(socket).await;
1935+
revoke_forward_session_token(client, session_token).await;
1936+
return Err(ForwardTcpConnectionError::from_status(status));
1937+
}
1938+
};
1939+
1940+
let result = bridge_local_socket_to_forward_stream(socket, opened).await;
1941+
revoke_forward_session_token(client, session_token).await;
1942+
result
1943+
}
1944+
1945+
/// An open `ForwardTcp` stream: the sender for local-to-gateway frames and the
1946+
/// gateway-to-local response stream.
1947+
type OpenForwardTcpStream = (
1948+
tokio::sync::mpsc::Sender<TcpForwardFrame>,
1949+
tonic::Streaming<TcpForwardFrame>,
1950+
);
1951+
1952+
async fn open_forward_tcp_stream(
1953+
client: &mut crate::tls::GrpcClient,
1954+
init: TcpForwardInit,
1955+
) -> std::result::Result<OpenForwardTcpStream, Status> {
18811956
use tokio_stream::wrappers::ReceiverStream;
18821957

18831958
let (tx, rx) = tokio::sync::mpsc::channel::<TcpForwardFrame>(16);
18841959
tx.send(TcpForwardFrame {
18851960
payload: Some(openshell_core::proto::tcp_forward_frame::Payload::Init(
1886-
TcpForwardInit {
1887-
sandbox_id,
1888-
service_id,
1889-
target: Some(tcp_forward_init::Target::Tcp(TcpRelayTarget {
1890-
host: target_host,
1891-
port: u32::from(target_port),
1892-
})),
1893-
authorization_token,
1894-
},
1961+
init,
18951962
)),
18961963
})
18971964
.await
1898-
.map_err(|_| ForwardTcpConnectionError::transient("failed to initialize forward stream"))?;
1965+
.map_err(|_| Status::internal("failed to initialize forward stream"))?;
1966+
let response = client
1967+
.forward_tcp(ReceiverStream::new(rx))
1968+
.await?
1969+
.into_inner();
1970+
Ok((tx, response))
1971+
}
18991972

1900-
let mut response = match client.forward_tcp(ReceiverStream::new(rx)).await {
1901-
Ok(response) => response.into_inner(),
1902-
Err(status) => {
1903-
let err = ForwardTcpConnectionError::from_status(status);
1904-
drain_and_shutdown_local_socket(socket).await;
1905-
return Err(err);
1906-
}
1907-
};
1973+
async fn revoke_forward_session_token(client: &mut crate::tls::GrpcClient, token: Option<String>) {
1974+
if let Some(token) = token {
1975+
let _ = client
1976+
.revoke_ssh_session(RevokeSshSessionRequest { token })
1977+
.await;
1978+
}
1979+
}
1980+
1981+
async fn bridge_local_socket_to_forward_stream(
1982+
socket: tokio::net::TcpStream,
1983+
(tx, mut response): OpenForwardTcpStream,
1984+
) -> std::result::Result<(), ForwardTcpConnectionError> {
1985+
use tokio::io::{AsyncReadExt, AsyncWriteExt};
19081986

19091987
let (mut local_read, mut local_write) = socket.into_split();
19101988

@@ -5966,11 +6044,12 @@ fn format_endpoint(endpoint: &openshell_core::proto::NetworkEndpoint) -> String
59666044
mod tests {
59676045
use super::{
59686046
PolicyGetView, ProvisioningStep, build_sandbox_resource_limits,
5969-
dockerfile_sources_supported_for_gateway, format_endpoint, format_log_line, git_sync_files,
5970-
has_main_process_result, parse_cli_setting_value, parse_credential_expiry_cli_value,
5971-
parse_driver_config_json, parse_secret_material_env_pairs, policy_revision_list_json,
5972-
policy_revision_to_json, provisioning_timeout_message, ready_false_condition_message,
5973-
resolve_from, sandbox_should_persist, sandbox_upload_plan, service_endpoint_to_json,
6047+
dockerfile_sources_supported_for_gateway, format_endpoint, format_log_line,
6048+
forward_requires_session_token, git_sync_files, has_main_process_result,
6049+
parse_cli_setting_value, parse_credential_expiry_cli_value, parse_driver_config_json,
6050+
parse_secret_material_env_pairs, policy_revision_list_json, policy_revision_to_json,
6051+
provisioning_timeout_message, ready_false_condition_message, resolve_from,
6052+
sandbox_should_persist, sandbox_upload_plan, service_endpoint_to_json,
59746053
service_expose_status_error, service_url_for_gateway, workspace_member_to_json,
59756054
};
59766055
use crate::TEST_ENV_LOCK;
@@ -7142,4 +7221,17 @@ mod tests {
71427221
let log = log_line("OCSF", "ocsf", message, "sandbox", &[]);
71437222
assert!(format_log_line(&log).ends_with(message));
71447223
}
7224+
7225+
#[test]
7226+
fn forward_requires_session_token_matches_only_the_legacy_gateway_error() {
7227+
assert!(forward_requires_session_token(&Status::unauthenticated(
7228+
"authorization_token is required for ForwardTcp"
7229+
)));
7230+
assert!(!forward_requires_session_token(&Status::unauthenticated(
7231+
"SSH session token not found"
7232+
)));
7233+
assert!(!forward_requires_session_token(&Status::permission_denied(
7234+
"authorization_token is required for ForwardTcp"
7235+
)));
7236+
}
71457237
}

0 commit comments

Comments
 (0)