Skip to content

Commit ee88761

Browse files
committed
fix(server): restrict compute callback listeners
Signed-off-by: Evan Lezar <[email protected]>
1 parent a5cef7d commit ee88761

6 files changed

Lines changed: 226 additions & 9 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -399,6 +399,7 @@ openshell logs <sandbox-name>
399399
| `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
400400
| Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
401401
| Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route |
402+
| Admin, health, reflection, or HTTP request is denied on a Docker/Podman callback address | Negotiated callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
402403
| Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
403404
| Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
404405
| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |

‎architecture/gateway.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,14 @@ health, metrics, or tunnel routes. The plaintext service router also rejects
3737
browser requests whose Fetch Metadata, Origin, or Referer headers indicate a
3838
cross-origin or sibling-subdomain request.
3939

40+
Docker and Podman may negotiate additional listeners that make the gateway
41+
reachable from their local sandbox network topology. Those listeners accept
42+
only gRPC methods classified as sandbox-callable by the gateway's generated
43+
authorization metadata. They reject user and administrator APIs, health,
44+
reflection, non-callback inference APIs, and HTTP routes before normal request
45+
authentication. The operator-configured primary listener retains the full
46+
multiplexed API surface.
47+
4048
Operators can configure a gateway-wide gRPC request rate limit. The limit is
4149
applied only to gRPC API traffic after protocol multiplexing; health, metrics,
4250
and local sandbox-service HTTP routes are not rate limited by this control.

‎crates/openshell-driver-podman/NETWORKING.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -276,6 +276,9 @@ maps both local callback aliases to that literal. Podman Machine still requests
276276
gateway loopback because its configured address is guest-visible and gvproxy
277277
terminates that route on host loopback. The gateway validates and binds every
278278
accepted callback listener; the primary listener can remain on loopback.
279+
Negotiated callback listeners expose only the gateway's sandbox-callable gRPC
280+
methods. Operator, health, reflection, and HTTP requests must use the primary
281+
listener.
279282

280283
### Layer 3 Inner Sandbox Network Namespace
281284

‎crates/openshell-server/src/lib.rs‎

Lines changed: 33 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -915,8 +915,12 @@ fn allow_plaintext_service_http(
915915
enabled: bool,
916916
listen_addr: SocketAddr,
917917
peer_addr: SocketAddr,
918+
listener_purpose: &GatewayListenerPurpose,
918919
) -> bool {
919-
enabled && listen_addr.ip().is_loopback() && peer_addr.ip().is_loopback()
920+
enabled
921+
&& matches!(listener_purpose, GatewayListenerPurpose::Primary)
922+
&& listen_addr.ip().is_loopback()
923+
&& peer_addr.ip().is_loopback()
920924
}
921925

922926
fn spawn_gateway_connection(
@@ -936,6 +940,7 @@ fn spawn_gateway_connection(
936940
enable_loopback_service_http,
937941
listen_addr,
938942
addr,
943+
&listener_purpose,
939944
) =>
940945
{
941946
if let Err(e) = service
@@ -950,7 +955,12 @@ fn spawn_gateway_connection(
950955
}
951956
}
952957
Ok(ConnectionProtocol::PlainHttp) => {
953-
warn!(client = %addr, listen = %listen_addr, "Rejected plaintext HTTP on non-loopback gateway listener");
958+
warn!(
959+
client = %addr,
960+
listen = %listen_addr,
961+
purpose = ?listener_purpose,
962+
"Rejected plaintext HTTP on gateway listener"
963+
);
954964
}
955965
Ok(ConnectionProtocol::Tls | ConnectionProtocol::Unknown) => {
956966
// acceptor.acceptor() snapshots the current TLS config;
@@ -1465,11 +1475,28 @@ mod tests {
14651475
let peer: SocketAddr = "127.0.0.1:54000".parse().unwrap();
14661476
let wildcard: SocketAddr = "0.0.0.0:8080".parse().unwrap();
14671477
let remote_peer: SocketAddr = "192.0.2.10:54000".parse().unwrap();
1478+
let primary = GatewayListenerPurpose::Primary;
1479+
let callback = GatewayListenerPurpose::ComputeDriverCallback {
1480+
driver_name: "test-driver".to_string(),
1481+
reason: "test callback requirement".to_string(),
1482+
};
14681483

1469-
assert!(allow_plaintext_service_http(true, loopback, peer));
1470-
assert!(!allow_plaintext_service_http(false, loopback, peer));
1471-
assert!(!allow_plaintext_service_http(true, wildcard, peer));
1472-
assert!(!allow_plaintext_service_http(true, loopback, remote_peer));
1484+
assert!(allow_plaintext_service_http(true, loopback, peer, &primary));
1485+
assert!(!allow_plaintext_service_http(
1486+
false, loopback, peer, &primary
1487+
));
1488+
assert!(!allow_plaintext_service_http(
1489+
true, wildcard, peer, &primary
1490+
));
1491+
assert!(!allow_plaintext_service_http(
1492+
true,
1493+
loopback,
1494+
remote_peer,
1495+
&primary
1496+
));
1497+
assert!(!allow_plaintext_service_http(
1498+
true, loopback, peer, &callback
1499+
));
14731500
}
14741501

14751502
#[tokio::test]

‎crates/openshell-server/src/multiplex.rs‎

Lines changed: 175 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
//! to either the gRPC service or HTTP endpoints based on the request headers.
88
99
use bytes::{Bytes, BytesMut};
10-
use http::{Extensions, HeaderValue, Request, Response};
10+
use http::{Extensions, HeaderValue, Request, Response, StatusCode};
1111
use http_body::Body;
1212
use http_body_util::{BodyExt, Full, LengthLimitError, Limited, StreamBody};
1313
use hyper::body::Incoming;
@@ -995,6 +995,38 @@ impl<G, H> MultiplexedService<G, H> {
995995
}
996996
}
997997

998+
fn listener_allows_request(
999+
listener_purpose: Option<&GatewayListenerPurpose>,
1000+
is_grpc: bool,
1001+
path: &str,
1002+
) -> bool {
1003+
match listener_purpose {
1004+
Some(GatewayListenerPurpose::ComputeDriverCallback { .. }) => {
1005+
is_grpc && crate::auth::sandbox_methods::is_sandbox_callable(path)
1006+
}
1007+
Some(GatewayListenerPurpose::Primary) | None => true,
1008+
}
1009+
}
1010+
1011+
fn callback_listener_rejection(is_grpc: bool) -> Response<BoxBody> {
1012+
if is_grpc {
1013+
let response: Response<tonic::body::Body> = tonic::Status::permission_denied(
1014+
"compute-driver callback listeners accept sandbox callback RPCs only",
1015+
)
1016+
.into_http();
1017+
let (parts, body) = response.into_parts();
1018+
let body = body.map_err(Into::into).boxed_unsync();
1019+
Response::from_parts(parts, BoxBody(body))
1020+
} else {
1021+
Response::builder()
1022+
.status(StatusCode::FORBIDDEN)
1023+
.body(boxed_body_from_bytes(Bytes::from_static(
1024+
b"compute-driver callback listeners accept gRPC callbacks only",
1025+
)))
1026+
.expect("static callback listener rejection response must be valid")
1027+
}
1028+
}
1029+
9981030
impl<G, H, GBody, HBody> hyper::service::Service<Request<Incoming>> for MultiplexedService<G, H>
9991031
where
10001032
G: tower::Service<Request<BoxBody>, Response = Response<GBody>> + Clone + Send + 'static,
@@ -1018,6 +1050,15 @@ where
10181050
.get("content-type")
10191051
.is_some_and(|v| v.as_bytes().starts_with(b"application/grpc"));
10201052

1053+
if !listener_allows_request(
1054+
req.extensions().get::<GatewayListenerPurpose>(),
1055+
is_grpc,
1056+
req.uri().path(),
1057+
) {
1058+
let response = callback_listener_rejection(is_grpc);
1059+
return Box::pin(async move { Ok(response) });
1060+
}
1061+
10211062
if is_grpc {
10221063
let method = grpc_method_from_path(req.uri().path());
10231064
let start = Instant::now();
@@ -1208,6 +1249,88 @@ mod tests {
12081249
);
12091250
}
12101251

1252+
fn callback_listener_purpose() -> GatewayListenerPurpose {
1253+
GatewayListenerPurpose::ComputeDriverCallback {
1254+
driver_name: "test-driver".to_string(),
1255+
reason: "test callback requirement".to_string(),
1256+
}
1257+
}
1258+
1259+
#[test]
1260+
fn callback_listener_allows_sandbox_callback_rpcs() {
1261+
let purpose = callback_listener_purpose();
1262+
let callback_paths = [
1263+
"/openshell.v1.OpenShell/ConnectSupervisor",
1264+
"/openshell.v1.OpenShell/RelayStream",
1265+
"/openshell.v1.OpenShell/GetSandboxConfig",
1266+
"/openshell.v1.OpenShell/ReportPolicyStatus",
1267+
"/openshell.v1.OpenShell/PushSandboxLogs",
1268+
"/openshell.v1.OpenShell/GetSandboxProviderEnvironment",
1269+
"/openshell.v1.OpenShell/SubmitPolicyAnalysis",
1270+
"/openshell.v1.OpenShell/RefreshSandboxToken",
1271+
"/openshell.inference.v1.Inference/GetInferenceBundle",
1272+
];
1273+
1274+
for path in callback_paths {
1275+
assert!(
1276+
listener_allows_request(Some(&purpose), true, path),
1277+
"callback listener should allow {path}"
1278+
);
1279+
}
1280+
}
1281+
1282+
#[test]
1283+
fn callback_listener_rejects_non_callback_routes() {
1284+
let purpose = callback_listener_purpose();
1285+
let rejected_grpc_paths = [
1286+
"/grpc.health.v1.Health/Check",
1287+
"/grpc.reflection.v1.ServerReflection/ServerReflectionInfo",
1288+
"/openshell.v1.OpenShell/ListSandboxes",
1289+
"/openshell.v1.OpenShell/DeleteSandbox",
1290+
"/openshell.v1.OpenShell/CreateProvider",
1291+
"/openshell.inference.v1.Inference/GetInferenceRoute",
1292+
"/openshell.inference.v1.Inference/SetInferenceRoute",
1293+
];
1294+
1295+
for path in rejected_grpc_paths {
1296+
assert!(
1297+
!listener_allows_request(Some(&purpose), true, path),
1298+
"callback listener should reject {path}"
1299+
);
1300+
}
1301+
assert!(!listener_allows_request(Some(&purpose), false, "/health"));
1302+
assert!(!listener_allows_request(Some(&purpose), false, "/service"));
1303+
}
1304+
1305+
#[test]
1306+
fn primary_listener_routing_is_unchanged() {
1307+
let primary = GatewayListenerPurpose::Primary;
1308+
let paths = [
1309+
"/grpc.health.v1.Health/Check",
1310+
"/openshell.v1.OpenShell/ListSandboxes",
1311+
"/openshell.inference.v1.Inference/GetInferenceRoute",
1312+
"/health",
1313+
"/service",
1314+
];
1315+
1316+
for path in paths {
1317+
assert!(listener_allows_request(Some(&primary), true, path));
1318+
assert!(listener_allows_request(Some(&primary), false, path));
1319+
assert!(listener_allows_request(None, true, path));
1320+
assert!(listener_allows_request(None, false, path));
1321+
}
1322+
}
1323+
1324+
#[test]
1325+
fn callback_listener_rejections_use_protocol_appropriate_statuses() {
1326+
let grpc = callback_listener_rejection(true);
1327+
assert_eq!(grpc.status(), StatusCode::OK);
1328+
assert_eq!(grpc.headers().get("grpc-status").unwrap(), "7");
1329+
1330+
let http = callback_listener_rejection(false);
1331+
assert_eq!(http.status(), StatusCode::FORBIDDEN);
1332+
}
1333+
12111334
#[derive(Clone)]
12121335
struct PostCommitTestInterceptor;
12131336

@@ -1309,13 +1432,20 @@ mod tests {
13091432
}
13101433

13111434
async fn start_http_server_with_middleware() -> std::net::SocketAddr {
1435+
start_http_server_with_middleware_on_listener(GatewayListenerPurpose::Primary).await
1436+
}
1437+
1438+
async fn start_http_server_with_middleware_on_listener(
1439+
listener_purpose: GatewayListenerPurpose,
1440+
) -> std::net::SocketAddr {
13121441
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
13131442
let addr = listener.local_addr().unwrap();
13141443

13151444
let http_service = crate::http::health_router(test_health_store().await);
13161445
let http_service = request_id_middleware!(http_service);
13171446

13181447
let service = MultiplexedService::new(http_service.clone(), http_service);
1448+
let service = GatewayListenerContextService::new(service, listener_purpose);
13191449

13201450
tokio::spawn(async move {
13211451
loop {
@@ -1334,8 +1464,9 @@ mod tests {
13341464
addr
13351465
}
13361466

1337-
async fn http1_get(
1467+
async fn http1_request(
13381468
addr: std::net::SocketAddr,
1469+
method: &str,
13391470
path: &str,
13401471
headers: &[(&str, &str)],
13411472
) -> Response<Incoming> {
@@ -1349,7 +1480,7 @@ mod tests {
13491480
});
13501481

13511482
let mut builder = Request::builder()
1352-
.method("GET")
1483+
.method(method)
13531484
.uri(format!("http://{addr}{path}"));
13541485
for (k, v) in headers {
13551486
builder = builder.header(*k, *v);
@@ -1358,6 +1489,47 @@ mod tests {
13581489
sender.send_request(req).await.unwrap()
13591490
}
13601491

1492+
async fn http1_get(
1493+
addr: std::net::SocketAddr,
1494+
path: &str,
1495+
headers: &[(&str, &str)],
1496+
) -> Response<Incoming> {
1497+
http1_request(addr, "GET", path, headers).await
1498+
}
1499+
1500+
#[tokio::test]
1501+
async fn callback_listener_filter_is_applied_before_route_dispatch() {
1502+
let addr = start_http_server_with_middleware_on_listener(callback_listener_purpose()).await;
1503+
1504+
let health = http1_get(addr, "/healthz", &[]).await;
1505+
assert_eq!(health.status(), StatusCode::FORBIDDEN);
1506+
1507+
let admin = http1_request(
1508+
addr,
1509+
"POST",
1510+
"/openshell.v1.OpenShell/ListSandboxes",
1511+
&[("content-type", "application/grpc")],
1512+
)
1513+
.await;
1514+
assert_eq!(admin.status(), StatusCode::OK);
1515+
assert_eq!(admin.headers().get("grpc-status").unwrap(), "7");
1516+
1517+
let callback = http1_request(
1518+
addr,
1519+
"POST",
1520+
"/openshell.v1.OpenShell/ConnectSupervisor",
1521+
&[("content-type", "application/grpc")],
1522+
)
1523+
.await;
1524+
assert_ne!(
1525+
callback
1526+
.headers()
1527+
.get("grpc-status")
1528+
.and_then(|value| value.to_str().ok()),
1529+
Some("7")
1530+
);
1531+
}
1532+
13611533
#[tokio::test]
13621534
async fn intercepted_grpc_body_collection_rejects_oversized_body() {
13631535
let oversized = Bytes::from(vec![0_u8; MAX_INTERCEPTED_GRPC_BODY_SIZE + 1]);

‎docs/reference/sandbox-compute-drivers.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,12 @@ It overrides the gateway's configured default runtime class for that sandbox,
107107
while a typed `SandboxTemplate.runtime_class_name` value from the API still
108108
takes precedence.
109109

110+
Docker and Podman callback listeners accept only supervisor callback gRPC
111+
methods. Use the gateway's primary endpoint for CLI, administrator, health,
112+
reflection, inference-route management, and HTTP requests. A
113+
`PermissionDenied` response from one of the sandbox-visible callback addresses
114+
is expected for those requests.
115+
110116
## Docker Driver
111117

112118
[Docker](https://www.docker.com/get-started/)-backed sandboxes run as containers on the gateway host. Use Docker for local development, single-machine gateways, and hosts that already use Docker Desktop or Docker Engine.

0 commit comments

Comments
 (0)