Skip to content

Commit 0f949a6

Browse files
authored
feat(inference): add sandbox-system inference route for platform-level inference (#209)
* feat(inference): add sandbox-system inference route for platform-level inference Add a separate system-level inference route that the sandbox supervisor can use in-process for platform functions (e.g., embedded agent harness for policy analysis), distinct from the user-facing inference.local endpoint. The system route is accessed via an in-process API on the supervisor, ensuring userland code in the sandbox netns cannot reach it. - Extend proto with route_name fields on Set/Get inference messages - Add ResolvedRoute.name field to the router for route segregation - Server resolves both user and sandbox-system routes in bundles - Sandbox partitions routes into user/system caches on refresh - Expose InferenceContext::system_inference() in-process API - CLI --sandbox flag targets the system route on set/get/update - Integration tests using mock:// routes for the full in-process path Closes #207 * refactor(cli): rename --sandbox flag to --system for inference commands The --sandbox flag could be misread as targeting a user-level sandbox operation. Rename to --system to clearly indicate it configures the platform-level system inference route. * style(cli): collapse short if-else per rustfmt 1.94 --------- Co-authored-by: John Myers <[email protected]>
1 parent 3a1fec0 commit 0f949a6

11 files changed

Lines changed: 680 additions & 63 deletions

File tree

‎architecture/inference-routing.md‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -225,12 +225,53 @@ Validation at load time requires either `api_key` or `api_key_env` to resolve, a
225225
| `503` | Upstream unavailable (timeout or connection failure) |
226226
| `413` | Request body exceeds 10 MiB buffer limit |
227227

228+
## System Inference Route
229+
230+
In addition to the user-facing `inference.local` route, the gateway supports a second managed route named `sandbox-system` for platform system functions (e.g. an embedded agent harness for policy analysis).
231+
232+
### Key differences from user inference
233+
234+
| Aspect | User (`inference.local`) | System (`sandbox-system`) |
235+
|--------|--------------------------|---------------------------|
236+
| **Consumer** | Agent code inside sandbox | Supervisor binary only |
237+
| **Access** | Proxy-intercepted CONNECT | In-process API on `InferenceContext` |
238+
| **Network surface** | HTTPS to `inference.local:443` | None -- function call |
239+
| **Route cache** | `InferenceContext.routes` | `InferenceContext.system_routes` |
240+
241+
### In-process API
242+
243+
`InferenceContext::system_inference()` provides the supervisor with direct access to inference using the system routes. It calls `Router::proxy_with_candidates()` with the system route cache -- the same backend proxy logic used for user inference, but without any CONNECT/TLS overhead.
244+
245+
```rust
246+
ctx.system_inference(
247+
"openai_chat_completions",
248+
"POST",
249+
"/v1/chat/completions",
250+
headers,
251+
body,
252+
).await
253+
```
254+
255+
### Access control
256+
257+
The system route is not exposed through the CONNECT proxy. The supervisor runs in the host network namespace and calls the router directly. User processes are in an isolated sandbox network namespace and cannot reach the in-process API.
258+
259+
### Bundle delivery
260+
261+
Both routes are included in `GetInferenceBundleResponse.routes` (which is `repeated ResolvedRoute`). The sandbox partitions routes by `ResolvedRoute.name` during `bundle_to_resolved_routes()`: routes named `sandbox-system` go to the system cache, everything else goes to the user cache. Both caches are refreshed on the same polling interval.
262+
263+
### Storage
264+
265+
The system route is stored as a separate `InferenceRoute` record in the gateway store with `name = "sandbox-system"`. The `SetClusterInferenceRequest.route_name` field selects which route to target (empty string defaults to `inference.local`).
266+
228267
## CLI Surface
229268

230269
Cluster inference commands:
231270

232-
- `openshell cluster inference set --provider <name> --model <id>` -- configures cluster inference by referencing a provider record name
233-
- `openshell cluster inference get` -- displays current cluster inference configuration
271+
- `openshell cluster inference set --provider <name> --model <id>` -- configures user-facing cluster inference
272+
- `openshell cluster inference set --system --provider <name> --model <id>` -- configures system inference
273+
- `openshell cluster inference get` -- displays both user and system inference configuration
274+
- `openshell cluster inference get --system` -- displays only the system inference configuration
234275

235276
The `--provider` flag references a provider record name (not a provider type). The provider must already exist in the cluster and have a supported inference type (`openai`, `anthropic`, or `nvidia`).
236277

‎crates/navigator-cli/src/main.rs‎

Lines changed: 34 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -795,6 +795,12 @@ enum InferenceCommands {
795795
/// Model identifier to force for generation calls.
796796
#[arg(long)]
797797
model: String,
798+
799+
/// Configure the system inference route instead of the user-facing
800+
/// route. System inference is used by platform functions (e.g. the
801+
/// agent harness) and is not accessible to user code.
802+
#[arg(long)]
803+
system: bool,
798804
},
799805

800806
/// Update gateway-level inference configuration (partial update).
@@ -806,10 +812,19 @@ enum InferenceCommands {
806812
/// Model identifier (unchanged if omitted).
807813
#[arg(long)]
808814
model: Option<String>,
815+
816+
/// Target the system inference route.
817+
#[arg(long)]
818+
system: bool,
809819
},
810820

811821
/// Get gateway-level inference provider and model.
812-
Get,
822+
Get {
823+
/// Show the system inference route instead of the user-facing route.
824+
/// When omitted, both routes are displayed.
825+
#[arg(long)]
826+
system: bool,
827+
},
813828
}
814829

815830
#[derive(Subcommand, Debug)]
@@ -1379,20 +1394,33 @@ async fn main() -> Result<()> {
13791394
let mut tls = tls.with_gateway_name(&ctx.name);
13801395
apply_edge_auth(&mut tls, &ctx.name);
13811396
match command {
1382-
InferenceCommands::Set { provider, model } => {
1383-
run::gateway_inference_set(endpoint, &provider, &model, &tls).await?;
1397+
InferenceCommands::Set {
1398+
provider,
1399+
model,
1400+
system,
1401+
} => {
1402+
let route_name = if system { "sandbox-system" } else { "" };
1403+
run::gateway_inference_set(endpoint, &provider, &model, route_name, &tls)
1404+
.await?;
13841405
}
1385-
InferenceCommands::Update { provider, model } => {
1406+
InferenceCommands::Update {
1407+
provider,
1408+
model,
1409+
system,
1410+
} => {
1411+
let route_name = if system { "sandbox-system" } else { "" };
13861412
run::gateway_inference_update(
13871413
endpoint,
13881414
provider.as_deref(),
13891415
model.as_deref(),
1416+
route_name,
13901417
&tls,
13911418
)
13921419
.await?;
13931420
}
1394-
InferenceCommands::Get => {
1395-
run::gateway_inference_get(endpoint, &tls).await?;
1421+
InferenceCommands::Get { system } => {
1422+
let route_name = if system { Some("sandbox-system") } else { None };
1423+
run::gateway_inference_get(endpoint, route_name, &tls).await?;
13961424
}
13971425
}
13981426
}

‎crates/navigator-cli/src/run.rs‎

Lines changed: 84 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2789,20 +2789,28 @@ pub async fn gateway_inference_set(
27892789
server: &str,
27902790
provider_name: &str,
27912791
model_id: &str,
2792+
route_name: &str,
27922793
tls: &TlsOptions,
27932794
) -> Result<()> {
27942795
let mut client = grpc_inference_client(server, tls).await?;
27952796
let response = client
27962797
.set_cluster_inference(SetClusterInferenceRequest {
27972798
provider_name: provider_name.to_string(),
27982799
model_id: model_id.to_string(),
2800+
route_name: route_name.to_string(),
27992801
})
28002802
.await
28012803
.into_diagnostic()?;
28022804

28032805
let configured = response.into_inner();
2804-
println!("{}", "Gateway inference configured:".cyan().bold());
2806+
let label = if configured.route_name == "sandbox-system" {
2807+
"System inference configured:"
2808+
} else {
2809+
"Gateway inference configured:"
2810+
};
2811+
println!("{}", label.cyan().bold());
28052812
println!();
2813+
println!(" {} {}", "Route:".dimmed(), configured.route_name);
28062814
println!(" {} {}", "Provider:".dimmed(), configured.provider_name);
28072815
println!(" {} {}", "Model:".dimmed(), configured.model_id);
28082816
println!(" {} {}", "Version:".dimmed(), configured.version);
@@ -2813,6 +2821,7 @@ pub async fn gateway_inference_update(
28132821
server: &str,
28142822
provider_name: Option<&str>,
28152823
model_id: Option<&str>,
2824+
route_name: &str,
28162825
tls: &TlsOptions,
28172826
) -> Result<()> {
28182827
if provider_name.is_none() && model_id.is_none() {
@@ -2825,7 +2834,9 @@ pub async fn gateway_inference_update(
28252834

28262835
// Fetch current config to use as base for the partial update.
28272836
let current = client
2828-
.get_cluster_inference(GetClusterInferenceRequest {})
2837+
.get_cluster_inference(GetClusterInferenceRequest {
2838+
route_name: route_name.to_string(),
2839+
})
28292840
.await
28302841
.into_diagnostic()?
28312842
.into_inner();
@@ -2837,35 +2848,94 @@ pub async fn gateway_inference_update(
28372848
.set_cluster_inference(SetClusterInferenceRequest {
28382849
provider_name: provider.to_string(),
28392850
model_id: model.to_string(),
2851+
route_name: route_name.to_string(),
28402852
})
28412853
.await
28422854
.into_diagnostic()?;
28432855

28442856
let configured = response.into_inner();
2845-
println!("{}", "Gateway inference updated:".cyan().bold());
2857+
let label = if configured.route_name == "sandbox-system" {
2858+
"System inference updated:"
2859+
} else {
2860+
"Gateway inference updated:"
2861+
};
2862+
println!("{}", label.cyan().bold());
28462863
println!();
2864+
println!(" {} {}", "Route:".dimmed(), configured.route_name);
28472865
println!(" {} {}", "Provider:".dimmed(), configured.provider_name);
28482866
println!(" {} {}", "Model:".dimmed(), configured.model_id);
28492867
println!(" {} {}", "Version:".dimmed(), configured.version);
28502868
Ok(())
28512869
}
28522870

2853-
pub async fn gateway_inference_get(server: &str, tls: &TlsOptions) -> Result<()> {
2871+
pub async fn gateway_inference_get(
2872+
server: &str,
2873+
route_name: Option<&str>,
2874+
tls: &TlsOptions,
2875+
) -> Result<()> {
28542876
let mut client = grpc_inference_client(server, tls).await?;
2855-
let response = client
2856-
.get_cluster_inference(GetClusterInferenceRequest {})
2857-
.await
2858-
.into_diagnostic()?;
28592877

2860-
let configured = response.into_inner();
2861-
println!("{}", "Gateway inference:".cyan().bold());
2862-
println!();
2863-
println!(" {} {}", "Provider:".dimmed(), configured.provider_name);
2864-
println!(" {} {}", "Model:".dimmed(), configured.model_id);
2865-
println!(" {} {}", "Version:".dimmed(), configured.version);
2878+
if let Some(name) = route_name {
2879+
// Show a single route (--system was specified).
2880+
let response = client
2881+
.get_cluster_inference(GetClusterInferenceRequest {
2882+
route_name: name.to_string(),
2883+
})
2884+
.await
2885+
.into_diagnostic()?;
2886+
2887+
let configured = response.into_inner();
2888+
let label = if name == "sandbox-system" {
2889+
"System inference:"
2890+
} else {
2891+
"Gateway inference:"
2892+
};
2893+
println!("{}", label.cyan().bold());
2894+
println!();
2895+
println!(" {} {}", "Provider:".dimmed(), configured.provider_name);
2896+
println!(" {} {}", "Model:".dimmed(), configured.model_id);
2897+
println!(" {} {}", "Version:".dimmed(), configured.version);
2898+
} else {
2899+
// Show both routes by default.
2900+
print_inference_route(&mut client, "Gateway inference", "").await;
2901+
println!();
2902+
print_inference_route(&mut client, "System inference", "sandbox-system").await;
2903+
}
28662904
Ok(())
28672905
}
28682906

2907+
async fn print_inference_route(
2908+
client: &mut crate::tls::GrpcInferenceClient,
2909+
label: &str,
2910+
route_name: &str,
2911+
) {
2912+
match client
2913+
.get_cluster_inference(GetClusterInferenceRequest {
2914+
route_name: route_name.to_string(),
2915+
})
2916+
.await
2917+
{
2918+
Ok(response) => {
2919+
let configured = response.into_inner();
2920+
println!("{}", format!("{label}:").cyan().bold());
2921+
println!();
2922+
println!(" {} {}", "Provider:".dimmed(), configured.provider_name);
2923+
println!(" {} {}", "Model:".dimmed(), configured.model_id);
2924+
println!(" {} {}", "Version:".dimmed(), configured.version);
2925+
}
2926+
Err(e) if e.code() == Code::NotFound => {
2927+
println!("{}", format!("{label}:").cyan().bold());
2928+
println!();
2929+
println!(" {}", "Not configured".dimmed());
2930+
}
2931+
Err(e) => {
2932+
println!("{}", format!("{label}:").cyan().bold());
2933+
println!();
2934+
println!(" {} {}", "Error:".red(), e.message());
2935+
}
2936+
}
2937+
}
2938+
28692939
pub fn git_repo_root(local_path: &Path) -> Result<PathBuf> {
28702940
let git_dir = if local_path.is_dir() {
28712941
local_path

‎crates/navigator-router/src/config.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ pub struct RouteConfig {
3535
/// caller during resolution.
3636
#[derive(Clone)]
3737
pub struct ResolvedRoute {
38+
/// Route name used for identification (e.g. "inference.local", "sandbox-system").
39+
pub name: String,
3840
pub endpoint: String,
3941
pub model: String,
4042
pub api_key: String,
@@ -48,6 +50,7 @@ pub struct ResolvedRoute {
4850
impl std::fmt::Debug for ResolvedRoute {
4951
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
5052
f.debug_struct("ResolvedRoute")
53+
.field("name", &self.name)
5154
.field("endpoint", &self.endpoint)
5255
.field("model", &self.model)
5356
.field("api_key", &"[REDACTED]")
@@ -119,6 +122,7 @@ impl RouteConfig {
119122
let (auth, default_headers) = auth_from_provider_type(self.provider_type.as_deref());
120123

121124
Ok(ResolvedRoute {
125+
name: self.name.clone(),
122126
endpoint: self.endpoint.clone(),
123127
model: self.model.clone(),
124128
api_key: self.resolve_api_key()?,
@@ -245,6 +249,7 @@ routes:
245249
#[test]
246250
fn resolved_route_debug_redacts_api_key() {
247251
let route = ResolvedRoute {
252+
name: "test".to_string(),
248253
endpoint: "https://api.example.com/v1".to_string(),
249254
model: "test-model".to_string(),
250255
api_key: "sk-super-secret-key-12345".to_string(),

‎crates/navigator-router/src/mock.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@ mod tests {
124124

125125
fn make_route(endpoint: &str, protocols: &[&str], model: &str) -> ResolvedRoute {
126126
ResolvedRoute {
127+
name: "test".to_string(),
127128
endpoint: endpoint.to_string(),
128129
model: model.to_string(),
129130
api_key: "key".to_string(),

‎crates/navigator-router/tests/backend_integration.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ use wiremock::{Mock, MockServer, ResponseTemplate};
88

99
fn mock_candidates(base_url: &str) -> Vec<ResolvedRoute> {
1010
vec![ResolvedRoute {
11+
name: "inference.local".to_string(),
1112
endpoint: base_url.to_string(),
1213
model: "meta/llama-3.1-8b-instruct".to_string(),
1314
api_key: "test-api-key".to_string(),
@@ -109,6 +110,7 @@ async fn proxy_upstream_401_returns_error() {
109110
async fn proxy_no_compatible_route_returns_error() {
110111
let router = Router::new().unwrap();
111112
let candidates = vec![ResolvedRoute {
113+
name: "inference.local".to_string(),
112114
endpoint: "http://localhost:1234".to_string(),
113115
model: "test".to_string(),
114116
api_key: "key".to_string(),
@@ -169,6 +171,7 @@ async fn proxy_strips_auth_header() {
169171
async fn proxy_mock_route_returns_canned_response() {
170172
let router = Router::new().unwrap();
171173
let candidates = vec![ResolvedRoute {
174+
name: "inference.local".to_string(),
172175
endpoint: "mock://test".to_string(),
173176
model: "mock/test-model".to_string(),
174177
api_key: "unused".to_string(),
@@ -302,6 +305,7 @@ async fn proxy_uses_x_api_key_for_anthropic_route() {
302305

303306
let router = Router::new().unwrap();
304307
let candidates = vec![ResolvedRoute {
308+
name: "inference.local".to_string(),
305309
endpoint: mock_server.uri(),
306310
model: "claude-sonnet-4-20250514".to_string(),
307311
api_key: "test-anthropic-key".to_string(),
@@ -359,6 +363,7 @@ async fn proxy_anthropic_does_not_send_bearer_auth() {
359363

360364
let router = Router::new().unwrap();
361365
let candidates = vec![ResolvedRoute {
366+
name: "inference.local".to_string(),
362367
endpoint: mock_server.uri(),
363368
model: "claude-sonnet-4-20250514".to_string(),
364369
api_key: "anthropic-key".to_string(),

0 commit comments

Comments
 (0)