You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0f949a6
Browse filesBrowse the repository at this point in the historyBrowse files
feat(inference): add sandbox-system inference route for platform-level inference (#209)
* feat(inference): add sandbox-system inference route for platform-level inference
Add a separate system-level inference route that the sandbox supervisor
can use in-process for platform functions (e.g., embedded agent harness
for policy analysis), distinct from the user-facing inference.local
endpoint. The system route is accessed via an in-process API on the
supervisor, ensuring userland code in the sandbox netns cannot reach it.
- Extend proto with route_name fields on Set/Get inference messages
- Add ResolvedRoute.name field to the router for route segregation
- Server resolves both user and sandbox-system routes in bundles
- Sandbox partitions routes into user/system caches on refresh
- Expose InferenceContext::system_inference() in-process API
- CLI --sandbox flag targets the system route on set/get/update
- Integration tests using mock:// routes for the full in-process path
Closes#207
* refactor(cli): rename --sandbox flag to --system for inference commands
The --sandbox flag could be misread as targeting a user-level sandbox
operation. Rename to --system to clearly indicate it configures the
platform-level system inference route.
* style(cli): collapse short if-else per rustfmt 1.94
---------
Co-authored-by: John Myers <[email protected]>
Copy file name to clipboardExpand all lines: architecture/inference-routing.md
+43-2Lines changed: 43 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -225,12 +225,53 @@ Validation at load time requires either `api_key` or `api_key_env` to resolve, a
225
225
|`503`| Upstream unavailable (timeout or connection failure) |
226
226
|`413`| Request body exceeds 10 MiB buffer limit |
227
227
228
+
## System Inference Route
229
+
230
+
In addition to the user-facing `inference.local` route, the gateway supports a second managed route named `sandbox-system` for platform system functions (e.g. an embedded agent harness for policy analysis).
231
+
232
+
### Key differences from user inference
233
+
234
+
| Aspect | User (`inference.local`) | System (`sandbox-system`) |
`InferenceContext::system_inference()` provides the supervisor with direct access to inference using the system routes. It calls `Router::proxy_with_candidates()` with the system route cache -- the same backend proxy logic used for user inference, but without any CONNECT/TLS overhead.
244
+
245
+
```rust
246
+
ctx.system_inference(
247
+
"openai_chat_completions",
248
+
"POST",
249
+
"/v1/chat/completions",
250
+
headers,
251
+
body,
252
+
).await
253
+
```
254
+
255
+
### Access control
256
+
257
+
The system route is not exposed through the CONNECT proxy. The supervisor runs in the host network namespace and calls the router directly. User processes are in an isolated sandbox network namespace and cannot reach the in-process API.
258
+
259
+
### Bundle delivery
260
+
261
+
Both routes are included in `GetInferenceBundleResponse.routes` (which is `repeated ResolvedRoute`). The sandbox partitions routes by `ResolvedRoute.name` during `bundle_to_resolved_routes()`: routes named `sandbox-system` go to the system cache, everything else goes to the user cache. Both caches are refreshed on the same polling interval.
262
+
263
+
### Storage
264
+
265
+
The system route is stored as a separate `InferenceRoute` record in the gateway store with `name = "sandbox-system"`. The `SetClusterInferenceRequest.route_name` field selects which route to target (empty string defaults to `inference.local`).
266
+
228
267
## CLI Surface
229
268
230
269
Cluster inference commands:
231
270
232
-
-`openshell cluster inference set --provider <name> --model <id>` -- configures cluster inference by referencing a provider record name
233
-
-`openshell cluster inference get` -- displays current cluster inference configuration
-`openshell cluster inference set --system --provider <name> --model <id>` -- configures system inference
273
+
-`openshell cluster inference get` -- displays both user and system inference configuration
274
+
-`openshell cluster inference get --system` -- displays only the system inference configuration
234
275
235
276
The `--provider` flag references a provider record name (not a provider type). The provider must already exist in the cluster and have a supported inference type (`openai`, `anthropic`, or `nvidia`).
0 commit comments