Skip to content

Commit 1ab15bf

Browse files
committed
feat(supervisor): push gateway desired state
Signed-off-by: Piotr Mlocek <[email protected]>
1 parent ba19ea3 commit 1ab15bf

68 files changed

Lines changed: 4631 additions & 2089 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎architecture/compute-runtimes.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -333,7 +333,7 @@ process-supervision leaf and launches the user workload after the sidecar
333333
serves bootstrap state over a local control socket. The network sidecar owns
334334
gateway credentials and sends policy plus workload-facing provider environment
335335
state to the process leaf over that socket. It also streams provider
336-
environment updates after settings polls so future process sessions see
336+
environment updates received from the gateway so future process sessions see
337337
updated provider env without giving the process leaf gateway access. The
338338
pre-workload process supervisor is the only accepted control client: the
339339
network sidecar verifies its UID, GID, and PID with peer credentials, removes

‎architecture/gateway.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -467,8 +467,8 @@ still referenced by a sandbox.
467467

468468
Policy and runtime settings are delivered together through the effective sandbox
469469
config path. A gateway-global policy can override sandbox-scoped policy. The
470-
sandbox supervisor polls for config revisions and hot-reloads dynamic policy
471-
when the policy engine accepts the update.
470+
gateway pushes changed snapshots over `ConnectSupervisor`, and the sandbox
471+
supervisor hot-reloads dynamic policy when the policy engine accepts the update.
472472

473473
External supervisor middleware registration is operator-owned configuration
474474
under `[[openshell.supervisor.middleware]]`. At startup the gateway connects to
@@ -505,8 +505,8 @@ include profile endpoint and binding changes.
505505

506506
Cluster inference routes store only `provider_name`, `model_id`, and optional
507507
timeout. The gateway resolves endpoint URLs, protocols, credentials, auth
508-
style, and route-shaping metadata from the provider record when supervisors call
509-
`GetInferenceBundle`. Supported provider types for cluster inference are
508+
style, and route-shaping metadata from the provider record when the gateway
509+
builds supervisor bootstrap and update snapshots. Supported provider types for cluster inference are
510510
`openai`, `anthropic`, `nvidia`, `deepinfra`, and `google-vertex-ai`.
511511

512512
The bundle carries enough information for sandbox-local routers to construct

‎architecture/google-vertex-ai-provider.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,8 @@ Gateway (openshell-server)
6565
│ ├── infer_vertex_publisher() model → publisher
6666
│ └── vertex_location_and_host() region → Vertex API host
6767
│
68-
└── GetInferenceBundleRequest (from sandbox on connect)
69-
└── resolve_route_by_name() re-resolves live route+credentials
68+
└── ConfigBootstrap / ConfigUpdate snapshot builder
69+
└── resolve_route_by_name() resolves live route+credentials
7070
7171
Router (openshell-router)
7272
│
@@ -412,10 +412,10 @@ Provider type normalization for the `ProviderRegistry` (non-inference providers
412412

413413
## 9. Inference Routing in the Sandbox
414414

415-
When a sandbox agent connects to `https://inference.local`, the sandbox fetches the
416-
inference bundle from the gateway (`GetInferenceBundleRequest`). The bundle contains one
417-
or more `ResolvedRoute` proto messages built by `resolve_route_by_name`. For a Vertex AI
418-
route the bundle contains:
415+
Before a sandbox agent connects to `https://inference.local`, the supervisor applies the
416+
inference snapshot delivered through `ConnectSupervisor`. The snapshot contains one or
417+
more `ResolvedRoute` proto messages built by `resolve_route_by_name`. For a Vertex AI
418+
route the snapshot contains:
419419

420420
```
421421
ResolvedRoute {

‎architecture/sandbox.md‎

Lines changed: 63 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Each sandbox workload has two trust levels:
1010

1111
| Process | Role |
1212
|---|---|
13-
| Supervisor | Starts as root inside the workload, prepares isolation, runs the proxy, fetches config, injects credentials, serves the relay socket, and launches child processes. |
13+
| Supervisor | Starts as root inside the workload, applies desired state, prepares isolation, runs the proxy, injects credentials, serves the relay socket, and launches child processes. |
1414
| Agent child | Runs as an unprivileged user with filesystem, process, and network restrictions applied. |
1515

1616
The supervisor keeps enough privilege to manage the sandbox, but the agent child
@@ -25,15 +25,55 @@ only when the set is already empty; any other outcome fails the spawn.
2525

2626
1. The compute runtime starts the workload with sandbox identity, callback
2727
endpoint, TLS or secret material, image metadata, and initial command.
28-
2. The supervisor loads policy and runtime settings from local files or the
29-
gateway, depending on mode.
30-
3. It prepares filesystem access, process restrictions, network namespace
28+
2. In gateway-backed mode, the supervisor opens `ConnectSupervisor` before it
29+
initializes gateway-owned runtime state. The gateway sends a complete
30+
configuration, provider environment, and inference bootstrap.
31+
3. The supervisor applies the bootstrap, or loads explicit local files in
32+
standalone mode. A required configuration or policy failure aborts startup.
33+
4. It prepares filesystem access, process restrictions, network namespace
3134
routing, trust stores, provider credential resolution, and inference routes.
32-
4. It launches the persisted canonical main-process argv and retains its PTY
35+
5. It launches the persisted canonical main-process argv and retains its PTY
3336
or pipes in the main-session multiplexer.
34-
5. It starts the policy proxy and local SSH server.
35-
6. It opens a supervisor session back to the gateway for connect, exec, file
36-
sync, config polling, and log push.
37+
6. It starts the policy proxy and local SSH server, then signals runtime-ready.
38+
7. The gateway enables connect, exec, file sync, and other relay operations only
39+
after both bootstrap initialization and runtime-ready are complete.
40+
41+
## Desired-State Delivery
42+
43+
`ConnectSupervisor` carries gateway-owned desired state. A fresh session always
44+
starts with a complete `ConfigBootstrap`. Bootstrap success marks the session
45+
initialized; `SupervisorRuntimeReady` separately proves that runtime-dependent
46+
services are available. Bootstrap failure or a 120-second timeout moves the
47+
sandbox to `Error`. Disconnect before initialization returns it to
48+
`Provisioning`.
49+
50+
After bootstrap, the gateway sends level-triggered `ConfigUpdate` messages for
51+
one component at a time. Request IDs correlate results, component-local sequence
52+
numbers reject stale delivery, and snapshot revisions remain equality-only
53+
content fingerprints. The gateway keeps at most one update in flight for each
54+
component and coalesces newer state. Failed live updates keep the sandbox
55+
`Ready` and add a component-specific degraded condition. Policy and inference
56+
retain their last-known-good runtime state. Invalid provider bindings fail
57+
closed by revoking static credential material while retaining fetched dynamic
58+
token grants.
59+
60+
Committed sandbox changes notify the session owner immediately. Provider and
61+
inference changes use a workspace-wide invalidation signal. A 30-second
62+
jittered reconciliation pass rebuilds snapshots for active locally owned
63+
sessions, repairing missed notifications and cross-replica writes. Reconnects
64+
discard queued state and begin again with one complete bootstrap.
65+
66+
Explicit local policy and inference files remain authoritative. Their component
67+
results report `RETAINED_LOCAL_OVERRIDE`, while non-conflicting gateway settings
68+
and provider state continue to update. A supervisor without a gateway does not
69+
open or wait for a session.
70+
71+
`GetSandboxConfig` remains a public read API. Policy status, policy analysis,
72+
log upload, credential refresh, and relay RPCs remain independent of desired
73+
state delivery. The old supervisor-only provider-environment and inference-bundle
74+
fetch RPCs no longer exist. Snapshot builders and component apply routines are
75+
shared implementation boundaries for a future direct transport; the runtime
76+
does not add a transport abstraction before that transport exists.
3777

3878
## Isolation Layers
3979

@@ -195,10 +235,9 @@ the registry. Public custom-CA PEM travels with the stable registration.
195235

196236
The slots live in a supervisor-owned `ExtensionCredentialStore` shared by every
197237
gateway connection the supervisor opens, so the registry's clients and the
198-
polling loop that rotates them observe the same credentials. Configuration
199-
polling runs far more frequently than credentials expire, so the loop rotates
200-
only when a credential is missing or has passed four fifths of its lifetime,
201-
and bounds its sleep by the soonest rotation deadline.
238+
independent credential-refresh task observe the same credentials. The task
239+
rotates only when a credential is missing or has passed four fifths of its
240+
lifetime.
202241

203242
Middleware cannot observe injected credentials or mutate supervisor-owned
204243
credential, routing, or framing headers. Body transformations are re-evaluated
@@ -442,9 +481,9 @@ policy structure.
442481
This holds even when the initial policy is enriched with baseline paths during
443482
startup: the enriched revision the supervisor synced back to the gateway is the
444483
revision it acknowledges, so a successfully constructed initial policy never
445-
remains `Pending`. If the first poll returns a different revision, the supervisor
446-
processes it through the normal reload path instead of treating it as already
447-
loaded.
484+
remains `Pending`. If a subsequent pushed snapshot carries a different revision,
485+
the supervisor processes it through the normal reload path instead of treating
486+
it as already loaded.
448487

449488
A newer sandbox-scoped revision can carry the same non-empty effective policy
450489
hash as the currently loaded revision, for example when provenance changes
@@ -459,19 +498,22 @@ Policy status delivery uses a FIFO background worker. Retryable delivery
459498
failures retain the ordered update and retry with capped exponential backoff;
460499
terminal errors are logged and discarded. The outbox is nonblocking and does
461500
not discard updates because of a fixed queue capacity, so status endpoint
462-
outages cannot block policy polling, enforcement, settings, or provider
463-
refreshes and cannot permanently lose the initial acknowledgement.
501+
outages cannot block desired-state application, enforcement, settings, or
502+
provider refreshes and cannot permanently lose the initial acknowledgement.
464503

465504
Only sandbox-scoped revisions (`PolicySource::Sandbox`, version greater than
466505
zero) are acknowledged. Global policies and local-file development policies do
467506
not use the sandbox revision API and produce no acknowledgement. When explicit
468-
local Rego and data files are configured, the supervisor continues polling the
469-
gateway for settings and provider refreshes but never replaces the local OPA
470-
engine with a gateway policy revision.
507+
local Rego and data files are configured, pushed gateway updates still apply
508+
settings and provider changes but never replace the local OPA engine with a
509+
gateway policy revision.
471510

472511
## Failure Behavior
473512

474-
- If gateway config polling fails, the sandbox keeps its last-known-good policy.
513+
- If desired-state delivery or application fails after startup, the sandbox
514+
remains ready and reports a degraded condition. Policy and inference retain
515+
their last-known-good state; invalid provider bindings follow the fail-closed
516+
revocation behavior above.
475517
- If a live policy or middleware-registry update is invalid, the supervisor
476518
rejects the combined update and keeps the current runtime pair.
477519
- If an operator-run middleware call fails, the selected config's `on_error`

‎architecture/security-policy.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -166,9 +166,10 @@ detection finding at startup naming the inactive controls.
166166
The gateway stores sandbox-authored policy revisions separately from derived
167167
effective sandbox configuration. Effective configuration can include
168168
gateway-global policy overrides and provider-profile policy layers. The
169-
supervisor polls for config revisions and attempts to load new dynamic policy
170-
into the in-process OPA engine; CLI reads of the latest sandbox policy use the
171-
same effective configuration path.
169+
gateway pushes effective configuration revisions over the supervisor session.
170+
The supervisor validates and loads changed dynamic policy into the in-process
171+
OPA engine; CLI reads of the latest sandbox policy use the same effective
172+
configuration path.
172173

173174
The supervisor validates complete effective policy generations before
174175
activation. Overlapping endpoint selectors may contribute request allow and

‎crates/openshell-cli/src/run.rs‎

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -42,15 +42,15 @@ use openshell_core::proto::{
4242
ExposeServiceRequest, GetCurrentUserRequest, GetDraftHistoryRequest, GetDraftPolicyRequest,
4343
GetGatewayConfigRequest, GetInferenceRouteRequest, GetProviderProfileRequest,
4444
GetProviderRefreshStatusRequest, GetProviderRequest, GetSandboxConfigRequest,
45-
GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxPolicyStatusRequest,
46-
GetSandboxRequest, GetServiceRequest, GpuResourceRequirements, ImportProviderProfilesRequest,
47-
LintProviderProfilesRequest, ListProviderProfilesRequest, ListProvidersRequest,
48-
ListSandboxPoliciesRequest, ListSandboxProvidersRequest, ListSandboxesRequest,
49-
ListServicesRequest, PolicySource, PolicyStatus, Provider,
50-
ProviderCredentialRefreshRecoveryAction, ProviderCredentialRefreshStatus,
51-
ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrantType, ProviderProfile,
52-
ProviderProfileDiagnostic, ProviderProfileImportItem, RejectDraftChunkRequest,
53-
ResourceRequirements, RevokeSshSessionRequest, RotateProviderCredentialRequest, Sandbox,
45+
GetSandboxLogsRequest, GetSandboxPolicyStatusRequest, GetSandboxRequest, GetServiceRequest,
46+
GpuResourceRequirements, ImportProviderProfilesRequest, LintProviderProfilesRequest,
47+
ListProviderProfilesRequest, ListProvidersRequest, ListSandboxPoliciesRequest,
48+
ListSandboxProvidersRequest, ListSandboxesRequest, ListServicesRequest, PolicySource,
49+
PolicyStatus, Provider, ProviderCredentialRefreshRecoveryAction,
50+
ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy,
51+
ProviderCredentialTokenGrantType, ProviderProfile, ProviderProfileDiagnostic,
52+
ProviderProfileImportItem, RejectDraftChunkRequest, ResourceRequirements,
53+
RevokeSshSessionRequest, RotateProviderCredentialRequest, Sandbox, SandboxConfigSnapshot,
5454
SandboxPhase, SandboxPolicy, SandboxSpec, SandboxTemplate, ServiceEndpointResponse,
5555
SetInferenceRouteRequest, SettingScope, StartSandboxRequest, StopSandboxRequest,
5656
TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest,
@@ -2109,7 +2109,7 @@ fn sandbox_to_json(sandbox: &Sandbox) -> serde_json::Value {
21092109

21102110
fn sandbox_detail_to_json(
21112111
sandbox: &Sandbox,
2112-
config: &GetSandboxConfigResponse,
2112+
config: &SandboxConfigSnapshot,
21132113
) -> Result<serde_json::Value> {
21142114
let mut value = sandbox_to_json(sandbox);
21152115
let obj = value
@@ -5935,7 +5935,7 @@ pub async fn gateway_settings_get(server: &str, json: bool, tls: &TlsOptions) ->
59355935
fn settings_to_json_sandbox(
59365936
name: &str,
59375937
workspace: &str,
5938-
response: &GetSandboxConfigResponse,
5938+
response: &SandboxConfigSnapshot,
59395939
) -> serde_json::Value {
59405940
let policy_source = if response.policy_source == PolicySource::Global as i32 {
59415941
"global"
@@ -7429,12 +7429,12 @@ mod tests {
74297429
PROGRESS_STEP_STARTING_SANDBOX,
74307430
};
74317431
use openshell_core::proto::{
7432-
GetSandboxConfigResponse, GpuResourceRequirements, PolicySource, PolicyStatus, Provider,
7433-
ProviderCredentialRefresh, ProviderCredentialRefreshRecoveryAction,
7434-
ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy,
7435-
ProviderCredentialTokenGrant, ProviderProfile, ProviderProfileCredential,
7436-
ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicyRevision,
7437-
SandboxStatus, datamodel::v1::ObjectMeta,
7432+
GpuResourceRequirements, PolicySource, PolicyStatus, Provider, ProviderCredentialRefresh,
7433+
ProviderCredentialRefreshRecoveryAction, ProviderCredentialRefreshStatus,
7434+
ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrant, ProviderProfile,
7435+
ProviderProfileCredential, ResourceRequirements, Sandbox, SandboxCondition,
7436+
SandboxConfigSnapshot, SandboxPhase, SandboxPolicyRevision, SandboxStatus,
7437+
datamodel::v1::ObjectMeta,
74387438
};
74397439

74407440
#[test]
@@ -8789,7 +8789,7 @@ mod tests {
87898789
sandbox.set_phase(SandboxPhase::Ready as i32);
87908790
sandbox.set_current_policy_version(2);
87918791

8792-
let config = GetSandboxConfigResponse {
8792+
let config = SandboxConfigSnapshot {
87938793
policy_source: PolicySource::Global as i32,
87948794
global_policy_version: 3,
87958795
..Default::default()
@@ -8815,7 +8815,7 @@ mod tests {
88158815
}),
88168816
..Default::default()
88178817
};
8818-
let config = GetSandboxConfigResponse {
8818+
let config = SandboxConfigSnapshot {
88198819
policy_source: PolicySource::Sandbox as i32,
88208820
version: 0,
88218821
..Default::default()

‎crates/openshell-cli/tests/ensure_providers_integration.rs‎

Lines changed: 8 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,12 @@ use openshell_core::proto::{
2020
DetachSandboxProviderRequest, DetachSandboxProviderResponse,
2121
ExchangeProviderSubjectTokenRequest, ExchangeProviderSubjectTokenResponse, ExecSandboxEvent,
2222
ExecSandboxInput, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest,
23-
GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest,
24-
GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest,
25-
GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse,
26-
ListProvidersRequest, ListProvidersResponse, ListSandboxProvidersRequest,
27-
ListSandboxProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, Provider,
28-
ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse,
29-
SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest,
30-
WatchSandboxRequest,
23+
GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, GetSandboxRequest,
24+
HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse,
25+
ListSandboxProvidersRequest, ListSandboxProvidersResponse, ListSandboxesRequest,
26+
ListSandboxesResponse, Provider, ProviderResponse, RevokeSshSessionRequest,
27+
RevokeSshSessionResponse, SandboxConfigSnapshot, SandboxResponse, SandboxStreamEvent,
28+
ServiceStatus, SupervisorMessage, UpdateProviderRequest, WatchSandboxRequest,
3129
};
3230
use openshell_core::{ObjectId, ObjectName};
3331
use std::collections::HashMap;
@@ -178,8 +176,8 @@ impl OpenShell for TestOpenShell {
178176
async fn get_sandbox_config(
179177
&self,
180178
_request: tonic::Request<GetSandboxConfigRequest>,
181-
) -> Result<Response<GetSandboxConfigResponse>, Status> {
182-
Ok(Response::new(GetSandboxConfigResponse::default()))
179+
) -> Result<Response<SandboxConfigSnapshot>, Status> {
180+
Ok(Response::new(SandboxConfigSnapshot::default()))
183181
}
184182

185183
async fn get_gateway_config(
@@ -189,15 +187,6 @@ impl OpenShell for TestOpenShell {
189187
Ok(Response::new(GetGatewayConfigResponse::default()))
190188
}
191189

192-
async fn get_sandbox_provider_environment(
193-
&self,
194-
_request: tonic::Request<GetSandboxProviderEnvironmentRequest>,
195-
) -> Result<Response<GetSandboxProviderEnvironmentResponse>, Status> {
196-
Ok(Response::new(
197-
GetSandboxProviderEnvironmentResponse::default(),
198-
))
199-
}
200-
201190
async fn create_ssh_session(
202191
&self,
203192
_request: tonic::Request<CreateSshSessionRequest>,

‎crates/openshell-cli/tests/mtls_integration.rs‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -145,9 +145,9 @@ impl OpenShell for TestOpenShell {
145145
async fn get_sandbox_config(
146146
&self,
147147
_request: tonic::Request<openshell_core::proto::GetSandboxConfigRequest>,
148-
) -> Result<Response<openshell_core::proto::GetSandboxConfigResponse>, Status> {
148+
) -> Result<Response<openshell_core::proto::SandboxConfigSnapshot>, Status> {
149149
Ok(Response::new(
150-
openshell_core::proto::GetSandboxConfigResponse::default(),
150+
openshell_core::proto::SandboxConfigSnapshot::default(),
151151
))
152152
}
153153

@@ -160,16 +160,6 @@ impl OpenShell for TestOpenShell {
160160
))
161161
}
162162

163-
async fn get_sandbox_provider_environment(
164-
&self,
165-
_request: tonic::Request<openshell_core::proto::GetSandboxProviderEnvironmentRequest>,
166-
) -> Result<Response<openshell_core::proto::GetSandboxProviderEnvironmentResponse>, Status>
167-
{
168-
Ok(Response::new(
169-
openshell_core::proto::GetSandboxProviderEnvironmentResponse::default(),
170-
))
171-
}
172-
173163
async fn create_ssh_session(
174164
&self,
175165
_request: tonic::Request<CreateSshSessionRequest>,

0 commit comments

Comments
 (0)