PR #3273 · head 262be8a9e8a42ae4363e8602193b620c60740a76 · stacked on #3265 · part of #1731.
A successful database write means the desired configuration is durable. It does not mean the sandbox has applied it. This PR gives a sandbox-scoped policy/settings update a durable operation that records what happened after the write.
flowchart LR
C[CLI or SDK] --> G[Gateway: authorize and validate]
G --> T[Database transaction]
T --> D[(Desired state + operation)]
D --> R[Indexed operation reconciler]
G --> Q[Bounded delivery queue]
R --> Q
Q --> B[Build current snapshot and associate exact target]
B --> S[Authenticated supervisor stream]
S --> A[Supervisor applies configuration]
A --> V[Gateway validates result correlation]
V --> O[(Commit terminal operation)]
O --> W[Wake waiter or read operation]
W --> C
The database transaction covers desired state, relevant sandbox metadata/resource-version checks and the operation. Network delivery and runtime application happen afterward. No database transaction remains open while waiting for the supervisor.
#3265 supplies streamed delivery, admission, provider readiness and request replay. #3273 adds durable policy/settings completion and recovery around that delivery path. Provider receipts share a storage envelope but use separate completion logic and stay outside sandbox-operation retry indexes. Global updates do not receive this sandbox-operation contract and reject WAIT_FOR_COMPLETION.
sequenceDiagram
participant C as Caller
participant G as Gateway
participant D as Database
participant S as Supervisor
C->>G: UpdateConfig(policy/settings, consistency, request_id)
G->>G: Authorize scope and validate mutation and wait mode
G->>D: Atomically commit desired state + operation
D-->>G: Operation identity
G->>D: Record mutation replay response before waiting
alt COMMIT_ONLY or unspecified
G-->>C: Committed response + operation
else WAIT_FOR_COMPLETION
G->>G: Subscribe and check durable operation
end
G->>G: Build snapshot and associate exact target before send
G->>S: Configuration update (update ID, sequence, revision)
S->>S: Apply and determine outcome
S-->>G: Correlated component result
G->>G: Validate current session and result correlation
G->>D: CAS terminal operation for matching target
G->>G: Notify local waiters after commit
opt Waiting caller is still connected
G-->>C: Terminal state and apply outcome
end
The diagram shows logical stages; commit-only callers may receive their response before delivery starts. Mutation replay is recorded separately from the desired-state transaction and before the caller's completion wait. request_id replays admission; the sandbox-scoped idempotency_key identifies the original operation. They are distinct contracts. Wait preferences do not change mutation identity.
WAIT_FOR_COMPLETION replaces the WAIT_FOR_APPLY symbol at wire value 2. “Completion” includes failure, supersession, cancellation and inactivity. The protobuf field remains named consistency, but selects the return point rather than a database isolation level. wait_timeout is a protobuf Duration; zero uses the server default of 60 seconds.
stateDiagram-v2
[*] --> Pending: active sandbox mutation commits
Pending --> Applied: Applied / IgnoredDuplicate / Degraded
Pending --> Superseded: newer target or IgnoredStale
Pending --> Failed: rejected or unsupported apply result
Pending --> Inactive: sandbox stopped or completed
Pending --> Cancelled: sandbox deleting or deleted
[*] --> Inactive: inactive at admission
[*] --> Cancelled: deleting at admission
State and outcome are separate: Applied with outcome Degraded must remain visible to the caller. RetainedLocalOverride, FailedRetainedLastKnownGood, FailedClosed, Unsupported and an unspecified apply outcome map to Failed. An operation reaches its terminal state through a conditional durable update; a stale result cannot complete a different target.
The snapshot's config_revision is an equality token. Policy/settings revisions provide the ordered desired-state targets; the opaque token is not a sequence number.
flowchart TD
P[(Committed pending operation)] --> T[RPC wait times out or client disconnects]
T --> K[Operation remains durable]
K --> R[Indexed reconciler claims due work with CAS]
R --> Q[Queue latest snapshot for the sandbox]
Q --> M{Target still matches?}
M -->|Newer desired target| X[Superseded]
M -->|Matches| S[Send and await correlated result]
S --> F[(Persist terminal outcome)]
F --> G[Read operation or replay request]
A wait timeout does not cancel or roll back the committed mutation. A caller can look up the operation; a retry with the same request identity can recover its committed response and choose commit-only behavior. Local notifications reduce latency; database polling lets waiters observe completion across gateway replicas or restarts. Recovery scans indexed due work in bounded batches and uses conditional claims/retry deadlines. Delivery may coalesce or repeat; there is no exactly-once network delivery guarantee.
An unchanged policy/settings request still creates a durable operation targeting the existing revisions. It allocates no new desired-state revision. A matching pending operation requires an acknowledgement even if the session already acknowledged an identical snapshot; ordinary duplicate suppression must not strand this new waiter.
Protocol revision 3 is required between this gateway and supervisor. Configuration preparation/admission and runtime readiness remain separate: accepting startup configuration does not itself mark the sandbox Ready. The supervisor launches the workload/relay and sends SupervisorRuntimeReady for the current activated configuration/session.
Operation lookup is workspace-authorized. Operation records retain revision identifiers, state/outcome, timestamps and bounded sanitized errors, not configuration or credential payloads. Provider credentials can exist in the in-memory/transport snapshot and must not be copied into the operation or diagnostic artifacts.
- proto/openshell.proto — Public update/wait/operation contract.
- crates/openshell-server/src/grpc/policy.rs — Admission, mutation and response construction.
- crates/openshell-server/src/grpc/mutation_replay.rs — Durable request replay.
- crates/openshell-server/src/config_update_operation.rs — Target association, terminal transitions, wait and recovery.
- crates/openshell-server/src/config_delivery.rs — Bounded queue, snapshot build and stream delivery.
- crates/openshell-server/src/supervisor_session.rs — Session correlation and acknowledgement handling.
- crates/openshell-server/src/provider_config_operation.rs — Provider receipt completion.
This explanation is pinned to the head above. It describes control flow and invariants, not a performance benchmark. pr-3273-summary.html in this gist provides standalone diagrams without external scripts.