Stage 3 completes the migration from supervisor polling to complete desired-state delivery over ConnectSupervisor. It adds a durable logical transaction for sandbox-scoped configuration mutations and makes server-side waiting explicit.
Base: NVIDIA/OpenShell PR #3265 (Stage 2)
Implementation: NVIDIA/OpenShell PR #3273 (Stage 3, draft)
- The database remains the source of desired state.
- A sandbox-scoped mutation stores desired state and its update operation atomically.
- No database transaction or row lock remains open during delivery or waiting.
COMMIT_ONLYreturns after the atomic commit.WAIT_FOR_APPLYwaits for a durable terminal state.- The exact target is
(policy_version, settings_revision); a result for another tuple cannot complete the operation. - Complete snapshots travel over the authenticated supervisor stream. The supervisor no longer polls configuration fetch RPCs.
- Operations contain revisions, response metadata, outcomes, timestamps, and bounded sanitized errors—never credential-bearing snapshots.
stateDiagram-v2
[*] --> Pending: desired state + operation commit
[*] --> Inactive: commit while Stopped/Completed
[*] --> Cancelled: commit while Deleting
Pending --> Applied: exact revision acknowledged\nAPPLIED / ALREADY_CURRENT / DEGRADED
Pending --> Failed: exact revision attempted\nand supervisor reports failure
Pending --> Superseded: newer policy or settings\nrevision observed first
Pending --> Inactive: authoritative phase becomes\nStopped/Completed
Pending --> Cancelled: sandbox deleted or Deleting
Applied --> [*]
Inactive --> [*]
Failed --> [*]
Superseded --> [*]
Cancelled --> [*]
Client cancellation or timeout is not a state transition. The durable operation remains pending and can complete later.
sequenceDiagram
participant C as CLI / SDK
participant R as Request gateway
participant DB as Durable store
participant W as Operation reconciler
participant O as Owner gateway
participant S as Supervisor
C->>R: UpdateConfig(consistency, idempotency_key)
R->>DB: BEGIN
R->>DB: CAS desired policy/settings
R->>DB: INSERT operation(target tuple, initial phase)
R->>DB: COMMIT
Note over R,DB: Atomicity ends here and no DB lock spans the network
alt COMMIT_ONLY
R-->>C: committed response + operation
else WAIT_FOR_APPLY
R->>DB: reread operation until terminal
end
par immediate publication
R->>O: publish latest complete snapshot via router
and crash/HA repair
W->>DB: page through pending operations
W->>O: rebuild and republish current snapshot
end
O->>S: ConfigUpdate(update_id, sequence, complete snapshot)
S->>O: ConfigUpdateResult(exact requested revision, outcome)
O->>DB: CAS operation to terminal state
DB-->>R: durable terminal result (wake hint or reread)
R-->>C: applied / inactive / failed / superseded / cancelled
- A crash after commit but before publication leaves a pending operation for redispatch.
- Request and owner gateways may differ; the waiter depends only on durable state.
- A disconnect does not imply inactivity. Only authoritative lifecycle state can produce
inactive. - Newer policy or settings revisions supersede an older tuple that was never observed as active.
- Acknowledgements are fenced by session, update ID, component sequence, component, and requested revision.
- Idempotency keys have a stable per-sandbox operation name. A retry returns the existing operation; a concurrent duplicate transaction rolls back and is safely retryable.
- Redispatch uses bounded exponential backoff and paginated scans.
- Error strings are sanitized and capped at 1024 bytes.
Stage 3 inherits the startup exchange from Stage 2. On the first
ConnectSupervisor stream, SupervisorHello may offer the policy baked into
the sandbox image. The gateway selects its existing policy when present,
otherwise the image policy, and sends the full selection as
StartupConfigCandidate.
The supervisor adds filesystem paths that exist in its image and returns
StartupConfigPrepared with unchanged, a complete prepared policy, or a
failure. The gateway treats a complete policy as an untrusted proposal. It
validates and persists the result before building the bootstrap in
SessionAccepted. Neither the candidate nor the preparation response becomes
runtime state.
This startup exchange does not create a durable configuration operation. It establishes the gateway-owned baseline before the session becomes routable. Later user-initiated mutations use the durable transaction described above. Reconnects omit the image policy, skip preparation, and receive the current gateway bootstrap directly.
UpdateConfigRequest adds:
consistency:COMMIT_ONLYorWAIT_FOR_APPLYidempotency_keywait_timeout_secs
UpdateConfigResponse includes ConfigUpdateOperation. GetConfigUpdateOperation retrieves durable progress after a timeout or client cancellation. Deadline errors carry operation-id metadata.
Terminal states are APPLIED, INACTIVE, FAILED, SUPERSEDED, and CANCELLED; PENDING is non-terminal.
- Commit intent and operation together; deliver only after commit.
- Use CAS for desired state and operation transitions.
- Compare exact monotonic target identities, not timestamps or content hashes alone.
- Keep network I/O outside database transactions.
- Treat notifications as latency hints and durable rereads as correctness.
- Make every non-terminal state recoverable by a background reconciler.
- Keep payloads level-triggered and complete so coalescing and reconnect are safe.
- Separate client deadlines from business outcomes.
- Sanitize and bound persisted errors; never persist secret delivery envelopes.
- Test crashes at every commit/delivery/ack boundary, concurrency, inactive phases, timeout, reconnect, and HA ownership changes.
- Scope each idempotency key to one immutable request intent; never reuse a key for a different mutation.
Supervisor protocol revision 3 is release-matched. Both peers reject older revisions, bootstrap is required, and runtime configuration changes are consumed only from ConnectSupervisor. The obsolete supervisor-only GetSandboxProviderEnvironment endpoint and client path are removed; provider credentials arrive only in encrypted bootstrap and live snapshots. Public configuration query APIs remain read-only and are not used for synchronization.