Skip to content

Instantly share code, notes, and snippets.

@pimlock
Last active September 18, 2026 01:02
Show Gist options
  • Select an option

  • Save pimlock/73aa746b48d82d84001cf8e0afa56bf6 to your computer and use it in GitHub Desktop.

Select an option

Save pimlock/73aa746b48d82d84001cf8e0afa56bf6 to your computer and use it in GitHub Desktop.
OpenShell Stage 3 durable configuration apply transactions

OpenShell Stage 3: durable configuration apply transactions

Stage 3 completes the migration from supervisor polling to complete desired-state delivery over ConnectSupervisor. It adds a durable logical transaction for sandbox-scoped configuration mutations and makes server-side waiting explicit.

Base: NVIDIA/OpenShell PR #3265 (Stage 2)

Implementation: NVIDIA/OpenShell PR #3273 (Stage 3, draft)

Contract

  • The database remains the source of desired state.
  • A sandbox-scoped mutation stores desired state and its update operation atomically.
  • No database transaction or row lock remains open during delivery or waiting.
  • COMMIT_ONLY returns after the atomic commit.
  • WAIT_FOR_APPLY waits for a durable terminal state.
  • The exact target is (policy_version, settings_revision); a result for another tuple cannot complete the operation.
  • Complete snapshots travel over the authenticated supervisor stream. The supervisor no longer polls configuration fetch RPCs.
  • Operations contain revisions, response metadata, outcomes, timestamps, and bounded sanitized errors—never credential-bearing snapshots.

State transition diagram

stateDiagram-v2
    [*] --> Pending: desired state + operation commit
    [*] --> Inactive: commit while Stopped/Completed
    [*] --> Cancelled: commit while Deleting

    Pending --> Applied: exact revision acknowledged\nAPPLIED / ALREADY_CURRENT / DEGRADED
    Pending --> Failed: exact revision attempted\nand supervisor reports failure
    Pending --> Superseded: newer policy or settings\nrevision observed first
    Pending --> Inactive: authoritative phase becomes\nStopped/Completed
    Pending --> Cancelled: sandbox deleted or Deleting

    Applied --> [*]
    Inactive --> [*]
    Failed --> [*]
    Superseded --> [*]
    Cancelled --> [*]
Loading

Client cancellation or timeout is not a state transition. The durable operation remains pending and can complete later.

Overall flow

sequenceDiagram
    participant C as CLI / SDK
    participant R as Request gateway
    participant DB as Durable store
    participant W as Operation reconciler
    participant O as Owner gateway
    participant S as Supervisor

    C->>R: UpdateConfig(consistency, idempotency_key)
    R->>DB: BEGIN
    R->>DB: CAS desired policy/settings
    R->>DB: INSERT operation(target tuple, initial phase)
    R->>DB: COMMIT
    Note over R,DB: Atomicity ends here and no DB lock spans the network

    alt COMMIT_ONLY
        R-->>C: committed response + operation
    else WAIT_FOR_APPLY
        R->>DB: reread operation until terminal
    end

    par immediate publication
        R->>O: publish latest complete snapshot via router
    and crash/HA repair
        W->>DB: page through pending operations
        W->>O: rebuild and republish current snapshot
    end

    O->>S: ConfigUpdate(update_id, sequence, complete snapshot)
    S->>O: ConfigUpdateResult(exact requested revision, outcome)
    O->>DB: CAS operation to terminal state
    DB-->>R: durable terminal result (wake hint or reread)
    R-->>C: applied / inactive / failed / superseded / cancelled
Loading

Failure and concurrency behavior

  • A crash after commit but before publication leaves a pending operation for redispatch.
  • Request and owner gateways may differ; the waiter depends only on durable state.
  • A disconnect does not imply inactivity. Only authoritative lifecycle state can produce inactive.
  • Newer policy or settings revisions supersede an older tuple that was never observed as active.
  • Acknowledgements are fenced by session, update ID, component sequence, component, and requested revision.
  • Idempotency keys have a stable per-sandbox operation name. A retry returns the existing operation; a concurrent duplicate transaction rolls back and is safely retryable.
  • Redispatch uses bounded exponential backoff and paginated scans.
  • Error strings are sanitized and capped at 1024 bytes.

Startup policy preparation

Stage 3 inherits the startup exchange from Stage 2. On the first ConnectSupervisor stream, SupervisorHello may offer the policy baked into the sandbox image. The gateway selects its existing policy when present, otherwise the image policy, and sends the full selection as StartupConfigCandidate.

The supervisor adds filesystem paths that exist in its image and returns StartupConfigPrepared with unchanged, a complete prepared policy, or a failure. The gateway treats a complete policy as an untrusted proposal. It validates and persists the result before building the bootstrap in SessionAccepted. Neither the candidate nor the preparation response becomes runtime state.

This startup exchange does not create a durable configuration operation. It establishes the gateway-owned baseline before the session becomes routable. Later user-initiated mutations use the durable transaction described above. Reconnects omit the image policy, skip preparation, and receive the current gateway bootstrap directly.

API shape

UpdateConfigRequest adds:

  • consistency: COMMIT_ONLY or WAIT_FOR_APPLY
  • idempotency_key
  • wait_timeout_secs

UpdateConfigResponse includes ConfigUpdateOperation. GetConfigUpdateOperation retrieves durable progress after a timeout or client cancellation. Deadline errors carry operation-id metadata.

Terminal states are APPLIED, INACTIVE, FAILED, SUPERSEDED, and CANCELLED; PENDING is non-terminal.

Implementation practices

  1. Commit intent and operation together; deliver only after commit.
  2. Use CAS for desired state and operation transitions.
  3. Compare exact monotonic target identities, not timestamps or content hashes alone.
  4. Keep network I/O outside database transactions.
  5. Treat notifications as latency hints and durable rereads as correctness.
  6. Make every non-terminal state recoverable by a background reconciler.
  7. Keep payloads level-triggered and complete so coalescing and reconnect are safe.
  8. Separate client deadlines from business outcomes.
  9. Sanitize and bound persisted errors; never persist secret delivery envelopes.
  10. Test crashes at every commit/delivery/ack boundary, concurrency, inactive phases, timeout, reconnect, and HA ownership changes.
  11. Scope each idempotency key to one immutable request intent; never reuse a key for a different mutation.

Polling removal

Supervisor protocol revision 3 is release-matched. Both peers reject older revisions, bootstrap is required, and runtime configuration changes are consumed only from ConnectSupervisor. The obsolete supervisor-only GetSandboxProviderEnvironment endpoint and client path are removed; provider credentials arrive only in encrypted bootstrap and live snapshots. Public configuration query APIs remain read-only and are not used for synchronization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment