Skip to content

Instantly share code, notes, and snippets.

@mrunalp
Created August 18, 2026 22:24
Show Gist options
  • Select an option

  • Save mrunalp/dce734aa68af6d7bb9e27b9bda9e292a to your computer and use it in GitHub Desktop.

Select an option

Save mrunalp/dce734aa68af6d7bb9e27b9bda9e292a to your computer and use it in GitHub Desktop.
OpenShell design: provider refresh material in credential storage

Provider Refresh Material in Credential Storage

Decision

Store every secret refresh-material field through the gateway's active credential driver. When the active driver is Vault, refresh tokens, OAuth client secrets, service-account private keys, and temporary AWS source secrets live in Vault. The persisted refresh-state object keeps non-secret material, opaque credential handles, scheduling metadata, and the authorization epoch.

Do not add a Vault-specific refresh path. Reuse the configured credential storage owner so the same behavior works with Vault, Kubernetes Secrets, and the default encrypted database credential store.

Current Behavior and Gap

Provider credential values already use credential storage:

  • CredentialRuntime::store_provider_credentials_with_object_id writes values through the selected driver and returns opaque handles.
  • apply_minted_credential stages newly minted access tokens through that runtime before updating the provider record.
  • The Vault driver stores each value in a managed Vault KV path and returns a CredentialHandle.

Refresh source material does not use that path. ConfigureProviderRefresh copies the request's complete material map into StoredProviderCredentialRefreshState.material. The refresh worker reads the map directly, and an OAuth response containing a rotated refresh token writes the new token back into the same map before the state is encoded into the gateway object store.

The protobuf secret option on material prevents generic observation and interceptor exposure. It does not encrypt the serialized object payload. Therefore Vault-backed providers currently put access tokens in Vault but keep refresh tokens and other refresh source secrets in the gateway database.

Goals

  • Persist no new refresh source secret in plaintext gateway object payloads.
  • Put refresh material in Vault when Vault is the configured credential storage owner.
  • Protect all secret refresh material, not only the field named refresh_token.
  • Preserve PR #2780's authorization epoch and stable workload-handle behavior.
  • Preserve safe compare-and-swap behavior when refresh, reconfiguration, and deletion race across gateway replicas.
  • Support OAuth refresh-token rotation without overwriting a credential still owned by another committed refresh generation.
  • Delete replaced or revoked refresh material from the backing credential store with retryable ownership metadata.
  • Migrate existing inline refresh secrets without requiring users to reauthorize every provider.

Non-goals

  • Selecting a different credential driver per provider or per material field.
  • Exposing credential-driver handles through the public provider refresh API.
  • Sending refresh material to a sandbox or supervisor.
  • Changing the stable workload credential handle introduced by PR #2780.
  • Providing secure erasure from historical database backups, SQLite free pages/WAL files, or Postgres dead tuples. Operators must rotate grants and apply backend-specific retention/compaction procedures when historical erasure is required.

Persisted Model

Extend StoredProviderCredentialRefreshState with typed references:

message StoredProviderRefreshMaterialHandle {
  string material_key = 1;
  openshell.datamodel.v1.CredentialHandle handle = 2;
}

message StoredProviderCredentialRefreshState {
  // Existing fields 1-18 remain unchanged.
  map<string, openshell.datamodel.v1.CredentialHandle>
      secret_material_handles = 19;
  repeated StoredProviderRefreshMaterialHandle pending_secret_deletions = 20;
}

Retain secret_material_keys for sensitivity classification and backward compatibility. Apply these invariants to new-format records:

  1. A key in secret_material_keys must not have a plaintext value in material.
  2. A configured secret key has exactly one active entry in secret_material_handles.
  3. material contains only non-secret values.
  4. Pending deletions are opaque handles only; they are never usable for minting.
  5. The authorization epoch is independent of handle identity. Automatic token rotation changes a storage handle but preserves the authorization epoch.

Use a deterministic, driver-safe logical credential key for each material slot, for example:

openshell.refresh.<sha256(provider credential key NUL material key)>

The synthetic key is valid as a Kubernetes Secret data key and avoids placing caller-controlled material names in backend paths. Continue to pass workspace, provider ID, provider name, and a unique per-write object ID to the driver. Add dedicated CredentialRuntime methods for refresh material so provider env keys and refresh material cannot be mixed accidentally.

Broaden the comments in credential_driver.proto from "provider credential" to "gateway-managed provider secret." No new driver RPC is required.

Secret Classification

The client-provided secret_material_keys list must not be the only security boundary. At configure time, compute the sensitive set as the union of:

  • keys marked secret: true by the authoritative provider profile;
  • keys supplied in secret_material_keys;
  • strategy-defined secret keys, including refresh_token, client_secret, private_key, aws_secret_access_key, and aws_session_token where those fields apply.

Reject a client- or profile-named secret key that is absent from material. Ignore strategy-defined optional fields that are absent, then persist the normalized, sorted set of sensitive keys that are actually present. This protects direct API callers and older CLIs that forget to classify a known secret.

Configure Flow

  1. Authorize and validate the request, profile binding, material, endpoint, scopes, and reserved env keys as today.
  2. Split the material into non-secret values and secret values using the server-derived sensitive set.
  3. Stage every secret value through the active credential driver with a unique configuration object ID and no existing handles. Never overwrite the committed generation in place.
  4. Build the refresh state with non-secret material, staged secret_material_handles, and a newly generated authorization epoch.
  5. Create or replace the refresh state with an explicit resource-version CAS. A concurrent refresh or configure causes ABORTED; it must not be silently overwritten.
  6. If the CAS fails, delete all staged handles. If it succeeds, move replaced handles into pending_secret_deletions and drain that list idempotently.
  7. Continue updating provider expiry metadata as today.

The RPC succeeds only after the new secret handles and refresh-state record are durable. If Vault is unavailable, configuration fails and no inline fallback is allowed.

Mint and Rotation Flow

Keep persisted and resolved material as distinct types. For example, ResolvedRefreshMaterial may hold a transient in-memory map, while minting functions must not accept a mutable persisted state. This prevents resolved secrets from being accidentally encoded back into the database.

For every manual or worker rotation:

  1. Load the provider and refresh state, recording the refresh state's resource version.
  2. Resolve all secret_material_handles in one driver batch and merge the results with non-secret material in memory.
  3. Mint using the transient material. Never log request forms or resolved values.
  4. If the issuer returns a rotated refresh token, stage it under a unique object ID and replace only the refresh_token handle in the candidate state. Add the previous handle to pending_secret_deletions.
  5. CAS the refresh state against the version read before the network request. On conflict or deletion, delete the staged token and discard all minted output. This preserves the stale-generation protection from PR #2780.
  6. After the refresh-state CAS succeeds, apply the minted access token to the provider through the existing staged credential path.
  7. Drain pending secret deletions. Cleanup failure does not roll back a successful token rotation; the durable pending entry allows a later worker pass to retry it.

Credential-driver resolution or storage failure is a refresh failure. Preserve the current access token only until its recorded expiry and fail closed after expiry. Never copy a resolved secret back into inline state as a fallback.

Deletion and Reconfiguration

Deletion needs a durable tombstone so an in-flight refresh cannot recreate source material and a backend outage cannot lose the cleanup reference:

  1. CAS the refresh state to deletion_timestamp_ms != 0, rotate/revoke its authorization epoch, and stop scheduling it.
  2. Exclude tombstoned states from active provider credential bindings while retaining their resource version in provider-environment revision input, so running supervisors observe revocation. The manual and background refresh paths skip tombstoned records.
  3. Delete active and pending material handles idempotently.
  4. Delete the refresh-state row only after all backend deletions succeed.

Make configure, delete, and refresh-state migration use the same per-record CAS contract. Provider deletion must tombstone and drain every refresh state before deleting the provider's injectable credential handles and provider record. Workspace deletion remains blocked while a provider or refresh tombstone exists.

Existing Record Migration

New code must continue reading legacy states where secret keys are present inline and secret_material_handles is empty.

During each worker sweep, migrate every legacy record before checking whether it is due:

  1. Derive the sensitive set from stored secret_material_keys plus the strategy-defined secret set.
  2. Stage the inline secret values through the active driver with unique object IDs.
  3. CAS a candidate state that removes those plaintext entries and adds the handles.
  4. Delete staged handles if another replica wins the CAS.
  5. Do not mint from a partially migrated record.

Run an eager sweep during gateway startup and expose counts for pending, succeeded, and failed migrations. A migration failure leaves the legacy record unchanged, marks readiness degraded for refresh processing, and retries; it must not delete the only usable copy or silently continue claiming that the material is externalized.

Because older Prost binaries do not preserve unknown fields when they decode and rewrite a message, handle-only refresh state is not safe during a mixed-version HA rollout. The first implementation must require a coordinated gateway rollout with all old refresh workers stopped before migration starts. Document the requirement and add a startup compatibility gate or an explicit operator-controlled migration gate. Do not rely on a normal rolling update without one of those protections.

After migration, old plaintext bytes may remain in database history. Recommend rotating long-lived refresh grants after rollout when removing historical copies is a security requirement.

Failure and Concurrency Rules

  • Stage externally, then CAS metadata, then clean up the old handle.
  • Never overwrite the committed Vault path before winning the refresh-state generation.
  • A CAS loser deletes only the handles it staged; it never deletes handles read from committed state.
  • A missing or malformed handle is FAILED_PRECONDITION, not an inline fallback.
  • Backend permission/authentication failures remain distinguishable from OAuth issuer failures in status and telemetry, without including handle strings or material keys that operators marked sensitive.
  • Multiple gateway replicas may mint concurrently, but only the state-version winner may commit its rotated refresh token or access token.
  • Pending deletion entries make post-CAS cleanup crash-recoverable for refresh material. A broader credential-driver orphan reconciler remains useful for existing provider credential writes but is not required to externalize refresh material safely.

Code Changes

  • proto/openshell.proto: add current and pending refresh-material handle fields; regenerate committed Go protobuf output.
  • proto/credential_driver.proto: clarify that the existing contract stores gateway-managed provider secrets, not only injectable credentials.
  • crates/openshell-server/src/credentials.rs: add typed store, resolve, and delete helpers for refresh material and driver-safe logical-key derivation.
  • crates/openshell-server/src/grpc/provider.rs: derive the sensitive set, stage configuration material, use refresh-state CAS, and clean up handles on configure/delete/provider-delete paths.
  • crates/openshell-server/src/provider_refresh.rs: resolve transient material, stage rotated refresh tokens, migrate legacy records, process tombstones, and retry pending deletion entries.
  • architecture/gateway.md: state that refresh source secrets are represented by credential-driver handles.
  • docs/sandboxes/providers-v2.mdx: document storage, migration, failure, and rollout behavior.
  • docs/reference/gateway-config.mdx: clarify that the selected credential driver owns injectable provider credentials and secret refresh material.
  • .agents/skills/openshell-cli/SKILL.md: update refresh troubleshooting and rollout guidance; run the sync-agent-infra checks.

The Vault driver itself should not require behavior changes beyond tests and possibly wording. Its current store, resolve, overwrite, delete, unique object_id, and managed-path validation are sufficient.

Test Plan

Unit and integration

  • Configure each strategy and assert that serialized refresh-state payloads do not contain the submitted secret bytes.
  • Verify profile-declared and strategy-defined secrets are externalized even when a direct API client omits secret_material_keys.
  • Verify non-secret material remains inline and minting receives the correctly reconstructed transient map.
  • Verify Vault/default/Kubernetes handle storage and batch resolution through the credential-runtime contract.
  • Verify a rotated OAuth refresh token replaces its handle, preserves the authorization epoch, and queues/deletes the old handle.
  • Race two rotations, rotation versus configure, rotation versus delete, and two legacy migrations; only one candidate may commit and every losing staged handle must be reclaimed.
  • Simulate a crash/backend failure after CAS and verify pending deletions are retried.
  • Verify tombstoned state cannot mint and is removed only after handle cleanup.
  • Verify provider deletion removes refresh-material handles as well as current access-token handles.
  • Verify legacy inline records migrate and remain usable without changing the authorization epoch.
  • Verify malformed/missing/wrong-driver handles fail closed and never fall back to inline values.
  • Verify no logs, status messages, OCSF events, or interceptor payloads contain refresh material.

End to end

  • Extend the provider refresh E2E so a fake issuer rotates its refresh token on every response and rejects reuse of the prior token.
  • Run with the Vault credential driver and verify the refresh state contains only handles while 12 access-token rotations continue to work for the same long-running process.
  • Restart the gateway between rotations and verify it resolves the current refresh token from Vault.
  • Reconfigure and delete refresh, then verify old Vault entries are removed and PR #2780's old workload handle is revoked.
  • Exercise Vault unavailability during configure, resolve, rotated-token store, and deletion; all paths must fail closed and recover on retry.

Acceptance Criteria

  • With credential_drivers = ["vault"], no newly configured or rotated refresh source secret appears in a provider_credential_refresh_state payload.
  • The Vault backend contains both injectable credentials and secret refresh material under independently managed handles.
  • OAuth providers that rotate refresh tokens continue rotating across gateway restarts and HA refresh races.
  • Explicit reconfiguration preserves PR #2780's revocation semantics while replacing and eventually deleting old refresh-material handles.
  • Delete provider refresh and delete provider remove their owned refresh material, retrying safely after backend failure.
  • Existing inline states migrate without reauthorization, with a documented coordinated rollout and an observable failure state.
  • Default encrypted storage and Kubernetes Secrets receive equivalent behavior without Vault-specific server branches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment