Skip to content

Commit e30c938

Browse files
EmilienMFrostGodbjw123
authored
feat(server): scope mutation locks and bound lock waits (NVIDIA#4147)
* test(server): add a local PostgreSQL test runner The openshell-server tests that need a real PostgreSQL server are ignored by default and had no shared way to run. The mutation-replay test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every contributor had to provision a database by hand, and the advisory-lock tests that follow in this series need the same setup. Add mise run test:rust:postgres. It runs every ignored postgres_* test in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE selects one) on the image pinned by the Kubernetes e2e fixture and removes it on exit. Tests run one at a time because advisory locks are database-wide. The runner always points the legacy replay variable at the selected database, so an inherited URL cannot send that test to a different server. test:postgres-runner checks that selection with a fake cargo and needs no database or container engine. CI does not run the PostgreSQL tests; TESTING.md documents the task. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * fix(server): hold the mutation guard while deleting a provider DeleteProvider checked that no sandbox referenced the provider and then deleted the record without holding the sandbox mutation guard. Sandbox create and provider attach take that guard while they write provider references, so one of them could add a reference after the attached sandbox check passed, and the delete then removed a provider that a sandbox spec still named. Take the guard before the attached sandbox check, as provider create and update already do, so the check and the delete run against a stable set of sandbox references, and reject an empty name after authorization but before the guard. A new test holds the guard, attaches the provider while the delete waits, and asserts that the delete fails with FailedPrecondition and leaves the provider in place. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * fix(server): release staged credentials when a refresh fails early A provider credential refresh stages the minted values under new credential handles before it commits the provider update, and deletes those handles when validation or persistence fails. Two earlier returns skipped that cleanup. The minted expiry was converted to a protobuf timestamp only after staging, so an expiry outside the timestamp range failed the refresh and left the staged values in the credential driver. A failure to acquire the sandbox mutation guard returned the same way. Convert the expiry before staging anything, so a bad value fails before any handle exists, and delete the staged handles when the guard cannot be acquired. A new test refreshes a stored credential with an expiry of i64::MAX and asserts that the call fails, the provider keeps its original handles, and the credential driver holds the same number of values as before. The guard failure path has no test here, because the SQLite guard used by unit tests cannot fail. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool The cross-object mutation guard took its advisory-lock session from the 10-connection data pool and closed it on every release, so each guarded mutation opened a new database connection, and lock holders competed with their own critical sections for data connections. A lock wait that hit the 10 second lock_timeout surfaced as INTERNAL. Add a lazy, dedicated 4-connection lock pool and a new persistence::mutation_lock module for the lock key, mode and set types. Store::acquire_distributed_mutation_guard now takes a lock set and a deadline, and each lock statement sets lock_timeout to the time left before that deadline, so a cancelled acquisition never waits past it on the server. Returned connections are scrubbed with pg_advisory_unlock_all() and reused, and each return is bounded by 5 seconds so a stalled session cannot keep a pool permit. A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a new LockTimeout error, and so does a wait for a lock connection during which every one was checked out at some point, or that started with less than a second left, too little to open one. The API returns it as UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry delay, because the mutation lock was not acquired, so the request's guarded writes did not run. Only the guard acquisition classifies 55P03; any other 55P03 stays a database error. A lock connection that PostgreSQL does not open by the deadline despite at least a second to do so (refused, out of connection slots, starting up, or not answering, all of which SQLx retries silently) is a database error, not a lock timeout. Mutations still take the legacy global key exclusively, so old and new replicas keep excluding each other during a rolling upgrade. Each PostgreSQL-backed gateway pod can now open up to 14 database connections (10 data plus 4 lock) instead of 10, and the chart's autoscaling.maxReplicas comment says what its default of 4 needs. The SSH host identity lock stays on a data connection, because sandbox creation takes it while holding its mutation guard and a second lock-pool connection there could exhaust the pool. The provider credential refresh lock stays there too, so a refresh still progresses while every lock connection is in use. Part of NVIDIA#3528 Co-authored-by: divesh <[email protected]> Signed-off-by: Emilien Macchi <[email protected]> * perf(server): partition mutation locks by workspace and sandbox Every cross-object mutation, and every lifecycle and supervisor update, took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory lock. Unrelated sandboxes and providers therefore serialized across all replicas, and one slow critical section delayed every other mutation in the fleet. Replace that lock with hierarchical intention locks on a global key, one key per workspace and one key per sandbox, each held shared or exclusive. Global settings and policy writers and platform-scope profile writers hold the global key exclusively. Provider and workspace-scoped profile writers, including DeleteProvider and credential refresh, hold their workspace key exclusively. Sandbox writers and supervisor reports hold their sandbox key exclusively under shared global and workspace keys. Lifecycle, driver-watch and reconcile paths take only process-local keys and keep relying on compare-and-swap across replicas. Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each guard takes its local keys and then its advisory locks in ascending order under one 10 second deadline, and no path nests guards. The settings mutex is removed, since the global and sandbox keys cover it. Sandbox settings are keyed by sandbox name, which a sandbox key no longer covers once another sandbox reuses the name, so a settings update returns NOT_FOUND unless its sandbox still holds the name after the read, and ABORTED if the settings row was recreated before the write. The global key keeps its legacy value, so older replicas, which hold it exclusively for every mutation, still exclude new ones during a rolling upgrade. Guards now also give up after 10 seconds on SQLite, and provider refresh reports a guard timeout as UNAVAILABLE instead of INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND. Startup endpoint-status reconciliation still holds the global key for its whole scan. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * fix(server): reconcile endpoint status on startup one sandbox at a time Startup endpoint-status reconciliation held the global mutation key exclusively for its whole scan. On PostgreSQL that blocked every guarded mutation on every replica until the starting gateway had walked all sandboxes, and the scan first waited behind every in-flight sandbox mutation. It also paged with an offset, so sandboxes deleted mid-scan shifted rows between pages, and a write that raced the scan (a Conflict from a lifecycle update on another replica, or a sandbox deleted after it was listed) failed gateway startup. Walk the sandboxes with keyset paging instead, and reset each candidate under its own sandbox-scoped guard, up to four at a time to match the lock pool. A candidate with a fresh shared owner is skipped before any guard is taken. Under the guard the reconciliation re-checks the owner, re-reads the sandbox, skips it when it is gone or no longer carries endpoint status, and writes against the version it just read. A Conflict or a database error at write time, which is how a row deleted after the re-read surfaces, is retried with a fresh read, at most five attempts per sandbox. A lock timeout on the guard is retried the same way, so a holder of the sandbox, its workspace or the global key that releases within the retry window cannot stop a replica from starting. No lock-timeout retry starts once the scan has run for three lock timeouts, so a scan stuck behind a long holder fails within about 40 seconds, inside the chart's 60-second startup probe, and logs the sandbox it gave up on. Store::list_all_messages loses its only caller and is removed. Reconciliation stays fail-closed: a holder that outlasts the retry window, an owner lookup failure, or a sandbox that keeps changing still fails startup, so stale endpoint success is never served. User-visible change: a restarting gateway no longer stalls mutations of unrelated sandboxes and providers on its peers while it reconciles. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * perf(server): skip the endpoint-status lock when a session was replaced When a supervisor session ends, the gateway resets that sandbox's tool server endpoint evidence unless a replacement session already owns endpoint observation. The replacement check ran only after the reset had read the sandbox and taken its sandbox mutation guard, so a reset whose supervisor had already reconnected still queued behind other mutations of that sandbox, and on PostgreSQL it used a lock-pool connection, only to find it had nothing to do. Check for a replacement first: a live local session, or a fresh owner record on a peer. When one exists, return without reading the sandbox or taking the guard. Otherwise take the guard and repeat the same check under it, because a replacement's pre-acknowledgement reset runs under the same sandbox key and resetting after it would wipe the replacement's fresh evidence. Both checks share one helper, so they cannot drift apart. When no replacement exists, the reset costs one extra owner-row read. This helps most when a replica notices a dead stream late, after the supervisor has already reconnected elsewhere (a network partition or a keepalive expiry). Deferring to a replacement leaves the old session's evidence to the replacement's pre-acknowledgement reset, which can time out on its bounded mutation lock wait. That reset's failure path removed the replacement and released its owner record but scheduled no invalidation, so the old session's success evidence could outlive both sessions. An old session that ended after its replacement registered already deferred this way, and the unguarded check now makes every disconnect that finds the replacement defer too. When the reset fails while the replacement is still current, also schedule the retried invalidation that follows a session end. That failure path keeps the lifecycle demotion from abandon_session_setup, but runs it in the spawned task before the retry and only while this session is still current, so a timed-out reset returns promptly. Part of NVIDIA#3528 Co-authored-by: divesh <[email protected]> Signed-off-by: Emilien Macchi <[email protected]> * test(server): cover staged credential cleanup on a lock timeout A provider credential refresh deletes its staged handles when it cannot acquire the provider mutation guard, but that path had no test: the SQLite guard used by unit tests could not fail. Mutation locks now acquire under a deadline that tests can shorten, so the timeout path can be exercised directly. The new test shortens the lock deadline to 50 ms, holds the workspace mutation guard, and refreshes a provider that already stores an AWS access key. The refresh returns UNAVAILABLE with the MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle, which still resolves to the old key, and the credential driver holds the same number of values as before the refresh. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * feat(server): export mutation lock wait and timeout metrics A handler mutation guard can now wait up to 10 seconds for its keys and then fail with UNAVAILABLE, but nothing showed operators how long guards waited or how often they gave up, short of reading warning logs on each replica. Export openshell_server_mutation_lock_wait_seconds, a histogram of the time one guard took to acquire all of its keys (local registry plus PostgreSQL advisory locks), recorded on success, and openshell_server_mutation_lock_timeouts_total, a counter of guard acquisitions that timed out. Both carry a bounded scope label: global, workspace or sandbox. The platform scope, whose workspace is empty, counts as global. The histogram uses the same 1 ms to 15 s buckets as the relay and peer histograms, and the timeout counter starts at 0 for every scope, so rate() works on an idle replica. The timeout warning now also logs the scope. A guard that fails for another reason, such as a lock connection PostgreSQL did not open, is not counted as a timeout and logs a "mutation lock acquisition failed" warning instead. Lifecycle, driver-watch and reconcile paths take only process-local locks and never time out, so they are not recorded. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * test(server): add PostgreSQL mutation lock integration tests The scoped mutation locks were only unit-tested on SQLite, where the database layer is a no-op. Add ignored postgres_* tests that run the advisory-lock layer against a real PostgreSQL through mise run test:rust:postgres, each in its own disposable schema (TestSchema). persistence/mutation_lock_pg_tests.rs uses two stores on one database as two replicas. It covers disjoint sandbox scopes holding at once, a workspace key blocking only sandboxes in that workspace, the global key blocking every scope, and exclusion in both directions against a raw holder of the legacy global key, whose 55P03 stays a database error outside a guard acquisition. It checks that timed-out and cancelled acquisitions release the keys they already took, that a released lock connection returns to the pool without residual locks, that a release stalled on the network closes its session and the pool recovers, and that the lock pool never grows past its size. It also checks that a full lock pool times out as lock contention, while a lock connection that PostgreSQL never answers fails as a database error. compute/mutation_guard.rs runs the random scope mix across two PostgreSQL runtimes and checks that a by-id guard waits for a workspace writer on the other replica. The provider refresh lock test moves onto the runner too, and checks that a refresh proceeds while every lock connection is held. A bench_postgres_* capacity check measures a paced burst of 500 reconnects 12 ms apart into one replica and fails when the p99 lock wait reaches a fifth of the lock timeout. Machine load moves those waits, so mise run test:rust:postgres skips it and mise run test:rust:postgres:bench runs it alone, through a new OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner. The tests add test-only helpers: the lock pool's size and idle count and the backend pid of a held lock session. Nothing changes outside tests and the test runner. Part of NVIDIA#3528 Co-authored-by: divesh <[email protected]> Signed-off-by: Emilien Macchi <[email protected]> * ci: run the PostgreSQL-backed server tests Branch Checks now runs the postgres_* tests through the same runner developers use, on an x86_64 Linux runner with Docker. They cover lock scope exclusion, interop with replicas still on the old global key, cancellation, lock-pool bounds and connection failures, which no other CI lane checks directly. The load-sensitive bench_postgres_* capacity checks stay local. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * docs: document scoped mutation locks and lock-pool sizing In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * feat(server): check mutation lock ordering in debug builds The mutation lock ordering rules were prose only, so a nested guard on a different key passed every test and could only deadlock under contention. Debug builds now register each held guard with the Tokio task (or the thread, outside a task) that acquired it, and panic before waiting when a local key, a blocking lifecycle gate or the SSH identity lock would break rules 1, 4 or 5. Startup endpoint-status reconciliation and the tests that hold a guard while driving guarded code run under lock_order::branch(), which marks deliberate concurrency within one task. Release builds compile the check out. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * fix(server): do not hold gateway shutdown for the endpoint-status retry When a replacement supervisor session failed its endpoint-status reset, the spawned cleanup task kept the session lifetime guard through retry_endpoint_status_after_supervisor_disconnect. That retry only ends once the reset is durable, so a gateway shutdown during a database or lock outage waited the full 10 s and reported ownership cleanup as incomplete. The guard now covers only the lifecycle demotion, as in finish_supervisor_session, and a SQLite test checks that shutdown completes while the retry is still blocked. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * refactor(server): make the platform profile lock scope explicit Platform-scope provider profile writers (import, update, delete) pass an empty workspace on purpose and need X(global), because they validate sandboxes in every workspace. MutationScope::profiles() now names that escalation and picks Global for the platform scope, instead of relying on MutationScope::Workspace("") to mean it. An empty workspace name now trips a debug assertion, and release builds keep X(global) as a fail-safe that over-locks rather than under-locks. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * perf(server): amortize local mutation lock table sweeps LocalMutationLocks swept its whole table on every key lookup, so a burst of N concurrent acquisitions on one replica cost O(N^2) under a blocking mutex. The table is now swept only when a new key grows it past the larger of 64 and twice the live entries left by the last sweep, which makes acquisition amortized O(1). Released entries, including those of cancelled waiters, stay bounded at about twice the peak live set. The lifecycle gate registry keeps its per-call sweep. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * feat(server): make the database pool ceiling configurable Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5 for on-disk SQLite. One pool is shared by every database-backed RPC, so that number is also the gateway's ceiling on concurrent database work. Once every connection is checked out, callers queue on acquire and sqlx logs "time to acquire exceeded slow threshold"; sandbox creates then time out and are retried, which adds load rather than shedding it. The right ceiling is deployment-specific — it depends on how many gateway replicas share the database and what max_connections the server itself allows — so it cannot be a single number baked into the binary. Expose it on the same config surface as the rest of the gateway's settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the TOML key database_max_connections, resolved in that precedence order. Helm deployments set the key through gatewayConfig. Omitting it keeps each backend's previous value, so existing deployments do not move. Values below 1 are rejected rather than silently replaced by the default: a zero pool would block every acquire, and quietly ignoring a typo would reproduce the ceiling the operator is trying to lift. An in-memory SQLite database stays pinned to one connection, since the database lives inside that connection. Refs: NVIDIA#2561 Signed-off-by: Bryce Wilkinson <[email protected]> (cherry picked from commit 33517b3) [emacchi: dropped the server.dbMaxConnections Helm value; operators set the key through gatewayConfig] Signed-off-by: Emilien Macchi <[email protected]> * feat(server): make the mutation lock pool size configurable The mutation lock pool was fixed at 4 connections per replica. That number rests on one reconnect-burst bench rather than scale testing, so a gateway that serves many more sandboxes needs a way to raise it without a rebuild. Add --db-lock-max-connections, OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections gateway.toml key next to the data pool knob, resolved in the same order; Helm deployments set it through gatewayConfig. The default stays 4, zero is rejected, SQLite ignores it with a warning, and the gateway logs both pool sizes at startup. PostgreSQL now needs at least 2 data connections, because the SSH identity lock keeps one while it queries the pool. The connection sizing docs, the HA guide formula and the debug skill now count data plus lock connections. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * feat(server): export mutation lock errors and lock connections in use A mutation lock acquisition that failed without timing out, such as a lock connection PostgreSQL never opened or a failed lock statement, only logged a warning, and readiness stays healthy because it pings the data pool. Count those failures in openshell_server_mutation_lock_errors_total{scope}, kept apart from timeouts because they point at PostgreSQL capacity or connectivity rather than contention. Export openshell_server_mutation_lock_connections_in_use as a gauge slot held by each checked-out lock connection, next to openshell_server_mutation_lock_connections_capacity, which reads the configured lock pool size; both exist only with PostgreSQL. The metrics catalog, the HA guide's signals and PromQL examples, and the debug skill cover the new series. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> * feat(server): time mutation lock holds and test a stalled holder Nothing bounds how long a mutation guard is held, so a holder stuck in a compute driver, credential backend or middleware call keeps its scope locked while waiters time out. Each guard now records its hold in openshell_server_mutation_lock_hold_seconds{scope} when it is released, and logs a warning when the hold outlasted the lock wait timeout, so operators can find the replica and scope behind those timeouts. A PostgreSQL test with a stalled holder pins the current contract: waiters on its sandbox, workspace and global scopes fail by their deadline, other scopes proceed, the holder keeps its key, and a full lock pool on one replica leaves the other working. The metrics catalog, the HA guide and the debug skill describe the new signals, which observe holds without bounding them. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]> --------- Signed-off-by: Emilien Macchi <[email protected]> Co-authored-by: divesh <[email protected]> Co-authored-by: Bryce Wilkinson <[email protected]>
1 parent a2f0520 commit e30c938

41 files changed

Lines changed: 7570 additions & 430 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/branch-checks.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -313,6 +313,36 @@ jobs:
313313
cargo nextest run --locked --profile ci --workspace --features openshell-server/test-support
314314
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml
315315
316+
rust-postgres-test:
317+
name: Rust PostgreSQL tests
318+
needs: pr_metadata
319+
if: needs.pr_metadata.outputs.should_run == 'true'
320+
# Bare runner: the test script starts a disposable PostgreSQL container
321+
# with the host's Docker.
322+
runs-on: linux-amd64-cpu8
323+
timeout-minutes: 30
324+
steps:
325+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
326+
with:
327+
persist-credentials: false
328+
329+
- uses: ./.github/actions/setup-nix
330+
with:
331+
prepare-shell: "true"
332+
cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }}
333+
334+
- uses: ./.github/actions/setup-rust
335+
with:
336+
cache-key: rust-checks-v2
337+
save-cache: "false"
338+
339+
- name: Test
340+
shell: nix develop -c bash -euo pipefail {0}
341+
env:
342+
CONTAINER_ENGINE: docker
343+
OPENSHELL_TELEMETRY_ENABLED: "false"
344+
run: tasks/scripts/run-postgres-tests.sh --locked --profile ci
345+
316346
rust-build-modes:
317347
name: Rust build modes (${{ matrix.system }})
318348
needs: pr_metadata

‎TESTING.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,28 @@ mise run test:rust # cargo test --workspace
4848

4949
Rust validation checks tracked Cargo lockfiles; run `mise run rust:lockfiles:check` to check them directly. If one is stale, refresh it with Cargo using its adjacent manifest, review the diff, and commit the update.
5050

51+
### PostgreSQL-backed tests
52+
53+
Tests that need a real PostgreSQL server, such as advisory-lock concurrency
54+
across two stores, are `#[ignore]`d and named `postgres_*`. Run them with:
55+
56+
```shell
57+
mise run test:rust:postgres
58+
```
59+
60+
The task starts a disposable PostgreSQL container with Docker or Podman
61+
(set `CONTAINER_ENGINE` to choose), runs the tests one at a time, and removes
62+
the container. Each test works in its own temporary schema, except the replay
63+
and provider refresh lock tests, which use the default schema. To use your own
64+
disposable database, set `OPENSHELL_TEST_POSTGRES_URL`. The task overrides
65+
`OPENSHELL_REPLAY_TEST_DATABASE_URL` so legacy tests use that same database.
66+
Never point it at a database that a running gateway uses: the tests take
67+
fleet-wide advisory locks.
68+
Branch Checks runs them on x86_64 Linux.
69+
Load-sensitive capacity checks are named `bench_postgres_*` instead, so that
70+
task skips them. Run them on an idle machine with
71+
`mise run test:rust:postgres:bench`.
72+
5173
### Native Windows validation
5274

5375
Use `mise run --skip-tools pre-commit` with the existing Rust/MSVC toolchain.

‎crates/openshell-core/src/config.rs‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,24 @@ pub struct Config {
243243
/// Database URL for persistence.
244244
pub database_url: String,
245245

246+
/// Connection ceiling for the persistence pool.
247+
///
248+
/// `None` leaves the backend's built-in default in place. The right
249+
/// ceiling depends on the deployment — how many gateway replicas share
250+
/// the database, and what `max_connections` a Postgres server itself
251+
/// allows — so it cannot be one number baked into the binary. An
252+
/// in-memory `SQLite` database ignores it: the database lives in its single
253+
/// connection.
254+
pub database_max_connections: Option<u32>,
255+
256+
/// Connection ceiling for the `PostgreSQL` mutation lock pool.
257+
///
258+
/// `None` keeps the built-in default of 4. Each mutation guard holds one
259+
/// lock connection, so this bounds how many guarded mutations a replica
260+
/// runs or waits on in `PostgreSQL` at once. `SQLite` has no lock pool and
261+
/// ignores it.
262+
pub database_lock_max_connections: Option<u32>,
263+
246264
/// Explicit compute driver configured for the gateway.
247265
/// `None` enables runtime auto-detection.
248266
pub compute_driver: Option<String>,
@@ -892,6 +910,8 @@ impl Config {
892910
mtls_auth: MtlsAuthConfig::default(),
893911
gateway_jwt: None,
894912
database_url: String::new(),
913+
database_max_connections: None,
914+
database_lock_max_connections: None,
895915
compute_driver: None,
896916
compute_driver_endpoints: BTreeMap::new(),
897917
credential_drivers: Vec::new(),
@@ -944,6 +964,27 @@ impl Config {
944964
self
945965
}
946966

967+
/// Create a new configuration with a database pool connection ceiling.
968+
///
969+
/// `None` keeps the persistence backend's default.
970+
#[must_use]
971+
pub const fn with_database_max_connections(mut self, max_connections: Option<u32>) -> Self {
972+
self.database_max_connections = max_connections;
973+
self
974+
}
975+
976+
/// Create a new configuration with a mutation lock pool connection ceiling.
977+
///
978+
/// `None` keeps the built-in default.
979+
#[must_use]
980+
pub const fn with_database_lock_max_connections(
981+
mut self,
982+
max_connections: Option<u32>,
983+
) -> Self {
984+
self.database_lock_max_connections = max_connections;
985+
self
986+
}
987+
947988
/// Create a new configuration with an explicit compute driver.
948989
#[must_use]
949990
pub fn with_compute_driver(mut self, driver: impl ToString) -> Self {

‎crates/openshell-server/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,8 @@ cargo test -p openshell-server --features test-support,prebuilt-z3
7070
cargo clippy -p openshell-server --all-targets --features test-support,prebuilt-z3 -- -D warnings
7171
```
7272

73-
The ignored `refresh_locks_coordinate_independent_replica_sessions` test requires
73+
The ignored `postgres_refresh_locks_coordinate_independent_replica_sessions` test
74+
runs under `mise run test:rust:postgres` (see `TESTING.md`) and still accepts
7475
`OPENSHELL_REFRESH_TEST_DATABASE_URL` pointing to a disposable PostgreSQL database.
7576
It exercises independent sessions without the local mutex. Do not point this
7677
fixture at an existing deployment database.

‎crates/openshell-server/src/cli.rs‎

Lines changed: 177 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,36 @@ struct RunArgs {
128128
#[arg(long, env = "OPENSHELL_DB_URL")]
129129
db_url: Option<String>,
130130

131+
/// Connection ceiling for the database pool, shared by every
132+
/// database-backed request.
133+
///
134+
/// When unset, the gateway uses its backend default: 10 with Postgres,
135+
/// 5 with an on-disk `SQLite` database. An in-memory `SQLite` database is
136+
/// always a single connection. Postgres needs at least 2, because the SSH
137+
/// identity lock keeps one connection while it queries the pool, so the
138+
/// gateway rejects 1 there at startup. Each replica opens its own pool,
139+
/// plus a mutation lock pool on Postgres (`--db-lock-max-connections`), so
140+
/// keep the total below what the database server itself admits.
141+
#[arg(
142+
long,
143+
env = "OPENSHELL_DB_MAX_CONNECTIONS",
144+
value_parser = clap::value_parser!(u32).range(1..)
145+
)]
146+
db_max_connections: Option<u32>,
147+
148+
/// Connection ceiling for the Postgres mutation lock pool.
149+
///
150+
/// Each mutation guard holds one lock connection, so this bounds how many
151+
/// guarded mutations a replica runs or waits on in Postgres at once.
152+
/// When unset, the gateway uses 4. Each replica opens these on top of
153+
/// its database pool. `SQLite` has no lock pool and ignores it.
154+
#[arg(
155+
long,
156+
env = "OPENSHELL_DB_LOCK_MAX_CONNECTIONS",
157+
value_parser = clap::value_parser!(u32).range(1..)
158+
)]
159+
db_lock_max_connections: Option<u32>,
160+
131161
/// Compute driver configured for this gateway.
132162
///
133163
/// Accepts one registered driver name. When unset, the gateway runs
@@ -516,7 +546,10 @@ fn prepare_server_config_with_drivers(
516546
config = config.with_metrics_bind_address(addr);
517547
}
518548

519-
config = config.with_database_url(db_url);
549+
config = config
550+
.with_database_url(db_url)
551+
.with_database_max_connections(args.db_max_connections)
552+
.with_database_lock_max_connections(args.db_lock_max_connections);
520553
if let Some(driver) = &args.compute_driver {
521554
config = config.with_compute_driver(driver);
522555
}
@@ -1267,6 +1300,18 @@ fn merge_file_into_args(args: &mut RunArgs, file: &GatewayFileSection, matches:
12671300
{
12681301
args.grpc_rate_limit_window_seconds = Some(window);
12691302
}
1303+
if let Some(max_connections) = file.database_max_connections
1304+
&& args.db_max_connections.is_none()
1305+
&& arg_defaulted(matches, "db_max_connections")
1306+
{
1307+
args.db_max_connections = Some(max_connections);
1308+
}
1309+
if let Some(max_connections) = file.database_lock_max_connections
1310+
&& args.db_lock_max_connections.is_none()
1311+
&& arg_defaulted(matches, "db_lock_max_connections")
1312+
{
1313+
args.db_lock_max_connections = Some(max_connections);
1314+
}
12701315
}
12711316

12721317
fn validate_grpc_rate_limit_args(requests: Option<u64>, window_seconds: Option<u64>) -> Result<()> {
@@ -3133,6 +3178,137 @@ grpc_rate_limit_window_seconds = 30
31333178
assert_eq!(args.grpc_rate_limit_window_seconds, Some(30));
31343179
}
31353180

3181+
#[test]
3182+
fn file_db_max_connections_populates_args_when_cli_omits() {
3183+
Environment::new()
3184+
.remove("OPENSHELL_DB_MAX_CONNECTIONS")
3185+
.run(|| {
3186+
let (mut args, matches) =
3187+
parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]);
3188+
assert_eq!(
3189+
args.db_max_connections, None,
3190+
"default leaves the pool alone"
3191+
);
3192+
let file = config_file_from_toml(
3193+
r"
3194+
[openshell.gateway]
3195+
database_max_connections = 64
3196+
",
3197+
);
3198+
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
3199+
3200+
assert_eq!(args.db_max_connections, Some(64));
3201+
});
3202+
}
3203+
3204+
#[test]
3205+
fn env_db_max_connections_overrides_file_value() {
3206+
Environment::new()
3207+
.set("OPENSHELL_DB_MAX_CONNECTIONS", "128")
3208+
.run(|| {
3209+
let (mut args, matches) =
3210+
parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]);
3211+
let file = config_file_from_toml(
3212+
r"
3213+
[openshell.gateway]
3214+
database_max_connections = 64
3215+
",
3216+
);
3217+
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
3218+
3219+
assert_eq!(args.db_max_connections, Some(128));
3220+
});
3221+
}
3222+
3223+
#[test]
3224+
fn db_max_connections_rejects_a_zero_or_unparseable_ceiling() {
3225+
Environment::new()
3226+
.remove("OPENSHELL_DB_MAX_CONNECTIONS")
3227+
.run(|| {
3228+
// Zero would deadlock every `acquire`, and a silent fallback to the
3229+
// default would reproduce the ceiling the operator is trying to lift.
3230+
for bad in ["0", "-5", "many"] {
3231+
assert!(
3232+
command()
3233+
.try_get_matches_from([
3234+
"openshell-gateway",
3235+
"--db-url",
3236+
"sqlite::memory:",
3237+
"--db-max-connections",
3238+
bad,
3239+
])
3240+
.is_err(),
3241+
"input {bad:?} must be rejected"
3242+
);
3243+
}
3244+
});
3245+
}
3246+
3247+
#[test]
3248+
fn file_db_lock_max_connections_populates_args_when_cli_omits() {
3249+
Environment::new()
3250+
.remove("OPENSHELL_DB_LOCK_MAX_CONNECTIONS")
3251+
.run(|| {
3252+
let (mut args, matches) =
3253+
parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]);
3254+
assert_eq!(
3255+
args.db_lock_max_connections, None,
3256+
"default leaves the lock pool alone"
3257+
);
3258+
let file = config_file_from_toml(
3259+
r"
3260+
[openshell.gateway]
3261+
database_lock_max_connections = 8
3262+
",
3263+
);
3264+
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
3265+
3266+
assert_eq!(args.db_lock_max_connections, Some(8));
3267+
});
3268+
}
3269+
3270+
#[test]
3271+
fn env_db_lock_max_connections_overrides_file_value() {
3272+
Environment::new()
3273+
.set("OPENSHELL_DB_LOCK_MAX_CONNECTIONS", "16")
3274+
.run(|| {
3275+
let (mut args, matches) =
3276+
parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]);
3277+
let file = config_file_from_toml(
3278+
r"
3279+
[openshell.gateway]
3280+
database_lock_max_connections = 8
3281+
",
3282+
);
3283+
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
3284+
3285+
assert_eq!(args.db_lock_max_connections, Some(16));
3286+
});
3287+
}
3288+
3289+
#[test]
3290+
fn db_lock_max_connections_rejects_a_zero_or_unparseable_ceiling() {
3291+
Environment::new()
3292+
.remove("OPENSHELL_DB_LOCK_MAX_CONNECTIONS")
3293+
.run(|| {
3294+
// A zero-sized lock pool would time out every guarded mutation.
3295+
for bad in ["0", "-5", "many"] {
3296+
assert!(
3297+
command()
3298+
.try_get_matches_from([
3299+
"openshell-gateway",
3300+
"--db-url",
3301+
"sqlite::memory:",
3302+
"--db-lock-max-connections",
3303+
bad,
3304+
])
3305+
.is_err(),
3306+
"input {bad:?} must be rejected"
3307+
);
3308+
}
3309+
});
3310+
}
3311+
31363312
#[test]
31373313
fn aux_listener_preserves_file_ip_against_public_bind() {
31383314
use std::net::SocketAddr;

0 commit comments

Comments
 (0)