Skip to content

Commit 0149e87

Browse files
committed
fix(server): bound data-pool lock holders
The SSH identity and provider-refresh locks each hold their PostgreSQL advisory lock on a data-pool connection, and their holders then query the same pool. At a small database_max_connections, a few holders could take every data connection and wait for one more until sqlx's 30-second acquire timeout, stalling every database request on the replica. Each PostgresStore now has max(1, max_connections / 2) data-pool lock slots. A holder takes a slot before its connection and keeps it until that connection has closed, so every holder can still get a second connection for its queries. A slot that stays busy for the 10-second mutation lock timeout fails with a lock timeout and a warning. The floor rationale in the config docs, man page and startup error now names both holders, and mutation_lock.rs documents the slot-sharing cycle the timeouts break at small pools. Part of NVIDIA#4307. Signed-off-by: Emilien Macchi <[email protected]>
1 parent 634ca7d commit 0149e87

10 files changed

Lines changed: 347 additions & 44 deletions

File tree

‎crates/openshell-server/src/cli.rs‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -136,11 +136,14 @@ struct RunArgs {
136136
///
137137
/// When unset, the gateway uses its backend default: 10 with Postgres,
138138
/// 5 with an on-disk `SQLite` database. An in-memory `SQLite` database is
139-
/// always a single connection. Postgres needs at least 2, because the SSH
140-
/// identity lock keeps one connection while it queries the pool, so the
141-
/// gateway rejects 1 there at startup. Each replica opens its own pool,
142-
/// plus a mutation lock pool on Postgres (`--db-lock-max-connections`), so
143-
/// keep the total below what the database server itself admits.
139+
/// always a single connection. Postgres needs at least 2, so the gateway
140+
/// rejects 1 there at startup: an SSH identity or provider-refresh lock
141+
/// holder keeps one data connection for its lock and queries the pool on
142+
/// another, and the gateway caps those holders at half of
143+
/// `database_max_connections` per replica. Each replica opens its own
144+
/// pool, plus a mutation lock pool on Postgres
145+
/// (`--db-lock-max-connections`), so keep the total below what the
146+
/// database server itself admits.
144147
#[arg(
145148
long,
146149
env = "OPENSHELL_DB_MAX_CONNECTIONS",

‎crates/openshell-server/src/config_file.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,10 @@ pub struct GatewayFileSection {
114114
/// which must stay under the server's own `max_connections`. Omit to keep
115115
/// the backend default (10 for Postgres, 5 for on-disk `SQLite`); an
116116
/// in-memory `SQLite` database is always a single connection. Postgres
117-
/// needs at least 2, because the SSH identity lock keeps one connection
118-
/// while it queries the pool.
117+
/// needs at least 2: an SSH identity or provider-refresh lock holder keeps
118+
/// one data connection for its lock and queries the pool on another, and
119+
/// the gateway caps those holders at half of `database_max_connections`
120+
/// per replica.
119121
#[serde(default)]
120122
pub database_max_connections: Option<u32>,
121123

‎crates/openshell-server/src/persistence/mod.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -481,8 +481,9 @@ impl Store {
481481
{
482482
return Err(CoreError::config(format!(
483483
"database_max_connections (--db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS) \
484-
is {configured}, but PostgreSQL needs at least {}: the SSH identity lock \
485-
keeps one data connection while it queries the pool",
484+
is {configured}, but PostgreSQL needs at least {}: an SSH identity or \
485+
provider-refresh lock holder keeps one data connection for its lock and \
486+
queries the pool on another",
486487
postgres::MIN_MAX_CONNECTIONS
487488
)));
488489
}

‎crates/openshell-server/src/persistence/mutation_lock.rs‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,24 @@
2525
//! it holds one: no nesting and no upgrade.
2626
//! 5. The SSH identity key, held on a data-pool connection, is a leaf: its
2727
//! holders take no mutation guard, local key, or lifecycle gate, so sandbox
28-
//! creation may wait for it while holding its guard.
28+
//! creation may wait for it while holding its guard. Its data-pool lock
29+
//! slot is shared with provider-refresh holders, which are not leaves: a
30+
//! refresh executor takes S(global) X(workspace) while it holds a slot.
31+
//! With no more slots than the 4 refresh executors
32+
//! (`database_max_connections` up to 9), creation waiting for a slot while
33+
//! executors wait for its keys can form a cycle, which only
34+
//! [`MUTATION_LOCK_TIMEOUT`] on the slot and lock waits breaks.
35+
//! Reserving slots for SSH identity would remove it.
36+
//! 6. A data-pool lock holder never waits for another data-pool lock. At the
37+
//! floor there is one slot, so nesting would wait on itself until the slot
38+
//! deadline. Debug builds don't check this rule, because both holders
39+
//! acquire inside spawned tasks.
2940
//!
3041
//! Within each layer every waiter on a key holds only smaller keys, and the
3142
//! local phase ends before the `PostgreSQL` phase starts, so no wait-for cycle
32-
//! can form. Debug builds check rules 1, 4, and 5 before every acquisition
33-
//! that can wait; see [`super::lock_order`].
43+
//! can form, except through data-pool lock slots (rule 5). Debug builds check
44+
//! rules 1, 4, and 5 before every acquisition that can wait; see
45+
//! [`super::lock_order`].
3446
//!
3547
//! The global key is the legacy cross-object key. Gateways from earlier
3648
//! releases hold it exclusively for every mutation, which conflicts with every

‎crates/openshell-server/src/persistence/mutation_lock_pg_tests.rs‎

Lines changed: 119 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,21 @@
44
//! `PostgreSQL` tests of the mutation advisory locks: exclusion between two
55
//! stores (as between two gateway replicas), exclusion against the legacy
66
//! global key, cleanup after timed-out and cancelled acquisitions, a stalled
7-
//! holder, the lock-pool bound and its in-use gauge, and lock connections that
8-
//! `PostgreSQL` does not open.
7+
//! holder, the lock-pool bound and its in-use gauge, lock connections that
8+
//! `PostgreSQL` does not open, and the bound on data-pool lock holders at the
9+
//! smallest data pool.
910
//!
1011
//! Advisory locks are database-wide, not per schema, so every test uses
1112
//! random workspace and sandbox ids, and `mise run test:rust:postgres` runs
12-
//! the tests one at a time. Apart from the pool-size test, which connects as
13-
//! the gateway does, the stores here run no migrations: the schema only
14-
//! scopes their connections.
13+
//! the tests one at a time. Apart from the pool-size and SSH identity tests,
14+
//! which connect as the gateway does, the stores here run no migrations: the
15+
//! schema only scopes their connections.
1516
1617
use super::mutation_lock::{
1718
GLOBAL_MUTATION_LOCK_KEY, LOCK_CONNECTION_MIN_BUDGET, MUTATION_LOCK_POOL_MAX_CONNECTIONS,
18-
MUTATION_LOCK_TIMEOUT, MutationLockKey, MutationLockSet,
19+
MUTATION_LOCK_TIMEOUT, MutationLockKey, MutationLockSet, SSH_IDENTITY_LOCK_KEY,
1920
};
20-
use super::postgres::LOCK_CONNECTION_RELEASE_TIMEOUT;
21+
use super::postgres::{LOCK_CONNECTION_RELEASE_TIMEOUT, MIN_MAX_CONNECTIONS};
2122
use super::test_postgres::TestSchema;
2223
use super::{
2324
DistributedMutationGuard, PersistenceError, PersistenceResult, PostgresStore, Store,
@@ -181,6 +182,117 @@ async fn postgres_concurrent_sandbox_creates_with_ssh_identity_do_not_exhaust_lo
181182
schema.drop_schema().await;
182183
}
183184

185+
/// Launch authentication for a supervisor whose SSH host key is not yet
186+
/// resolved.
187+
fn ssh_launch_authentication() -> openshell_core::jwt::SandboxLaunchAuthentication {
188+
use openshell_core::jwt::{
189+
CredentialEpoch, SandboxLaunchAuthentication, SecretJwt, SessionRotation,
190+
SupervisorAuthBundle,
191+
};
192+
use openshell_core::{SandboxSessionId, sandbox_generation::SandboxGenerationId};
193+
194+
SandboxLaunchAuthentication {
195+
supervisor: SupervisorAuthBundle {
196+
session_id: SandboxSessionId::new(),
197+
runtime_generation: SandboxGenerationId::parse("generation").unwrap(),
198+
session_rotation: SessionRotation::new(1).unwrap(),
199+
auth_epoch: CredentialEpoch::new(1).unwrap(),
200+
gateway_token: SecretJwt::parse("gateway-token").unwrap(),
201+
gateway_expires_at: 0,
202+
sandbox_token: SecretJwt::parse("sandbox-token").unwrap(),
203+
sandbox_expires_at: 0,
204+
ssh_host_private_key: None,
205+
ocsf_otlp_headers: None,
206+
},
207+
gateway_id: "gateway".to_string(),
208+
verification_keys: Vec::new(),
209+
}
210+
}
211+
212+
/// At the smallest data pool `PostgreSQL` accepts, a provider-refresh holder
213+
/// and an SSH identity holder on one replica leave a data connection free for
214+
/// queries: the SSH identity holder waits for the one data-pool lock slot
215+
/// instead of taking the last connection.
216+
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
217+
#[ignore = "requires a disposable PostgreSQL database in OPENSHELL_TEST_POSTGRES_URL; run mise run test:rust:postgres"]
218+
async fn postgres_data_pool_lock_holders_leave_a_connection_free_at_floor() {
219+
use crate::credentials::CredentialRuntime;
220+
use crate::ssh_identity::SshIdentityStore;
221+
use openshell_core::proto::Sandbox;
222+
use openshell_core::proto::datamodel::v1::ObjectMeta;
223+
use std::sync::Arc;
224+
225+
let fixture = LockFixture::new().await;
226+
let store =
227+
Store::connect_with_pool_sizes(fixture.schema.url(), Some(MIN_MAX_CONNECTIONS), None)
228+
.await
229+
.expect("connect a store with the smallest data pool");
230+
let shared = Arc::new(store.clone());
231+
let config = crate::Config::new(None)
232+
.with_credential_drivers(["test-static"])
233+
.with_default_credential_driver(Some("test-static"));
234+
let identities = SshIdentityStore::new(
235+
shared.clone(),
236+
CredentialRuntime::from_config_with_store(&config, shared).unwrap(),
237+
);
238+
let id = random_id("sb");
239+
let sandbox = Sandbox {
240+
metadata: Some(ObjectMeta {
241+
id: id.clone(),
242+
name: id.clone(),
243+
workspace: "default".to_string(),
244+
..Default::default()
245+
}),
246+
..Default::default()
247+
};
248+
store
249+
.put_message(&sandbox)
250+
.await
251+
.expect("store the sandbox");
252+
253+
let refresh = store
254+
.acquire_refresh_guard(&random_id("provider"), "ACCESS_TOKEN")
255+
.await
256+
.expect("hold a provider refresh lock");
257+
let preparing = {
258+
let identities = identities.clone();
259+
let mut sandbox = sandbox.clone();
260+
tokio::spawn(async move {
261+
identities
262+
.prepare(&mut sandbox, &mut ssh_launch_authentication())
263+
.await
264+
.map(|()| sandbox)
265+
})
266+
};
267+
// Long enough for the SSH identity holder to take a data connection if
268+
// nothing stops it.
269+
tokio::time::sleep(Duration::from_secs(1)).await;
270+
271+
tokio::time::timeout(PROCEEDS_WITHIN, store.get_message::<Sandbox>(&id))
272+
.await
273+
.expect("a query must find a free data connection while both lock holders run")
274+
.expect("load the sandbox")
275+
.expect("the sandbox exists");
276+
assert_eq!(
277+
fixture.lock_count(SSH_IDENTITY_LOCK_KEY).await,
278+
0,
279+
"the SSH identity holder waits for a data-pool lock slot, not in PostgreSQL"
280+
);
281+
282+
drop(refresh);
283+
let prepared = tokio::time::timeout(PROCEEDS_WITHIN, preparing)
284+
.await
285+
.expect("the SSH identity holder proceeds once the refresh lock is released")
286+
.expect("prepare task")
287+
.expect("prepare the SSH identity");
288+
assert!(
289+
!prepared.host_key_fingerprint.is_empty(),
290+
"prepare publishes the SSH host key fingerprint"
291+
);
292+
293+
fixture.finish(vec![store]).await;
294+
}
295+
184296
/// Drop client traffic while keeping sockets open, like a failed network path.
185297
/// Client EOF still closes the upstream socket so `PostgreSQL` can release locks.
186298
struct StallingProxy {

0 commit comments

Comments
 (0)