Skip to content

Commit af95972

Browse files
committed
fix(server): scope the SSH identity lock per sandbox
SSH host identity provisioning and cleanup took one fleet-wide advisory key and a process-wide mutex, so create, start, restart and delete ran that step for one sandbox at a time across the fleet, with each waiting create holding a lock-pool connection. The SSH identity guard now takes the legacy key in shared mode and then an exclusive key derived from the sandbox id, on one data-pool connection and one data-pool lock slot. The local mutex is keyed by sandbox id too. Steps for different sandboxes run at once, steps for the same sandbox still exclude each other on every replica, and gateways from earlier releases, which hold the legacy key exclusively, keep excluding new ones during a rolling upgrade. One 10-second deadline now covers a data-pool lock holder's slot, its connection and every lock statement, as it already did for mutation locks, so the second lock statement doesn't extend the wait under create's mutation guard. A lock that times out returns a lock timeout. Part of #4307. Closes #4686 Signed-off-by: Emilien Macchi <[email protected]>
1 parent 0149e87 commit af95972

8 files changed

Lines changed: 611 additions & 73 deletions

File tree

‎crates/openshell-server/src/compute/mutation_guard.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1373,7 +1373,7 @@ mod tests {
13731373
async move {
13741374
let _ssh = runtime
13751375
.store
1376-
.acquire_ssh_identity_mutation_guard()
1376+
.acquire_ssh_identity_mutation_guard("f")
13771377
.await
13781378
.unwrap();
13791379
drop(
@@ -1397,7 +1397,7 @@ mod tests {
13971397
drop(
13981398
runtime
13991399
.store
1400-
.acquire_ssh_identity_mutation_guard()
1400+
.acquire_ssh_identity_mutation_guard("g")
14011401
.await
14021402
.unwrap(),
14031403
);

‎crates/openshell-server/src/persistence/mod.rs‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -412,15 +412,29 @@ impl Store {
412412

413413
/// Independent of the mutation locks: creation already holds its sandbox
414414
/// mutation guard when it provisions a supervisor's durable SSH identity.
415+
///
416+
/// On `PostgreSQL` it takes the legacy
417+
/// [`mutation_lock::SSH_IDENTITY_LOCK_KEY`] shared and then the sandbox's
418+
/// [`mutation_lock::sandbox_ssh_identity_lock_key`] exclusively, in that
419+
/// order on one data-pool connection. Steps for different sandboxes run at
420+
/// once, and gateways from earlier releases, which hold the legacy key
421+
/// exclusively, still exclude every one of them.
415422
pub(crate) async fn acquire_ssh_identity_mutation_guard(
416423
&self,
424+
sandbox_id: &str,
417425
) -> PersistenceResult<SshIdentityMutationGuard> {
418426
lock_order::check(lock_order::Lock::SshIdentity);
419427
match self {
420428
Self::Postgres(store) => Ok(SshIdentityMutationGuard {
421429
_postgres: Some(
422430
store
423-
.acquire_data_pool_lock(mutation_lock::SSH_IDENTITY_LOCK_KEY)
431+
.acquire_data_pool_locks(&[
432+
(mutation_lock::SSH_IDENTITY_LOCK_KEY, LockMode::Shared),
433+
(
434+
mutation_lock::sandbox_ssh_identity_lock_key(sandbox_id),
435+
LockMode::Exclusive,
436+
),
437+
])
424438
.await?,
425439
),
426440
_order: lock_order::hold(lock_order::Lock::SshIdentity),
@@ -449,7 +463,11 @@ impl Store {
449463
let key = i64::from_be_bytes(hash[..8].try_into().expect("eight digest bytes"));
450464
match self {
451465
Self::Postgres(store) => Ok(RefreshMutationGuard {
452-
_postgres: Some(store.acquire_data_pool_lock(key).await?),
466+
_postgres: Some(
467+
store
468+
.acquire_data_pool_locks(&[(key, LockMode::Exclusive)])
469+
.await?,
470+
),
453471
}),
454472
Self::Sqlite(_) => Ok(RefreshMutationGuard { _postgres: None }),
455473
}

‎crates/openshell-server/src/persistence/mutation_lock.rs‎

Lines changed: 54 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,12 @@
2323
//! ascending order, all on one lock-pool connection.
2424
//! 4. A task never acquires a mutation guard or a local lifecycle lock while
2525
//! it holds one: no nesting and no upgrade.
26-
//! 5. The SSH identity key, held on a data-pool connection, is a leaf: its
27-
//! holders take no mutation guard, local key, or lifecycle gate, so sandbox
28-
//! creation may wait for it while holding its guard. Its data-pool lock
29-
//! slot is shared with provider-refresh holders, which are not leaves: a
26+
//! 5. The SSH identity keys, S([`SSH_IDENTITY_LOCK_KEY`]) and then
27+
//! X([`sandbox_ssh_identity_lock_key`]) on one data-pool connection and one
28+
//! data-pool lock slot, together form a leaf: their holders take no
29+
//! mutation guard, local key, or lifecycle gate, so sandbox creation may
30+
//! wait for them while holding its guard. Their data-pool lock slot is
31+
//! shared with provider-refresh holders, which are not leaves: a
3032
//! refresh executor takes S(global) X(workspace) while it holds a slot.
3133
//! With no more slots than the 4 refresh executors
3234
//! (`database_max_connections` up to 9), creation waiting for a slot while
@@ -40,7 +42,10 @@
4042
//!
4143
//! Within each layer every waiter on a key holds only smaller keys, and the
4244
//! local phase ends before the `PostgreSQL` phase starts, so no wait-for cycle
43-
//! can form, except through data-pool lock slots (rule 5). Debug builds check
45+
//! can form, except through data-pool lock slots (rule 5). The SSH identity
46+
//! keys break key order: a waiter on a sandbox's key holds the legacy key,
47+
//! which can be larger. They are always taken in one fixed order, legacy key
48+
//! first (rule 5), which keeps that layer free of cycles. Debug builds check
4449
//! rules 1, 4, and 5 before every acquisition that can wait; see
4550
//! [`super::lock_order`].
4651
//!
@@ -61,13 +66,17 @@ use std::time::Duration;
6166
/// "OPENSHLL" and stay within `PostgreSQL`'s signed 64-bit key space.
6267
pub const GLOBAL_MUTATION_LOCK_KEY: i64 = 0x4f50_454e_5348_4c4c;
6368

64-
/// Advisory-lock key that serializes sandbox SSH host identity provisioning
65-
/// and cleanup across replicas.
69+
/// Legacy compatibility key of sandbox SSH host identity provisioning and
70+
/// cleanup.
6671
///
67-
/// Never change this value: every gateway in a fleet must take the same key.
68-
/// It is held on a data-pool connection, not the lock pool, because sandbox
69-
/// creation takes it while holding its mutation guard. The bytes spell
70-
/// "OSSHHOST".
72+
/// Never change this value: gateways from earlier releases hold it
73+
/// exclusively for every SSH identity step, and this release holds it shared
74+
/// next to [`sandbox_ssh_identity_lock_key`], so old and new replicas keep
75+
/// excluding each other during a rolling upgrade. The shared lock exists only
76+
/// to exclude earlier releases and can be dropped once no supported release
77+
/// takes the key exclusively. It is held on a data-pool connection, not the
78+
/// lock pool, because sandbox creation takes it while holding its mutation
79+
/// guard. The bytes spell "OSSHHOST".
7180
pub const SSH_IDENTITY_LOCK_KEY: i64 = 0x4f53_5348_484f_5354;
7281

7382
/// Upper bound on acquiring one mutation lock set.
@@ -156,6 +165,16 @@ impl MutationLockKey<'_> {
156165
}
157166
}
158167

168+
/// Advisory-lock key of one sandbox's SSH host identity, held exclusively
169+
/// after a shared [`SSH_IDENTITY_LOCK_KEY`].
170+
///
171+
/// Derived like [`MutationLockKey::advisory_key`] but with its own kind, so it
172+
/// never equals the sandbox's mutation lock key, which sandbox creation holds
173+
/// while it takes this one.
174+
pub fn sandbox_ssh_identity_lock_key(sandbox_id: &str) -> i64 {
175+
derived_key(b"ssh-identity", sandbox_id)
176+
}
177+
159178
fn derived_key(kind: &[u8], value: &str) -> i64 {
160179
let digest = Sha256::new()
161180
.chain_update(KEY_DOMAIN)
@@ -276,6 +295,30 @@ mod tests {
276295
);
277296
}
278297

298+
#[test]
299+
fn sandbox_ssh_identity_keys_match_golden_values() {
300+
// Computed independently from the documented byte layout with the
301+
// "ssh-identity" kind. Changing either breaks mutual exclusion with
302+
// running replicas.
303+
assert_eq!(
304+
sandbox_ssh_identity_lock_key("00000000-0000-0000-0000-000000000001"),
305+
1_276_723_582_497_569_200
306+
);
307+
assert_eq!(
308+
sandbox_ssh_identity_lock_key("sb-1"),
309+
-7_153_211_921_910_715_692
310+
);
311+
}
312+
313+
#[test]
314+
fn sandbox_ssh_identity_keys_separate_from_other_keys() {
315+
assert_ne!(
316+
sandbox_ssh_identity_lock_key("sb-1"),
317+
MutationLockKey::Sandbox("sb-1").advisory_key()
318+
);
319+
assert_ne!(sandbox_ssh_identity_lock_key("sb-1"), SSH_IDENTITY_LOCK_KEY);
320+
}
321+
279322
#[test]
280323
fn reserved_keys_are_remapped() {
281324
assert_ne!(

0 commit comments

Comments
 (0)