Skip to content

Commit 49df4d7

Browse files
authored
fix(server): drain supervisor ownership cleanup on shutdown (NVIDIA#3547)
Closes NVIDIA#3546 Signed-off-by: Evan Lezar <[email protected]>
1 parent bdffa10 commit 49df4d7

5 files changed

Lines changed: 258 additions & 12 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1044,6 +1044,15 @@ The same relay pattern backs interactive SSH, command execution, file sync, and
10441044
local service forwarding. The gateway tracks live sessions in memory and
10451045
persists session records so tokens can expire or be revoked.
10461046

1047+
Graceful gateway shutdown closes supervisor-session admission before stopping
1048+
local compute. It then signals the remaining control sessions to exit and waits
1049+
up to ten seconds for their cleanup, including conditional deletion of persisted
1050+
ownership. Pending connection setup and sessions already removed from the live
1051+
registry remain tracked until cleanup finishes. This lets a replacement
1052+
supervisor claim ownership immediately after restart without deleting a newer
1053+
replica's claim. An incomplete drain is reported as a shutdown error. Closing
1054+
these control sessions does not stop Kubernetes-owned workloads.
1055+
10471056
Relay liveness has two backstops so a reset supervisor session cannot leave a
10481057
request parked forever. The gateway runs server-side HTTP/2 keepalive on
10491058
supervisor connections, and each exec relay's SSH client uses SSH keepalive: an

‎crates/openshell-server/src/lib.rs‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1016,17 +1016,26 @@ pub(crate) async fn run_server(
10161016
shutdown_signal().await;
10171017
info!("Shutdown signal received; stopping gateway");
10181018
state.gateway_shutting_down.store(true, Ordering::Release);
1019+
state.supervisor_sessions.close_admission();
10191020
let _ = shutdown_tx.send(true);
10201021

10211022
if let Err(err) = listener_task.await {
10221023
warn!(error = %err, "Gateway listener task failed during shutdown");
10231024
}
10241025

1025-
state
1026-
.compute
1027-
.cleanup_on_shutdown()
1028-
.await
1026+
let compute_cleanup = state.compute.cleanup_on_shutdown().await;
1027+
// A stopped supervisor may still have a detached task deleting its owner
1028+
// record. Drain it even when compute cleanup failed before exiting Tokio.
1029+
let session_cleanup = state
1030+
.supervisor_sessions
1031+
.shutdown(Duration::from_secs(10))
1032+
.await;
1033+
if let Err(err) = &session_cleanup {
1034+
warn!(error = %err, "Gateway supervisor session cleanup incomplete");
1035+
}
1036+
compute_cleanup
10291037
.map_err(|err| Error::execution(format!("gateway shutdown cleanup failed: {err}")))?;
1038+
session_cleanup.map_err(Error::execution)?;
10301039

10311040
Ok(())
10321041
}

‎crates/openshell-server/src/supervisor_session.rs‎

Lines changed: 222 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33

44
use std::collections::HashMap;
55
use std::pin::Pin;
6-
use std::sync::atomic::Ordering;
6+
use std::sync::atomic::{AtomicBool, Ordering};
77
use std::sync::{Arc, Mutex};
88
use std::time::{Duration, Instant};
99

10-
use tokio::sync::{mpsc, oneshot};
10+
use tokio::sync::{OwnedRwLockReadGuard, RwLock, mpsc, oneshot, watch};
1111
use tokio_stream::wrappers::ReceiverStream;
1212
use tonic::metadata::{Ascii, MetadataValue};
1313
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
@@ -20,8 +20,8 @@ use openshell_core::proto::{
2020
PeerRelayFrame, PeerRelayInit, ProviderReadinessObservation, RelayFrame, RelayInit, RelayOpen,
2121
ReportEndpointStatusRequest, ReportEndpointStatusResponse, ReportMainProcessExitRequest,
2222
ReportMainProcessExitResponse, ReportProviderReadinessRequest, ReportProviderReadinessResponse,
23-
Sandbox, SandboxPhase, SessionAccepted, SshRelayTarget, SupervisorMessage, gateway_message,
24-
open_shell_client, peer_relay_frame, relay_open, supervisor_message,
23+
Sandbox, SandboxPhase, SessionAccepted, SshRelayTarget, SupervisorHello, SupervisorMessage,
24+
gateway_message, open_shell_client, peer_relay_frame, relay_open, supervisor_message,
2525
};
2626
use openshell_core::transport_errors::is_expected_transport_close_status;
2727

@@ -324,6 +324,11 @@ pub struct SupervisorSessionRegistry {
324324
sessions: Mutex<HashMap<String, LiveSession>>,
325325
/// `channel_id` -> oneshot sender for the reverse CONNECT stream.
326326
pending_relays: Mutex<HashMap<String, PendingRelay>>,
327+
/// Read guards cover owner publication through final cleanup, even after
328+
/// a session has left `sessions`. Shutdown waits for the write guard.
329+
session_lifetimes: Arc<RwLock<()>>,
330+
admission_closed: AtomicBool,
331+
shutdown: watch::Sender<bool>,
327332
}
328333

329334
struct PendingRelay {
@@ -355,6 +360,36 @@ impl SupervisorSessionRegistry {
355360
Self::default()
356361
}
357362

363+
fn track_session(&self) -> Result<OwnedRwLockReadGuard<()>, Status> {
364+
// Acquire tracking BEFORE checking admission. A concurrent shutdown
365+
// either sees this reader and waits for it, or prevents its admission.
366+
let lifetime = Arc::clone(&self.session_lifetimes)
367+
.try_read_owned()
368+
.map_err(|_| Status::unavailable("gateway is shutting down"))?;
369+
if self.admission_closed.load(Ordering::Acquire) {
370+
return Err(Status::unavailable("gateway is shutting down"));
371+
}
372+
Ok(lifetime)
373+
}
374+
375+
/// Prevent existing HTTP connections from creating new owner records.
376+
pub(crate) fn close_admission(&self) {
377+
self.admission_closed.store(true, Ordering::Release);
378+
}
379+
380+
/// Close control sessions and wait for tracked ownership cleanup. Call
381+
/// after compute shutdown so supervisors can finish normal stop reporting.
382+
pub(crate) async fn shutdown(&self, timeout: Duration) -> Result<(), String> {
383+
self.close_admission();
384+
self.shutdown.send_replace(true);
385+
tokio::time::timeout(timeout, self.session_lifetimes.write())
386+
.await
387+
.map(|_| ())
388+
.map_err(|_| {
389+
format!("supervisor session ownership cleanup did not complete within {timeout:?}")
390+
})
391+
}
392+
358393
/// Register a live supervisor session for the given sandbox.
359394
///
360395
/// If a previous session exists for the same sandbox, its shutdown signal
@@ -1761,6 +1796,34 @@ pub async fn handle_connect_supervisor(
17611796
// supervisors remain usable but cannot assert provider installation.
17621797
let provider_readiness = ProviderReadinessEvidence::from_hello(&hello)?;
17631798

1799+
let session_lifetime = state.supervisor_sessions.track_session()?;
1800+
let state = Arc::clone(state);
1801+
// Keep setup alive if the RPC caller disconnects after publication. The
1802+
// tracking guard moves into the session task or outlives early cleanup.
1803+
tokio::spawn(establish_supervisor_session(
1804+
state,
1805+
inbound,
1806+
hello,
1807+
provider_readiness,
1808+
session_lifetime,
1809+
))
1810+
.await
1811+
.map_err(|error| Status::internal(format!("supervisor session setup failed: {error}")))?
1812+
}
1813+
1814+
async fn establish_supervisor_session(
1815+
state: Arc<ServerState>,
1816+
mut inbound: tonic::Streaming<SupervisorMessage>,
1817+
hello: SupervisorHello,
1818+
provider_readiness: ProviderReadinessEvidence,
1819+
session_lifetime: OwnedRwLockReadGuard<()>,
1820+
) -> Result<
1821+
Response<
1822+
Pin<Box<dyn tokio_stream::Stream<Item = Result<GatewayMessage, Status>> + Send + 'static>>,
1823+
>,
1824+
Status,
1825+
> {
1826+
let sandbox_id = hello.sandbox_id.clone();
17641827
let session_id = Uuid::new_v4().to_string();
17651828
let owner_peer_endpoint = state.peer_endpoint.as_deref().map_or_else(
17661829
|| local_owner_endpoint(&state.replica_id),
@@ -1808,7 +1871,7 @@ pub async fn handle_connect_supervisor(
18081871
// results before acknowledging the session so it cannot inherit evidence
18091872
// reported by the superseded stream.
18101873
if let Err(error) = crate::grpc::policy::reset_endpoint_status_for_supervisor_session(
1811-
state,
1874+
&state,
18121875
&sandbox_id,
18131876
&session_id,
18141877
)
@@ -1905,9 +1968,10 @@ pub async fn handle_connect_supervisor(
19051968
}
19061969

19071970
// Step 4: Spawn the session loop that reads inbound messages.
1908-
let state_clone = Arc::clone(state);
1971+
let state_clone = Arc::clone(&state);
19091972
let sandbox_id_clone = sandbox_id.clone();
19101973
tokio::spawn(async move {
1974+
let _session_lifetime = session_lifetime;
19111975
let mut owner_guard = owner_guard;
19121976
run_session_loop(
19131977
&state_clone,
@@ -2030,13 +2094,18 @@ async fn run_session_loop(
20302094
mut shutdown_rx: oneshot::Receiver<()>,
20312095
owner_guard: &mut OwnerGuard,
20322096
) {
2097+
let mut gateway_shutdown = state.supervisor_sessions.shutdown.subscribe();
20332098
let heartbeat_interval = Duration::from_secs(u64::from(HEARTBEAT_INTERVAL_SECS));
20342099
let mut heartbeat_timer = tokio::time::interval(heartbeat_interval);
20352100
// Skip the first immediate tick.
20362101
heartbeat_timer.tick().await;
20372102

20382103
loop {
20392104
tokio::select! {
2105+
() = async { let _ = gateway_shutdown.wait_for(|shutdown| *shutdown).await; } => {
2106+
info!(sandbox_id = %sandbox_id, session_id = %session_id, "supervisor session: gateway shutting down");
2107+
break;
2108+
}
20402109
_ = &mut shutdown_rx => {
20412110
info!(sandbox_id = %sandbox_id, session_id = %session_id, "supervisor session: superseded by reconnect, shutting down");
20422111
break;
@@ -2214,6 +2283,153 @@ mod tests {
22142283
oneshot::channel::<()>().0
22152284
}
22162285

2286+
#[tokio::test]
2287+
async fn shutdown_waits_for_owner_release_after_session_leaves_registry() {
2288+
let registry = Arc::new(SupervisorSessionRegistry::new());
2289+
let owner_index = Arc::new(SupervisorOwnerIndex::new(test_store().await, OWNER_TTL));
2290+
let lifetime = registry.track_session().unwrap();
2291+
let owner = owner_index
2292+
.publish(
2293+
"sb-1",
2294+
"old-session",
2295+
"old-instance",
2296+
1,
2297+
"old-replica",
2298+
"local://old",
2299+
)
2300+
.await
2301+
.unwrap();
2302+
let (tx, _rx) = mpsc::channel(1);
2303+
registry.register("sb-1".into(), "old-session".into(), tx, make_shutdown());
2304+
2305+
let (removed_tx, removed_rx) = oneshot::channel();
2306+
let (release_tx, release_rx) = oneshot::channel();
2307+
let cleanup_registry = Arc::clone(&registry);
2308+
let cleanup_index = owner_index.clone();
2309+
let cleanup = tokio::spawn(async move {
2310+
let _lifetime = lifetime;
2311+
cleanup_registry.remove_if_current("sb-1", "old-session");
2312+
removed_tx.send(()).unwrap();
2313+
release_rx.await.unwrap();
2314+
cleanup_index.release_if_current(&owner).await.unwrap();
2315+
});
2316+
removed_rx.await.unwrap();
2317+
assert!(registry.sessions.lock().unwrap().is_empty());
2318+
2319+
let shutdown = registry.shutdown(Duration::from_secs(5));
2320+
tokio::pin!(shutdown);
2321+
assert!(futures_util::poll!(&mut shutdown).is_pending());
2322+
assert!(matches!(
2323+
owner_index
2324+
.publish(
2325+
"sb-1",
2326+
"new-session",
2327+
"new-instance",
2328+
1,
2329+
"new-replica",
2330+
"local://new"
2331+
)
2332+
.await,
2333+
Err(OwnerError::AlreadyOwned)
2334+
));
2335+
2336+
release_tx.send(()).unwrap();
2337+
shutdown.await.unwrap();
2338+
cleanup.await.unwrap();
2339+
assert!(owner_index.read("sb-1").await.unwrap().is_none());
2340+
owner_index
2341+
.publish(
2342+
"sb-1",
2343+
"new-session",
2344+
"new-instance",
2345+
1,
2346+
"new-replica",
2347+
"local://new",
2348+
)
2349+
.await
2350+
.expect("restart can immediately claim ownership after shutdown");
2351+
}
2352+
2353+
#[tokio::test]
2354+
async fn shutdown_waits_for_admitted_setup_and_rejects_new_connections() {
2355+
let registry = SupervisorSessionRegistry::new();
2356+
// An RPC has been admitted but has not yet registered a live session.
2357+
let lifetime = registry.track_session().unwrap();
2358+
registry.close_admission();
2359+
assert_eq!(
2360+
registry.track_session().unwrap_err().code(),
2361+
tonic::Code::Unavailable
2362+
);
2363+
// Closing admission alone must leave stop reporting available.
2364+
assert!(!*registry.shutdown.borrow());
2365+
2366+
let shutdown = registry.shutdown(Duration::from_secs(5));
2367+
tokio::pin!(shutdown);
2368+
assert!(futures_util::poll!(&mut shutdown).is_pending());
2369+
// An admitted setup that starts its session loop after cancellation
2370+
// must still observe the shutdown signal immediately.
2371+
let mut late_subscriber = registry.shutdown.subscribe();
2372+
assert!(*late_subscriber.wait_for(|closing| *closing).await.unwrap());
2373+
drop(lifetime);
2374+
shutdown.await.unwrap();
2375+
assert_eq!(
2376+
registry.track_session().unwrap_err().code(),
2377+
tonic::Code::Unavailable
2378+
);
2379+
}
2380+
2381+
#[tokio::test]
2382+
async fn shutdown_cleanup_does_not_delete_replacement_owner() {
2383+
let registry = Arc::new(SupervisorSessionRegistry::new());
2384+
let owner_index = SupervisorOwnerIndex::new(test_store().await, OWNER_TTL);
2385+
let lifetime = registry.track_session().unwrap();
2386+
let old = owner_index
2387+
.publish("sb-1", "old", "instance", 1, "replica-a", "local://a")
2388+
.await
2389+
.unwrap();
2390+
let replacement = owner_index
2391+
.publish("sb-1", "new", "instance", 2, "replica-b", "local://b")
2392+
.await
2393+
.unwrap();
2394+
let shutdown = registry.shutdown(Duration::from_secs(5));
2395+
tokio::pin!(shutdown);
2396+
assert!(futures_util::poll!(&mut shutdown).is_pending());
2397+
owner_index.release_if_current(&old).await.unwrap();
2398+
drop(lifetime);
2399+
shutdown.await.unwrap();
2400+
let persisted = owner_index.read("sb-1").await.unwrap().unwrap();
2401+
assert_eq!(persisted.session_id, replacement.session_id);
2402+
assert_eq!(persisted.owner_replica_id, replacement.owner_replica_id);
2403+
}
2404+
2405+
#[tokio::test]
2406+
async fn shutdown_reports_bounded_failure_when_cleanup_stalls() {
2407+
let registry = SupervisorSessionRegistry::new();
2408+
let lifetime = registry.track_session().unwrap();
2409+
let error = registry
2410+
.shutdown(Duration::from_millis(10))
2411+
.await
2412+
.unwrap_err();
2413+
assert!(error.contains("ownership cleanup did not complete"));
2414+
assert!(*registry.shutdown.borrow());
2415+
assert_eq!(
2416+
registry.track_session().unwrap_err().code(),
2417+
tonic::Code::Unavailable
2418+
);
2419+
drop(lifetime);
2420+
registry.shutdown(Duration::from_secs(1)).await.unwrap();
2421+
}
2422+
2423+
#[tokio::test]
2424+
async fn shutdown_without_sessions_completes_and_closes_admission() {
2425+
let registry = SupervisorSessionRegistry::new();
2426+
registry.shutdown(Duration::from_secs(1)).await.unwrap();
2427+
assert_eq!(
2428+
registry.track_session().unwrap_err().code(),
2429+
tonic::Code::Unavailable
2430+
);
2431+
}
2432+
22172433
#[test]
22182434
fn peer_tls_client_config_requires_certificate_and_key_together() {
22192435
let config = PeerTlsClientConfig {

‎docs/reference/sandbox-compute-drivers.mdx‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,19 @@ Delete remains independent and removes compute plus driver-owned persistent
3333
state. While a sandbox is stopped, gateway access paths and exposed services
3434
remain unavailable.
3535

36-
Restarting the gateway preserves this intent. The gateway does not stop Docker
37-
or Podman containers during shutdown. At startup it sends idempotent start
36+
Restarting the gateway preserves this intent. During graceful shutdown, the
37+
gateway stops running-intent Docker, Podman, and MicroVM sandboxes through their
38+
drivers without recording an explicit user stop. At startup it sends idempotent start
3839
requests for Docker, Podman, and MicroVM sandboxes that were intended to run;
3940
already-running resources are unchanged, retained stopped compute is restarted,
4041
and explicitly stopped sandboxes remain stopped. Kubernetes workloads continue
4142
running independently of the gateway process.
4243

44+
Before exiting, the gateway waits up to ten seconds for supervisor-session
45+
ownership cleanup so replacement supervisors can reconnect after restart.
46+
If tracked cleanup exceeds this deadline, the gateway reports a shutdown
47+
error. Persistence errors during ownership release are logged separately.
48+
4349
The gateway forwards one exact, persisted main-process specification to every
4450
driver. Drivers serialize that specification in
4551
`OPENSHELL_MAIN_PROCESS_SPEC`; they do not install an idle `sleep` workload or

‎skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,12 @@ not follow this local gateway lifecycle. Internal and external drivers follow
289289
the same rule: `GetCapabilities.gateway_manages_lifecycle` must be true for the
290290
gateway to run shutdown and startup sweeps.
291291

292+
The gateway also drains supervisor-session ownership cleanup before exiting.
293+
If shutdown reports `Gateway supervisor session cleanup incomplete`, inspect
294+
the associated persistence errors: a stopped supervisor's owner record may
295+
remain until its lease expires and temporarily block reconnection. Successful
296+
compute stop alone does not confirm that session cleanup finished.
297+
292298
### Step 5: Check Podman-Backed Gateways
293299

294300
```bash

0 commit comments

Comments
 (0)