Skip to content

Commit 8758cbf

Browse files
authored
feat(server): keep pool sandbox sessions on their supervisor's gateway (NVIDIA#5052)
* feat(server): place pool sandbox sessions with their supervisor's replica A shared supervisor pool sandbox's session now goes to the gateway replica serving its owning member's stream, instead of the replica the ring picks by sandbox ID. Each pool reconciliation pass publishes the sandbox-to-replica map; sessions fall back to the ring when the parent is unknown, not a live peer, or this replica is shutting down. Signed-off-by: divesh <[email protected]> * feat(server): move live pool sessions to their supervisor's replica A pool sandbox session served by a replica other than the one serving its owning member is redirected there on its next heartbeat, reusing the shutdown handoff so the sandbox stays Ready. This re-homes sessions left on the wrong replica by a rolling restart. Signed-off-by: divesh <[email protected]> * refactor(server): drop gateway-side pool session placement Removes the per-sandbox parent map and the heartbeat relocation; pool sessions will dial their supervisor's gateway directly instead. Signed-off-by: divesh <[email protected]> * feat(supervisor): dial pool sandbox sessions to the host's gateway replica HostAccepted now carries the accepting replica's peer endpoint. A shared host's sandbox sessions dial it directly, so each session runs on the replica serving its supervisor. A stopping replica sheds its host streams, and redirected sessions follow the host to its next replica. Closes NVIDIA#5051. Signed-off-by: divesh <[email protected]> * test(e2e): keep pool sessions on their supervisor's gateway through failover Add the shared-supervisor-tree lane: two gateway replicas on PostgreSQL with a two-member supervisor pool. After start, a gateway crash, a gateway rolling restart, and a supervisor crash, every pool sandbox's session runs on the replica serving its supervisor and exec reaches every sandbox. Signed-off-by: divesh <[email protected]> * fix(supervisor): move established pool sessions when their host changes gateway A host whose stream ends, for example when its supervisor token expires, can reattach to another replica. Its sandbox sessions now watch for that and reconnect to the host's new replica instead of staying on the old one. Signed-off-by: divesh <[email protected]> * test(e2e): fold the supervisor tree checks into the shared-supervisor-ha lane Run the shared-supervisor-ha lane with two gateway replicas on PostgreSQL and move the gateway crash, rolling restart, and supervisor crash colocation test into it, replacing the separate shared-supervisor-tree lane. Signed-off-by: divesh <[email protected]> * fix(supervisor): redial the accepting gateway replica after a host stream ends A host's stream ends routinely, for example when its supervisor token expires. Redialing the replica that accepted it, before falling back to the Service, keeps the host and its sandbox sessions on that replica instead of moving them all on every reconnect. Signed-off-by: divesh <[email protected]> * fix(supervisor): follow a host move that lands during session setup An established session marked its host's current gateway as seen before watching it, so a move that landed while the session was connecting was never followed. Compare the current gateway first, then wait for changes. Signed-off-by: divesh <[email protected]> * fix(supervisor): dial the configured gateway at once when the host's replica has no peer endpoint Validate the advertised peer endpoint scheme with the shared supervisor, and document that pool sessions follow their supervisor's gateway. Signed-off-by: divesh <[email protected]> * test(e2e): harden the supervisor tree check's port-forward and time limit Signed-off-by: divesh <[email protected]> --------- Signed-off-by: divesh <[email protected]>
1 parent c3c6746 commit 8758cbf

20 files changed

Lines changed: 648 additions & 39 deletions

File tree

‎.config/nextest.toml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,11 @@ kubernetes-shared-supervisor-pool = { max-threads = 1 }
3232
filter = "test(/gateway_(scale_and_rollout|pod_rolls)$/)"
3333
test-group = "kubernetes-ha"
3434

35-
# Both change the shared supervisor pool of the release they run against: one
36-
# kills a member, the other scales the pool. Never run them at once. Test
37-
# names, not binary(), for the reason above.
35+
# These change the shared supervisor pool of the release they run against:
36+
# they kill members or gateways, or scale the pool. Never run them at once.
37+
# Test names, not binary(), for the reason above.
3838
[[profile.e2e-kubernetes.overrides]]
39-
filter = "test(/^(losing_a_supervisor_pod_moves_its_sandboxes_without_restarting_them|scaling_a_supervisor_pool_up_and_down_keeps_its_sandboxes_running)$/)"
39+
filter = "test(/^(losing_a_supervisor_pod_moves_its_sandboxes_without_restarting_them|scaling_a_supervisor_pool_up_and_down_keeps_its_sandboxes_running|pool_sessions_follow_their_supervisors_gateway)$/)"
4040
test-group = "kubernetes-shared-supervisor-pool"
4141

4242
# Scaling the pool runs two bounded Helm upgrades (up to 6 minutes each), waits
@@ -46,6 +46,11 @@ test-group = "kubernetes-shared-supervisor-pool"
4646
filter = "test(/^scaling_a_supervisor_pool_up_and_down_keeps_its_sandboxes_running$/)"
4747
slow-timeout = { period = "60s", terminate-after = 20 }
4848

49+
# Three bounded rollouts (up to 5 minutes each) and four settle and exec rounds.
50+
[[profile.e2e-kubernetes.overrides]]
51+
filter = "test(/^pool_sessions_follow_their_supervisors_gateway$/)"
52+
slow-timeout = { period = "60s", terminate-after = 20 }
53+
4954
# Relative to the profile store dir (`e2e/rust/target/nextest/e2e-kubernetes/`).
5055
[profile.e2e-kubernetes.junit]
5156
path = "../../../../../results/e2e-kubernetes.xml"

‎TESTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -724,7 +724,7 @@ Available task variants:
724724
| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
725725
| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |
726726
| `e2e:kubernetes:shared-supervisor` | One-member shared supervisor pool for the gateway's sandboxes (gateway TLS through the port-forward) |
727-
| `e2e:kubernetes:shared-supervisor-ha` | Two-replica shared supervisor pool: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted |
727+
| `e2e:kubernetes:shared-supervisor-ha` | Two-replica shared supervisor pool behind two gateway replicas on PostgreSQL: kills the member that owns a sandbox and checks the sandbox is adopted, not restarted; crashes and restarts gateways and checks each sandbox's session stays on its supervisor's gateway |
728728
| `e2e:kubernetes:shared-supervisor-scaling` | One-member shared supervisor pool: scales it to two members and back with `helm upgrade`, checking sandboxes stay running and the removed member hands its sandboxes off |
729729

730730
Kubernetes e2e environment variables:

‎crates/openshell-server/src/grpc/supervisor_host.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,16 @@ impl SupervisorHost for SupervisorHostService {
5959
Status::failed_precondition("this gateway does not host a shared supervisor")
6060
})?
6161
.clone();
62+
// A stopping replica sheds its hosts so they reattach to a live one
63+
// before their sandbox sessions follow them there.
64+
let mut shutdown = self.state.supervisor_sessions.subscribe_shutdown();
65+
if *shutdown.borrow() {
66+
return Err(Status::unavailable("gateway is shutting down"));
67+
}
6268
let outbound = controller.attach(identity, request.into_inner()).await?;
69+
let outbound = futures::StreamExt::take_until(outbound, async move {
70+
let _ = shutdown.wait_for(|shutdown| *shutdown).await;
71+
});
6372
Ok(Response::new(Box::pin(outbound)))
6473
}
6574
}

‎crates/openshell-server/src/lib.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -843,6 +843,11 @@ pub(crate) async fn run_server(
843843
"shared supervisor assignment stream enabled; minting supervisor tokens"
844844
);
845845
info!("shared supervisor pool enabled; reconciling pool membership and ownership");
846+
// Sandbox sessions dial this endpoint with their tokens.
847+
if let Some(peer_endpoint) = state.peer_endpoint.as_deref() {
848+
validate_peer_endpoint_scheme(&state.config, peer_endpoint)?;
849+
}
850+
controller.set_peer_endpoint(state.peer_endpoint.clone().unwrap_or_default());
846851
supervisor_pool::SupervisorPool::new(
847852
state.store.clone(),
848853
controller.clone(),

‎crates/openshell-server/src/supervisor_session.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -435,6 +435,11 @@ impl SupervisorSessionRegistry {
435435
self.admission_closed.store(true, Ordering::Release);
436436
}
437437

438+
/// Watch for [`Self::shutdown`] beginning.
439+
pub(crate) fn subscribe_shutdown(&self) -> watch::Receiver<bool> {
440+
self.shutdown.subscribe()
441+
}
442+
438443
/// Close control sessions and wait for tracked ownership cleanup. Call
439444
/// after compute shutdown so supervisors can finish normal stop reporting.
440445
pub(crate) async fn shutdown(&self, timeout: Duration) -> Result<(), String> {

‎crates/openshell-supervisor-control/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@ re-implementing the control contracts.
4646
stream attaches, ends, or announces leaving, so the gateway refreshes
4747
membership records at once. The gateway closes a member's stream after
4848
`pool_timing::GATEWAY_SILENCE_TIMEOUT` of silence, and drops a member that
49-
stops reading as soon.
49+
stops reading as soon. `HostAccepted` carries this replica's peer endpoint
50+
(`set_peer_endpoint`); the host dials its instances' sandbox sessions
51+
there, so each session runs on the replica serving its supervisor.
5052
- `pool_timing`: the timing contract between pool hosts and the gateway: the
5153
15 s membership TTL, the 2 s record refresh, 1 s heartbeats, host silence
5254
detection and bounded reconnect attempts, and `reconnect_budget`, the worst

‎crates/openshell-supervisor-control/src/controller.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -360,6 +360,8 @@ struct Inner {
360360
/// stream attached or ended, or a member announced leaving.
361361
membership: watch::Sender<u64>,
362362
observed: Mutex<Observed>,
363+
/// This replica's peer endpoint, sent in every `HostAccepted`.
364+
peer_endpoint: Mutex<String>,
363365
}
364366

365367
/// Gateway-owned desired assignments and observed status of the shared
@@ -410,10 +412,17 @@ impl SupervisorHostController {
410412
revision: watch::Sender::new(0),
411413
membership: watch::Sender::new(0),
412414
observed: Mutex::new(Observed::default()),
415+
peer_endpoint: Mutex::new(String::new()),
413416
}),
414417
}
415418
}
416419

420+
/// Tell attaching hosts to dial their sandbox sessions to `peer_endpoint`.
421+
/// Call it before any host attaches.
422+
pub fn set_peer_endpoint(&self, peer_endpoint: String) {
423+
*lock(&self.inner.peer_endpoint) = peer_endpoint;
424+
}
425+
417426
/// Offer every instance's runtime `recovery_timeout_secs` instead of
418427
/// [`POOL_RECOVERY_TIMEOUT_SECS`]. Call it before any host attaches.
419428
pub fn set_recovery_timeout_secs(&self, recovery_timeout_secs: u32) {
@@ -850,13 +859,15 @@ impl SupervisorHostController {
850859
let (tx, rx) = mpsc::channel(OUTBOUND_QUEUE);
851860
let expires_at = identity.expires_at;
852861
let (session_id, superseded) = self.begin_session(identity, &hello)?;
862+
let peer_endpoint = lock(&self.inner.peer_endpoint).clone();
853863
tx.send(Ok(HostGatewayMessage {
854864
payload: Some(host_gateway_message::Payload::Accepted(HostAccepted {
855865
session_id: session_id.to_string(),
856866
heartbeat_interval: Some(prost_types::Duration {
857867
seconds: i64::try_from(heartbeat.as_secs()).unwrap_or(15),
858868
nanos: 0,
859869
}),
870+
peer_endpoint,
860871
})),
861872
}))
862873
.await

‎crates/openshell-supervisor-process/src/delegated.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ pub async fn start_boundary_access(
9393
config_apply_tx: tokio::sync::mpsc::Sender<crate::supervisor_session::ConfigApplyRequest>,
9494
config_push_updates: tokio::sync::watch::Sender<bool>,
9595
host_key: Option<russh::keys::PrivateKey>,
96+
gateway_home: Option<tokio::sync::watch::Receiver<Option<String>>>,
9697
) -> Result<BoundaryAccess> {
9798
let instance_id = uuid::Uuid::new_v4().to_string();
9899
let terminating = Arc::new(AtomicBool::new(false));
@@ -130,6 +131,7 @@ pub async fn start_boundary_access(
130131
session_id_updates: supervisor_session_updates,
131132
config_apply_tx,
132133
config_push_updates,
134+
gateway_home,
133135
},
134136
);
135137
(Some(task), Some(accepted))
@@ -374,6 +376,7 @@ mod tests {
374376
tokio::sync::mpsc::channel(1).0,
375377
tokio::sync::watch::channel(false).0,
376378
None,
379+
None,
377380
)
378381
.await
379382
.expect("access without SSH is portable");
@@ -415,6 +418,7 @@ mod tests {
415418
tokio::sync::mpsc::channel(1).0,
416419
tokio::sync::watch::channel(false).0,
417420
None,
421+
None,
418422
)
419423
.await;
420424
assert!(

‎crates/openshell-supervisor-process/src/supervisor_session.rs‎

Lines changed: 155 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ use openshell_core::transport_errors::is_expected_transport_close_status;
4444

4545
const INITIAL_BACKOFF: Duration = Duration::from_secs(1);
4646
const MAX_BACKOFF: Duration = Duration::from_secs(30);
47+
/// How long a shared host's session waits for the host to attach to a
48+
/// gateway replica before dialing another address.
49+
const GATEWAY_HOME_WAIT: Duration = Duration::from_secs(5);
4750
const CONFIG_APPLY_TIMEOUT: Duration = Duration::from_mins(1);
4851

4952
/// A stream-delivered configuration update awaiting application by the
@@ -129,6 +132,9 @@ pub struct SessionRuntimeContext {
129132
/// Publishes whether the current session pushes configuration, so the
130133
/// runtime stops polling while it does.
131134
pub config_push_updates: watch::Sender<bool>,
135+
/// Peer endpoint of the gateway replica serving this supervisor's shared
136+
/// host. When set, sessions dial it instead of the configured address.
137+
pub gateway_home: Option<watch::Receiver<Option<String>>>,
132138
}
133139

134140
/// Parse a gRPC endpoint URI into an OCSF `Endpoint` (host + port). Falls back
@@ -423,6 +429,7 @@ pub fn spawn_with_readiness(
423429
session_id_updates: runtime.session_id_updates,
424430
config_apply_tx: runtime.config_apply_tx,
425431
config_push_updates: runtime.config_push_updates,
432+
gateway_home: runtime.gateway_home,
426433
ready_tx,
427434
};
428435
(tokio::spawn(run_session_loop(config)), ready_rx)
@@ -442,6 +449,7 @@ struct SessionConfig {
442449
/// Publishes whether the current session pushes configuration. A
443450
/// reconnect to a gateway that does not push resumes polling.
444451
config_push_updates: watch::Sender<bool>,
452+
gateway_home: Option<watch::Receiver<Option<String>>>,
445453
ready_tx: watch::Sender<bool>,
446454
}
447455

@@ -455,9 +463,22 @@ async fn run_session_loop(config: SessionConfig) {
455463
// back to the configured address, which load-balances across replicas.
456464
let mut target = config.gateway.clone();
457465
let mut redirected = false;
466+
// A shared host's sessions dial the replica serving the host, as
467+
// redirected so that replica serves them, unless they are following a
468+
// redirect away from it.
469+
let mut gateway_home = config.gateway_home.clone();
470+
let mut follow_home = true;
458471

459472
loop {
460473
attempt += 1;
474+
if follow_home
475+
&& let Some(home) = gateway_home.as_mut()
476+
&& let Some(endpoint) = wait_for_home(home, None).await
477+
{
478+
target = config.gateway.with_url(endpoint);
479+
redirected = true;
480+
}
481+
follow_home = true;
461482

462483
let result = run_single_session(&config, &target, redirected, attempt).await;
463484
if let Some(updates) = &config.session_id_updates {
@@ -485,10 +506,27 @@ async fn run_session_loop(config: SessionConfig) {
485506
peer_endpoint = %peer_endpoint,
486507
"supervisor session: following gateway redirect"
487508
);
488-
target = config.gateway.with_url(peer_endpoint);
509+
// Prefer the replica the shared host moved to.
510+
let home = match gateway_home.as_mut() {
511+
Some(home) => wait_for_home(home, Some(target.url())).await,
512+
None => None,
513+
};
514+
target = config.gateway.with_url(home.unwrap_or(peer_endpoint));
489515
redirected = true;
516+
follow_home = false;
490517
// No backoff: this is an expected handoff, not a failure.
491518
}
519+
Ok(SessionOutcome::HomeMoved { peer_endpoint }) => {
520+
backoff.on_redirect(config.ready_tx.send_replace(false));
521+
info!(
522+
sandbox_id = %config.sandbox_id,
523+
peer_endpoint = %peer_endpoint,
524+
"supervisor session: following the shared host to another gateway"
525+
);
526+
target = config.gateway.with_url(peer_endpoint);
527+
redirected = true;
528+
follow_home = false;
529+
}
492530
Err(e) => {
493531
let accepted = config.ready_tx.send_replace(false);
494532
let event = session_failed_event(
@@ -555,6 +593,44 @@ impl ReconnectBackoff {
555593
}
556594
}
557595

596+
/// The peer endpoint of the replica serving the shared host, once it is one
597+
/// other than `avoid`, waiting up to [`GATEWAY_HOME_WAIT`] while no replica
598+
/// serves the host. `None` at once when the serving replica has no endpoint.
599+
async fn wait_for_home(
600+
home: &mut watch::Receiver<Option<String>>,
601+
avoid: Option<&str>,
602+
) -> Option<String> {
603+
tokio::time::timeout(
604+
GATEWAY_HOME_WAIT,
605+
home.wait_for(|home| home.as_deref().is_some_and(|home| Some(home) != avoid)),
606+
)
607+
.await
608+
.ok()?
609+
.ok()
610+
.and_then(|home| (*home).clone())
611+
.filter(|home| !home.is_empty())
612+
}
613+
614+
/// The peer endpoint of the shared host, once it is one other than `current`,
615+
/// including a move that landed before this is first polled. Pending forever
616+
/// without a shared host.
617+
async fn home_moved(home: Option<&mut watch::Receiver<Option<String>>>, current: &str) -> String {
618+
let Some(home) = home else {
619+
return std::future::pending().await;
620+
};
621+
loop {
622+
let endpoint = home.borrow_and_update().clone();
623+
if let Some(endpoint) =
624+
endpoint.filter(|endpoint| !endpoint.is_empty() && endpoint != current)
625+
{
626+
return endpoint;
627+
}
628+
if home.changed().await.is_err() {
629+
return std::future::pending().await;
630+
}
631+
}
632+
}
633+
558634
/// Keep asking not to be redirected when a redirected attempt never got a
559635
/// session, so a stale ring pointing at a dead replica cannot bounce us back
560636
/// to it. A session that was accepted and later dropped starts placement over.
@@ -579,6 +655,9 @@ enum SessionOutcome {
579655
peer_endpoint: String,
580656
owner_replica_id: String,
581657
},
658+
/// The shared host reattached to another replica. The caller reconnects
659+
/// there.
660+
HomeMoved { peer_endpoint: String },
582661
}
583662

584663
async fn run_single_session(
@@ -693,9 +772,13 @@ async fn run_single_session(
693772
// Main loop: receive gateway messages + send heartbeats.
694773
let mut heartbeat_interval = tokio::time::interval(Duration::from_secs(heartbeat_secs));
695774
heartbeat_interval.tick().await; // skip immediate tick
775+
let mut home = config.gateway_home.clone();
696776

697777
loop {
698778
tokio::select! {
779+
peer_endpoint = home_moved(home.as_mut(), target.url()) => {
780+
return Ok(SessionOutcome::HomeMoved { peer_endpoint });
781+
}
699782
msg = inbound.message() => {
700783
let msg = match map_session_stream_message(
701784
msg,
@@ -1636,6 +1719,77 @@ mod ocsf_event_tests {
16361719
assert!(!redirect_survives_failure(false, true));
16371720
}
16381721

1722+
#[tokio::test(start_paused = true)]
1723+
async fn shared_host_sessions_wait_for_the_hosts_gateway() {
1724+
let (tx, mut home) = watch::channel(None);
1725+
assert_eq!(wait_for_home(&mut home, None).await, None);
1726+
1727+
tx.send_replace(Some("https://gw-a:8443".to_string()));
1728+
assert_eq!(
1729+
wait_for_home(&mut home, None).await.as_deref(),
1730+
Some("https://gw-a:8443")
1731+
);
1732+
assert_eq!(
1733+
wait_for_home(&mut home, Some("https://gw-a:8443")).await,
1734+
None
1735+
);
1736+
1737+
let mut moving = home.clone();
1738+
let moved =
1739+
tokio::spawn(
1740+
async move { wait_for_home(&mut moving, Some("https://gw-a:8443")).await },
1741+
);
1742+
tokio::task::yield_now().await;
1743+
tx.send_replace(Some("https://gw-b:8443".to_string()));
1744+
assert_eq!(moved.await.unwrap().as_deref(), Some("https://gw-b:8443"));
1745+
}
1746+
1747+
#[tokio::test]
1748+
async fn a_host_gateway_without_a_peer_endpoint_is_not_waited_for() {
1749+
let (_tx, mut home) = watch::channel(Some(String::new()));
1750+
let started = tokio::time::Instant::now();
1751+
assert_eq!(wait_for_home(&mut home, None).await, None);
1752+
assert!(started.elapsed() < GATEWAY_HOME_WAIT);
1753+
1754+
let moved = tokio::time::timeout(
1755+
Duration::from_millis(50),
1756+
home_moved(Some(&mut home), "https://gw-a:8443"),
1757+
)
1758+
.await;
1759+
assert!(moved.is_err(), "an endpointless gateway is not a move");
1760+
}
1761+
1762+
#[tokio::test]
1763+
async fn a_host_move_during_session_setup_is_followed() {
1764+
let (tx, mut home) = watch::channel(Some("https://gw-a:8443".to_string()));
1765+
// The session dialed A; the host moved to B before it was accepted,
1766+
// and the move was already observed.
1767+
tx.send_replace(Some("https://gw-b:8443".to_string()));
1768+
home.mark_unchanged();
1769+
assert_eq!(
1770+
home_moved(Some(&mut home), "https://gw-a:8443").await,
1771+
"https://gw-b:8443"
1772+
);
1773+
}
1774+
1775+
#[tokio::test]
1776+
async fn an_established_session_follows_its_host_to_another_gateway() {
1777+
let (tx, mut home) = watch::channel(Some("https://gw-a:8443".to_string()));
1778+
let moved =
1779+
tokio::spawn(async move { home_moved(Some(&mut home), "https://gw-a:8443").await });
1780+
1781+
tx.send_replace(None);
1782+
tx.send_replace(Some("https://gw-a:8443".to_string()));
1783+
tokio::task::yield_now().await;
1784+
assert!(
1785+
!moved.is_finished(),
1786+
"a reattach to the same gateway is not a move"
1787+
);
1788+
1789+
tx.send_replace(Some("https://gw-b:8443".to_string()));
1790+
assert_eq!(moved.await.unwrap(), "https://gw-b:8443");
1791+
}
1792+
16391793
#[test]
16401794
fn reconnect_backoff_doubles_until_a_session_is_accepted() {
16411795
let mut backoff = ReconnectBackoff::new();

0 commit comments

Comments
 (0)