Skip to content

Commit 761589e

Browse files
committed
refactor(compute): make sandbox readiness gateway-owned across all drivers
Introduce compute_phase_components and apply_readiness_conditions to centralise the gateway's phase composition logic. The public SandboxPhase is now determined by combining the backend phase reported by the driver with supervisor session presence, independent of the driver implementation. Remove SupervisorReadiness from the driver contract. Running containers always report BackendReady; the gateway owns the Ready decision. Rename the dispatcher match to three arms so that status-less events for existing sandboxes are a documented no-op rather than a silent pass-through. Drop backend_ready_no_session and the SupervisorNotConnected condition. The BackendReady driver condition plus the Provisioning phase already communicates that the backend is up but the supervisor has not connected. The redundant condition added noise without new information. Closes #1951 Signed-off-by: Evan Lezar <[email protected]>
1 parent 716ea50 commit 761589e

5 files changed

Lines changed: 477 additions & 134 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,60 @@ Each runtime receives a sandbox spec from the gateway and is responsible for:
1616
- Reporting lifecycle and platform events back to the gateway.
1717
- Cleaning up runtime-owned resources.
1818

19+
Drivers report **backend state only**. A driver snapshot with `Ready=True` means
20+
the underlying compute resource (container, pod, VM) is healthy and running —
21+
nothing more. Drivers must not gate on supervisor session state or hold
22+
references to gateway-internal types. The gateway owns the public
23+
`SandboxPhase::Ready` decision. This applies equally to extension drivers
24+
implementing `ComputeDriver` out of tree.
25+
1926
Drivers own runtime-specific platform event interpretation. When an event should
2027
drive client provisioning UI, the driver attaches the shared
2128
`openshell.progress.*` metadata defined in `openshell-core` instead of requiring
2229
clients to parse Kubernetes reasons, VM cache states, or other driver-local
2330
reason strings.
2431

32+
## Sandbox Readiness Composition
33+
34+
The gateway composes driver backend state with supervisor session presence to
35+
produce the public `SandboxPhase`. This composition is gateway-owned and applied
36+
uniformly across all drivers:
37+
38+
```
39+
backend_phase = derive_phase(driver_status)
40+
41+
public_phase =
42+
if backend_phase in {Error, Deleting}: → pass through (terminal precedence)
43+
if backend_phase == Ready && session connected: → Ready
44+
if backend_phase == Ready && no session: → Provisioning
45+
if backend_phase in {Provisioning, Unknown} && session: → Ready
46+
if backend_phase in {Provisioning, Unknown} && no session: → Provisioning
47+
```
48+
49+
When `public_phase == Ready` the sandbox is usable through the gateway — both the
50+
backend resource is healthy and a supervisor session is registered. A sandbox whose
51+
backend reports ready but has no supervisor session yet holds `Provisioning`; the
52+
driver's `BackendReady=True` condition is visible in the sandbox status for operators
53+
who need to distinguish that state from a sandbox still provisioning its compute resource.
54+
55+
**Session precedence over lagging driver snapshots:** A supervisor session can only be
56+
established by a running workload. When `set_supervisor_session_state` promotes the
57+
store record to `Ready` on session connect, a driver watch event may still arrive
58+
shortly after carrying a stale `Provisioning` or `Unknown` backend phase. The
59+
composition rule treats a connected session as the stronger signal and keeps `Ready`
60+
in that case, preventing a lagging snapshot from undoing the session-driven promotion.
61+
62+
**HA deployments:** Supervisor sessions are process-local. A gateway replica that
63+
does not own the active supervisor session holds the public phase at `Provisioning`.
64+
The owning replica's `supervisor_session_connected` write propagates through the
65+
shared store and reconcile loop. This is correct behavior — a replica should not
66+
claim `Ready` for a session it does not hold.
67+
68+
**Extension point:** The readiness decision is a safety invariant, not an
69+
operator-configurable hook. The driver contract is the correct extension point for
70+
custom backend readiness semantics. RFC-0010 lifecycle hooks may observe readiness
71+
transitions via `post_commit`; they do not override the composition rule.
72+
2573
The capability RPC reports driver identity, version, and the default sandbox
2674
image used by the gateway. GPU availability stays driver-local and is validated
2775
when a sandbox create request asks for GPU resources.

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 56 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -78,15 +78,53 @@ const HOST_OPENSHELL_INTERNAL: &str = "host.openshell.internal";
7878
const HOST_DOCKER_INTERNAL: &str = "host.docker.internal";
7979
const DOCKER_NETWORK_DRIVER: &str = "bridge";
8080

81-
/// Queried by the Docker driver to decide when a sandbox's supervisor
82-
/// relay is live. Implementations return `true` once a sandbox has an
83-
/// active `ConnectSupervisor` session registered.
81+
/// Default image holding the Linux `openshell-sandbox` binary. The gateway
82+
/// pulls this image and extracts the binary to a host-side cache when no
83+
/// explicit `supervisor_bin`, configured `supervisor_image`, sibling binary,
84+
/// or local build is available.
85+
const DEFAULT_DOCKER_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor";
86+
87+
/// Return the default `ghcr.io/nvidia/openshell/supervisor:<tag>` reference
88+
/// used when no supervisor binary override is provided.
89+
pub fn default_docker_supervisor_image() -> String {
90+
format!(
91+
"{DEFAULT_DOCKER_SUPERVISOR_IMAGE_REPO}:{}",
92+
default_docker_supervisor_image_tag()
93+
)
94+
}
95+
96+
/// Image tag baked in at compile time to pair the gateway with a matching
97+
/// supervisor image.
8498
///
85-
/// The driver cannot observe the supervisor's SSH socket directly (it
86-
/// lives inside the container), so it leans on this signal to flip the
87-
/// Ready condition from `DependenciesNotReady` to `True`.
88-
pub trait SupervisorReadiness: Send + Sync + 'static {
89-
fn is_supervisor_connected(&self, sandbox_id: &str) -> bool;
99+
/// Build pipelines pass `OPENSHELL_IMAGE_TAG` explicitly. The `IMAGE_TAG`
100+
/// fallback covers image build wrappers that already tag the gateway and
101+
/// supervisor together. Standalone release binaries also patch the Cargo
102+
/// package version, so use it when it has been set to a real release value.
103+
fn default_docker_supervisor_image_tag() -> String {
104+
resolve_default_docker_supervisor_image_tag(
105+
option_env!("OPENSHELL_IMAGE_TAG"),
106+
option_env!("IMAGE_TAG"),
107+
env!("CARGO_PKG_VERSION"),
108+
)
109+
}
110+
111+
fn resolve_default_docker_supervisor_image_tag(
112+
openshell_image_tag: Option<&'static str>,
113+
image_tag: Option<&'static str>,
114+
cargo_pkg_version: &'static str,
115+
) -> String {
116+
let tag = openshell_image_tag
117+
.filter(|tag| !tag.is_empty())
118+
.or_else(|| image_tag.filter(|tag| !tag.is_empty()))
119+
.unwrap_or_else(|| {
120+
if cargo_pkg_version.is_empty() || cargo_pkg_version == "0.0.0" {
121+
"dev"
122+
} else {
123+
cargo_pkg_version
124+
}
125+
});
126+
127+
tag.replace('+', "-")
90128
}
91129

92130
/// Gateway-local configuration for the Docker compute driver.
@@ -211,7 +249,6 @@ pub struct DockerComputeDriver {
211249
config: DockerDriverRuntimeConfig,
212250
events: broadcast::Sender<WatchSandboxesEvent>,
213251
pending: Arc<Mutex<HashMap<String, PendingSandboxRecord>>>,
214-
supervisor_readiness: Arc<dyn SupervisorReadiness>,
215252
gpu_selector: Arc<CdiGpuDefaultSelector>,
216253
}
217254

@@ -308,11 +345,7 @@ type WatchStream =
308345
Pin<Box<dyn Stream<Item = Result<WatchSandboxesEvent, Status>> + Send + 'static>>;
309346

310347
impl DockerComputeDriver {
311-
pub async fn new(
312-
config: &Config,
313-
docker_config: &DockerComputeConfig,
314-
supervisor_readiness: Arc<dyn SupervisorReadiness>,
315-
) -> CoreResult<Self> {
348+
pub async fn new(config: &Config, docker_config: &DockerComputeConfig) -> CoreResult<Self> {
316349
let socket_path = docker_config
317350
.socket_path
318351
.clone()
@@ -394,7 +427,6 @@ impl DockerComputeDriver {
394427
},
395428
events: broadcast::channel(WATCH_BUFFER).0,
396429
pending: Arc::new(Mutex::new(HashMap::new())),
397-
supervisor_readiness,
398430
gpu_selector: Arc::new(CdiGpuDefaultSelector::new(
399431
cdi_gpu_inventory,
400432
allow_all_default_gpu,
@@ -605,9 +637,9 @@ impl DockerComputeDriver {
605637
let container = self
606638
.find_managed_container_summary(sandbox_id, sandbox_name)
607639
.await?;
608-
if let Some(sandbox) = container.and_then(|summary| {
609-
sandbox_from_container_summary(&summary, self.supervisor_readiness.as_ref())
610-
}) {
640+
if let Some(sandbox) =
641+
container.and_then(|summary| sandbox_from_container_summary(&summary))
642+
{
611643
return Ok(Some(sandbox));
612644
}
613645

@@ -618,9 +650,7 @@ impl DockerComputeDriver {
618650
let containers = self.list_managed_container_summaries().await?;
619651
let container_sandboxes = containers
620652
.iter()
621-
.filter_map(|summary| {
622-
sandbox_from_container_summary(summary, self.supervisor_readiness.as_ref())
623-
})
653+
.filter_map(sandbox_from_container_summary)
624654
.collect::<Vec<_>>();
625655
let mut by_id = self.pending_snapshot_map().await;
626656
for sandbox in container_sandboxes {
@@ -1122,8 +1152,7 @@ impl DockerComputeDriver {
11221152
if let Some(summary) = self
11231153
.find_managed_container_summary(sandbox_id, sandbox_name)
11241154
.await?
1125-
&& let Some(sandbox) =
1126-
sandbox_from_container_summary(&summary, self.supervisor_readiness.as_ref())
1155+
&& let Some(sandbox) = sandbox_from_container_summary(&summary)
11271156
{
11281157
self.publish_sandbox_snapshot(sandbox);
11291158
}
@@ -2732,10 +2761,7 @@ fn parse_memory_limit(value: &str) -> Result<Option<i64>, Status> {
27322761
Ok(Some((amount * multiplier).round() as i64))
27332762
}
27342763

2735-
fn sandbox_from_container_summary(
2736-
summary: &ContainerSummary,
2737-
readiness: &dyn SupervisorReadiness,
2738-
) -> Option<DriverSandbox> {
2764+
fn sandbox_from_container_summary(summary: &ContainerSummary) -> Option<DriverSandbox> {
27392765
let labels = summary.labels.as_ref()?;
27402766
let id = labels.get(LABEL_SANDBOX_ID)?.clone();
27412767
let name = labels.get(LABEL_SANDBOX_NAME)?.clone();
@@ -2744,27 +2770,21 @@ fn sandbox_from_container_summary(
27442770
.cloned()
27452771
.unwrap_or_default();
27462772

2747-
let supervisor_connected = readiness.is_supervisor_connected(&id);
27482773
Some(DriverSandbox {
27492774
id,
27502775
name: name.clone(),
27512776
namespace,
27522777
spec: None,
2753-
status: Some(driver_status_from_summary(
2754-
summary,
2755-
&name,
2756-
supervisor_connected,
2757-
)),
2778+
status: Some(driver_status_from_summary(summary, &name)),
27582779
})
27592780
}
27602781

27612782
fn driver_status_from_summary(
27622783
summary: &ContainerSummary,
27632784
sandbox_name: &str,
2764-
supervisor_connected: bool,
27652785
) -> DriverSandboxStatus {
27662786
let state = summary.state.unwrap_or(ContainerSummaryStateEnum::EMPTY);
2767-
let (ready, reason, message, deleting) = container_ready_condition(state, supervisor_connected);
2787+
let (ready, reason, message, deleting) = container_ready_condition(state);
27682788

27692789
DriverSandboxStatus {
27702790
sandbox_name: summary_container_name(summary).unwrap_or_else(|| sandbox_name.to_string()),
@@ -2784,25 +2804,10 @@ fn driver_status_from_summary(
27842804

27852805
fn container_ready_condition(
27862806
state: ContainerSummaryStateEnum,
2787-
supervisor_connected: bool,
27882807
) -> (&'static str, &'static str, &'static str, bool) {
27892808
match state {
27902809
ContainerSummaryStateEnum::RUNNING => {
2791-
if supervisor_connected {
2792-
(
2793-
"True",
2794-
"SupervisorConnected",
2795-
"Supervisor relay is live",
2796-
false,
2797-
)
2798-
} else {
2799-
(
2800-
"False",
2801-
"DependenciesNotReady",
2802-
"Container is running; waiting for supervisor relay",
2803-
false,
2804-
)
2805-
}
2810+
("True", "BackendReady", "Container is running", false)
28062811
}
28072812
ContainerSummaryStateEnum::CREATED => ("False", "Starting", "Container created", false),
28082813
ContainerSummaryStateEnum::RESTARTING => (

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 9 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -140,14 +140,6 @@ fn inspected_volume(driver: &str, options: HashMap<String, String>) -> bollard::
140140
}
141141
}
142142

143-
struct DisconnectedSupervisorReadiness;
144-
145-
impl SupervisorReadiness for DisconnectedSupervisorReadiness {
146-
fn is_supervisor_connected(&self, _sandbox_id: &str) -> bool {
147-
false
148-
}
149-
}
150-
151143
fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDriver {
152144
let allow_all_default_gpu = config.allow_all_default_gpu;
153145
DockerComputeDriver {
@@ -158,7 +150,6 @@ fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDr
158150
config,
159151
events: broadcast::channel(WATCH_BUFFER).0,
160152
pending: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
161-
supervisor_readiness: Arc::new(DisconnectedSupervisorReadiness),
162153
gpu_selector: Arc::new(CdiGpuDefaultSelector::new(
163154
CdiGpuInventory::default(),
164155
allow_all_default_gpu,
@@ -1727,34 +1718,19 @@ fn driver_status_keeps_running_sandboxes_provisioning_with_stable_message() {
17271718
..running.clone()
17281719
};
17291720

1730-
let running_status = driver_status_from_summary(&running, "demo", false);
1731-
let running_later_status = driver_status_from_summary(&running_later, "demo", false);
1732-
assert_eq!(running_status.conditions[0].status, "False");
1733-
assert_eq!(running_status.conditions[0].reason, "DependenciesNotReady");
1734-
assert_eq!(
1735-
running_status.conditions[0].message,
1736-
"Container is running; waiting for supervisor relay"
1737-
);
1721+
// A running container always emits Ready=True with BackendReady. The gateway
1722+
// composes this with supervisor-session presence to decide public SandboxPhase.
1723+
let running_status = driver_status_from_summary(&running, "demo");
1724+
let running_later_status = driver_status_from_summary(&running_later, "demo");
1725+
assert_eq!(running_status.conditions[0].status, "True");
1726+
assert_eq!(running_status.conditions[0].reason, "BackendReady");
1727+
assert_eq!(running_status.conditions[0].message, "Container is running");
17381728
assert_eq!(running_status.conditions, running_later_status.conditions);
17391729

1740-
let exited_status = driver_status_from_summary(&exited, "demo", false);
1730+
let exited_status = driver_status_from_summary(&exited, "demo");
17411731
assert_eq!(exited_status.conditions[0].status, "False");
17421732
assert_eq!(exited_status.conditions[0].reason, "ContainerExited");
17431733
assert_eq!(exited_status.conditions[0].message, "Container exited");
1744-
1745-
// With a live supervisor session, a RUNNING container flips Ready=True
1746-
// so ExecSandbox and other "sandbox must be ready" gates can proceed.
1747-
let running_connected = driver_status_from_summary(&running, "demo", true);
1748-
assert_eq!(running_connected.conditions[0].status, "True");
1749-
assert_eq!(
1750-
running_connected.conditions[0].reason,
1751-
"SupervisorConnected"
1752-
);
1753-
1754-
// Supervisor readiness is ignored for non-RUNNING states -- an exited
1755-
// container must not report Ready=True.
1756-
let exited_connected = driver_status_from_summary(&exited, "demo", true);
1757-
assert_eq!(exited_connected.conditions[0].status, "False");
17581734
}
17591735

17601736
#[test]
@@ -1772,7 +1748,7 @@ fn driver_status_marks_restarting_sandboxes_as_error() {
17721748
..Default::default()
17731749
};
17741750

1775-
let status = driver_status_from_summary(&restarting, "demo", false);
1751+
let status = driver_status_from_summary(&restarting, "demo");
17761752
assert_eq!(status.conditions[0].status, "False");
17771753
assert_eq!(status.conditions[0].reason, "ContainerRestarting");
17781754
assert_eq!(

0 commit comments

Comments
 (0)