Skip to content

Commit c5aa55a

Browse files
committed
refactor(compute): make sandbox readiness gateway-owned across all drivers
Introduce compute_phase_components and apply_readiness_conditions to centralise the gateway's phase composition logic. The public SandboxPhase is now determined by combining the backend phase reported by the driver with supervisor session presence, independent of the driver implementation. Remove SupervisorReadiness from the driver contract. Running containers always report BackendReady; the gateway owns the Ready decision. Rename the dispatcher match to three arms so that status-less events for existing sandboxes are a documented no-op rather than a silent pass-through. Drop backend_ready_no_session and the SupervisorNotConnected condition. The BackendReady driver condition plus the Provisioning phase already communicates that the backend is up but the supervisor has not connected. The redundant condition added noise without new information. Closes #1951 Signed-off-by: Evan Lezar <[email protected]>
1 parent 81ad1ca commit c5aa55a

5 files changed

Lines changed: 473 additions & 115 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,60 @@ Each runtime receives a sandbox spec from the gateway and is responsible for:
1616
- Reporting lifecycle and platform events back to the gateway.
1717
- Cleaning up runtime-owned resources.
1818

19+
Drivers report **backend state only**. A driver snapshot with `Ready=True` means
20+
the underlying compute resource (container, pod, VM) is healthy and running —
21+
nothing more. Drivers must not gate on supervisor session state or hold
22+
references to gateway-internal types. The gateway owns the public
23+
`SandboxPhase::Ready` decision. This applies equally to extension drivers
24+
implementing `ComputeDriver` out of tree.
25+
1926
Drivers own runtime-specific platform event interpretation. When an event should
2027
drive client provisioning UI, the driver attaches the shared
2128
`openshell.progress.*` metadata defined in `openshell-core` instead of requiring
2229
clients to parse Kubernetes reasons, VM cache states, or other driver-local
2330
reason strings.
2431

32+
## Sandbox Readiness Composition
33+
34+
The gateway composes driver backend state with supervisor session presence to
35+
produce the public `SandboxPhase`. This composition is gateway-owned and applied
36+
uniformly across all drivers:
37+
38+
```
39+
backend_phase = derive_phase(driver_status)
40+
41+
public_phase =
42+
if backend_phase in {Error, Deleting}: → pass through (terminal precedence)
43+
if backend_phase == Ready && session connected: → Ready
44+
if backend_phase == Ready && no session: → Provisioning
45+
if backend_phase in {Provisioning, Unknown} && session: → Ready
46+
if backend_phase in {Provisioning, Unknown} && no session: → Provisioning
47+
```
48+
49+
When `public_phase == Ready` the sandbox is usable through the gateway — both the
50+
backend resource is healthy and a supervisor session is registered. A sandbox whose
51+
backend reports ready but has no supervisor session yet holds `Provisioning`; the
52+
driver's `BackendReady=True` condition is visible in the sandbox status for operators
53+
who need to distinguish that state from a sandbox still provisioning its compute resource.
54+
55+
**Session precedence over lagging driver snapshots:** A supervisor session can only be
56+
established by a running workload. When `set_supervisor_session_state` promotes the
57+
store record to `Ready` on session connect, a driver watch event may still arrive
58+
shortly after carrying a stale `Provisioning` or `Unknown` backend phase. The
59+
composition rule treats a connected session as the stronger signal and keeps `Ready`
60+
in that case, preventing a lagging snapshot from undoing the session-driven promotion.
61+
62+
**HA deployments:** Supervisor sessions are process-local. A gateway replica that
63+
does not own the active supervisor session holds the public phase at `Provisioning`.
64+
The owning replica's `supervisor_session_connected` write propagates through the
65+
shared store and reconcile loop. This is correct behavior — a replica should not
66+
claim `Ready` for a session it does not hold.
67+
68+
**Extension point:** The readiness decision is a safety invariant, not an
69+
operator-configurable hook. The driver contract is the correct extension point for
70+
custom backend readiness semantics. RFC-0010 lifecycle hooks may observe readiness
71+
transitions via `post_commit`; they do not override the composition rule.
72+
2573
The capability RPC reports driver identity, version, and the default sandbox
2674
image used by the gateway. GPU availability stays driver-local and is validated
2775
when a sandbox create request asks for GPU resources.

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 56 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -79,15 +79,53 @@ const HOST_OPENSHELL_INTERNAL: &str = "host.openshell.internal";
7979
const HOST_DOCKER_INTERNAL: &str = "host.docker.internal";
8080
const DOCKER_NETWORK_DRIVER: &str = "bridge";
8181

82-
/// Queried by the Docker driver to decide when a sandbox's supervisor
83-
/// relay is live. Implementations return `true` once a sandbox has an
84-
/// active `ConnectSupervisor` session registered.
82+
/// Default image holding the Linux `openshell-sandbox` binary. The gateway
83+
/// pulls this image and extracts the binary to a host-side cache when no
84+
/// explicit `supervisor_bin`, configured `supervisor_image`, sibling binary,
85+
/// or local build is available.
86+
const DEFAULT_DOCKER_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor";
87+
88+
/// Return the default `ghcr.io/nvidia/openshell/supervisor:<tag>` reference
89+
/// used when no supervisor binary override is provided.
90+
pub fn default_docker_supervisor_image() -> String {
91+
format!(
92+
"{DEFAULT_DOCKER_SUPERVISOR_IMAGE_REPO}:{}",
93+
default_docker_supervisor_image_tag()
94+
)
95+
}
96+
97+
/// Image tag baked in at compile time to pair the gateway with a matching
98+
/// supervisor image.
8599
///
86-
/// The driver cannot observe the supervisor's SSH socket directly (it
87-
/// lives inside the container), so it leans on this signal to flip the
88-
/// Ready condition from `DependenciesNotReady` to `True`.
89-
pub trait SupervisorReadiness: Send + Sync + 'static {
90-
fn is_supervisor_connected(&self, sandbox_id: &str) -> bool;
100+
/// Build pipelines pass `OPENSHELL_IMAGE_TAG` explicitly. The `IMAGE_TAG`
101+
/// fallback covers image build wrappers that already tag the gateway and
102+
/// supervisor together. Standalone release binaries also patch the Cargo
103+
/// package version, so use it when it has been set to a real release value.
104+
fn default_docker_supervisor_image_tag() -> String {
105+
resolve_default_docker_supervisor_image_tag(
106+
option_env!("OPENSHELL_IMAGE_TAG"),
107+
option_env!("IMAGE_TAG"),
108+
env!("CARGO_PKG_VERSION"),
109+
)
110+
}
111+
112+
fn resolve_default_docker_supervisor_image_tag(
113+
openshell_image_tag: Option<&'static str>,
114+
image_tag: Option<&'static str>,
115+
cargo_pkg_version: &'static str,
116+
) -> String {
117+
let tag = openshell_image_tag
118+
.filter(|tag| !tag.is_empty())
119+
.or_else(|| image_tag.filter(|tag| !tag.is_empty()))
120+
.unwrap_or_else(|| {
121+
if cargo_pkg_version.is_empty() || cargo_pkg_version == "0.0.0" {
122+
"dev"
123+
} else {
124+
cargo_pkg_version
125+
}
126+
});
127+
128+
tag.replace('+', "-")
91129
}
92130

93131
/// Gateway-local configuration for the Docker compute driver.
@@ -212,7 +250,6 @@ pub struct DockerComputeDriver {
212250
config: DockerDriverRuntimeConfig,
213251
events: broadcast::Sender<WatchSandboxesEvent>,
214252
pending: Arc<Mutex<HashMap<String, PendingSandboxRecord>>>,
215-
supervisor_readiness: Arc<dyn SupervisorReadiness>,
216253
gpu_selector: Arc<CdiGpuDefaultSelector>,
217254
}
218255

@@ -309,11 +346,7 @@ type WatchStream =
309346
Pin<Box<dyn Stream<Item = Result<WatchSandboxesEvent, Status>> + Send + 'static>>;
310347

311348
impl DockerComputeDriver {
312-
pub async fn new(
313-
config: &Config,
314-
docker_config: &DockerComputeConfig,
315-
supervisor_readiness: Arc<dyn SupervisorReadiness>,
316-
) -> CoreResult<Self> {
349+
pub async fn new(config: &Config, docker_config: &DockerComputeConfig) -> CoreResult<Self> {
317350
let socket_path = docker_config
318351
.socket_path
319352
.clone()
@@ -395,7 +428,6 @@ impl DockerComputeDriver {
395428
},
396429
events: broadcast::channel(WATCH_BUFFER).0,
397430
pending: Arc::new(Mutex::new(HashMap::new())),
398-
supervisor_readiness,
399431
gpu_selector: Arc::new(CdiGpuDefaultSelector::new(
400432
cdi_gpu_inventory,
401433
allow_all_default_gpu,
@@ -606,9 +638,9 @@ impl DockerComputeDriver {
606638
let container = self
607639
.find_managed_container_summary(sandbox_id, sandbox_name)
608640
.await?;
609-
if let Some(sandbox) = container.and_then(|summary| {
610-
sandbox_from_container_summary(&summary, self.supervisor_readiness.as_ref())
611-
}) {
641+
if let Some(sandbox) =
642+
container.and_then(|summary| sandbox_from_container_summary(&summary))
643+
{
612644
return Ok(Some(sandbox));
613645
}
614646

@@ -619,9 +651,7 @@ impl DockerComputeDriver {
619651
let containers = self.list_managed_container_summaries().await?;
620652
let container_sandboxes = containers
621653
.iter()
622-
.filter_map(|summary| {
623-
sandbox_from_container_summary(summary, self.supervisor_readiness.as_ref())
624-
})
654+
.filter_map(sandbox_from_container_summary)
625655
.collect::<Vec<_>>();
626656
let mut by_id = self.pending_snapshot_map().await;
627657
for sandbox in container_sandboxes {
@@ -1123,8 +1153,7 @@ impl DockerComputeDriver {
11231153
if let Some(summary) = self
11241154
.find_managed_container_summary(sandbox_id, sandbox_name)
11251155
.await?
1126-
&& let Some(sandbox) =
1127-
sandbox_from_container_summary(&summary, self.supervisor_readiness.as_ref())
1156+
&& let Some(sandbox) = sandbox_from_container_summary(&summary)
11281157
{
11291158
self.publish_sandbox_snapshot(sandbox);
11301159
}
@@ -2738,10 +2767,7 @@ fn parse_memory_limit(value: &str) -> Result<Option<i64>, Status> {
27382767
Ok(Some((amount * multiplier).round() as i64))
27392768
}
27402769

2741-
fn sandbox_from_container_summary(
2742-
summary: &ContainerSummary,
2743-
readiness: &dyn SupervisorReadiness,
2744-
) -> Option<DriverSandbox> {
2770+
fn sandbox_from_container_summary(summary: &ContainerSummary) -> Option<DriverSandbox> {
27452771
let labels = summary.labels.as_ref()?;
27462772
let id = labels.get(LABEL_SANDBOX_ID)?.clone();
27472773
let name = labels.get(LABEL_SANDBOX_NAME)?.clone();
@@ -2754,28 +2780,22 @@ fn sandbox_from_container_summary(
27542780
.cloned()
27552781
.unwrap_or_default();
27562782

2757-
let supervisor_connected = readiness.is_supervisor_connected(&id);
27582783
Some(DriverSandbox {
27592784
id,
27602785
name: name.clone(),
27612786
namespace,
27622787
spec: None,
2763-
status: Some(driver_status_from_summary(
2764-
summary,
2765-
&name,
2766-
supervisor_connected,
2767-
)),
2788+
status: Some(driver_status_from_summary(summary, &name)),
27682789
workspace,
27692790
})
27702791
}
27712792

27722793
fn driver_status_from_summary(
27732794
summary: &ContainerSummary,
27742795
sandbox_name: &str,
2775-
supervisor_connected: bool,
27762796
) -> DriverSandboxStatus {
27772797
let state = summary.state.unwrap_or(ContainerSummaryStateEnum::EMPTY);
2778-
let (ready, reason, message, deleting) = container_ready_condition(state, supervisor_connected);
2798+
let (ready, reason, message, deleting) = container_ready_condition(state);
27792799

27802800
DriverSandboxStatus {
27812801
sandbox_name: summary_container_name(summary).unwrap_or_else(|| sandbox_name.to_string()),
@@ -2795,25 +2815,10 @@ fn driver_status_from_summary(
27952815

27962816
fn container_ready_condition(
27972817
state: ContainerSummaryStateEnum,
2798-
supervisor_connected: bool,
27992818
) -> (&'static str, &'static str, &'static str, bool) {
28002819
match state {
28012820
ContainerSummaryStateEnum::RUNNING => {
2802-
if supervisor_connected {
2803-
(
2804-
"True",
2805-
"SupervisorConnected",
2806-
"Supervisor relay is live",
2807-
false,
2808-
)
2809-
} else {
2810-
(
2811-
"False",
2812-
"DependenciesNotReady",
2813-
"Container is running; waiting for supervisor relay",
2814-
false,
2815-
)
2816-
}
2821+
("True", "BackendReady", "Container is running", false)
28172822
}
28182823
ContainerSummaryStateEnum::CREATED => ("False", "Starting", "Container created", false),
28192824
ContainerSummaryStateEnum::RESTARTING => (

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 9 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -141,14 +141,6 @@ fn inspected_volume(driver: &str, options: HashMap<String, String>) -> bollard::
141141
}
142142
}
143143

144-
struct DisconnectedSupervisorReadiness;
145-
146-
impl SupervisorReadiness for DisconnectedSupervisorReadiness {
147-
fn is_supervisor_connected(&self, _sandbox_id: &str) -> bool {
148-
false
149-
}
150-
}
151-
152144
fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDriver {
153145
let allow_all_default_gpu = config.allow_all_default_gpu;
154146
DockerComputeDriver {
@@ -159,7 +151,6 @@ fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDr
159151
config,
160152
events: broadcast::channel(WATCH_BUFFER).0,
161153
pending: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
162-
supervisor_readiness: Arc::new(DisconnectedSupervisorReadiness),
163154
gpu_selector: Arc::new(CdiGpuDefaultSelector::new(
164155
CdiGpuInventory::default(),
165156
allow_all_default_gpu,
@@ -1728,34 +1719,19 @@ fn driver_status_keeps_running_sandboxes_provisioning_with_stable_message() {
17281719
..running.clone()
17291720
};
17301721

1731-
let running_status = driver_status_from_summary(&running, "demo", false);
1732-
let running_later_status = driver_status_from_summary(&running_later, "demo", false);
1733-
assert_eq!(running_status.conditions[0].status, "False");
1734-
assert_eq!(running_status.conditions[0].reason, "DependenciesNotReady");
1735-
assert_eq!(
1736-
running_status.conditions[0].message,
1737-
"Container is running; waiting for supervisor relay"
1738-
);
1722+
// A running container always emits Ready=True with BackendReady. The gateway
1723+
// composes this with supervisor-session presence to decide public SandboxPhase.
1724+
let running_status = driver_status_from_summary(&running, "demo");
1725+
let running_later_status = driver_status_from_summary(&running_later, "demo");
1726+
assert_eq!(running_status.conditions[0].status, "True");
1727+
assert_eq!(running_status.conditions[0].reason, "BackendReady");
1728+
assert_eq!(running_status.conditions[0].message, "Container is running");
17391729
assert_eq!(running_status.conditions, running_later_status.conditions);
17401730

1741-
let exited_status = driver_status_from_summary(&exited, "demo", false);
1731+
let exited_status = driver_status_from_summary(&exited, "demo");
17421732
assert_eq!(exited_status.conditions[0].status, "False");
17431733
assert_eq!(exited_status.conditions[0].reason, "ContainerExited");
17441734
assert_eq!(exited_status.conditions[0].message, "Container exited");
1745-
1746-
// With a live supervisor session, a RUNNING container flips Ready=True
1747-
// so ExecSandbox and other "sandbox must be ready" gates can proceed.
1748-
let running_connected = driver_status_from_summary(&running, "demo", true);
1749-
assert_eq!(running_connected.conditions[0].status, "True");
1750-
assert_eq!(
1751-
running_connected.conditions[0].reason,
1752-
"SupervisorConnected"
1753-
);
1754-
1755-
// Supervisor readiness is ignored for non-RUNNING states -- an exited
1756-
// container must not report Ready=True.
1757-
let exited_connected = driver_status_from_summary(&exited, "demo", true);
1758-
assert_eq!(exited_connected.conditions[0].status, "False");
17591735
}
17601736

17611737
#[test]
@@ -1773,7 +1749,7 @@ fn driver_status_marks_restarting_sandboxes_as_error() {
17731749
..Default::default()
17741750
};
17751751

1776-
let status = driver_status_from_summary(&restarting, "demo", false);
1752+
let status = driver_status_from_summary(&restarting, "demo");
17771753
assert_eq!(status.conditions[0].status, "False");
17781754
assert_eq!(status.conditions[0].reason, "ContainerRestarting");
17791755
assert_eq!(

0 commit comments

Comments
 (0)