Skip to content

Commit 6702450

Browse files
authored
refactor(compute): unify gateway restart reconciliation
Remove the Docker-specific gateway shutdown cleanup and reconcile persisted running intent through ComputeDriver::StartSandbox for Docker, Podman, and VM drivers. Explicitly stopped sandboxes remain stopped. Closes #2417 Signed-off-by: Drew Newberry <[email protected]>
1 parent 2eb0880 commit 6702450

12 files changed

Lines changed: 261 additions & 259 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,15 @@ compute to zero, and VM retains its launch request and writable overlay beside
106106
a stop marker. Delete remains a separate operation that removes these
107107
resources.
108108

109+
Gateway shutdown does not stop Docker or Podman sandbox containers. The
110+
gateway-owned VM driver process exits, so its running VM launchers exit with
111+
it. On gateway startup, persisted running intent for Docker, Podman, and VM is
112+
reconciled through the shared idempotent `StartSandbox` RPC before watch
113+
processing begins. This restarts a retained local resource when necessary and
114+
is a no-op when it is already running. Explicitly `Stopped` sandboxes are
115+
excluded from the sweep. Kubernetes workloads are cluster-owned and continue
116+
running without a startup `StartSandbox` call.
117+
109118
## Deletion Lifecycle
110119

111120
Lifecycle requests use per-sandbox gates to serialize stop, start, and

‎crates/openshell-driver-docker/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ policy. Start starts that same container, so files in the resolved OCI
2626
workspace remain available. A durably stopped sandbox is excluded from
2727
gateway startup recovery and stays stopped across gateway restarts. Delete
2828
continues to force-remove the container and clean up driver-owned material.
29+
Gateway shutdown does not stop running Docker containers. On startup, the
30+
gateway sends an idempotent `StartSandbox` request for each sandbox whose
31+
persisted phase requires running compute, recovering a retained container if
32+
the Docker runtime stopped it while the gateway was offline.
2933

3034
Before creating the container, the driver inspects the final sandbox image and
3135
captures its immutable image ID, raw OCI `Config.User`, and OCI

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 0 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1055,69 +1055,6 @@ impl DockerComputeDriver {
10551055
}
10561056
}
10571057

1058-
pub async fn stop_managed_containers_on_shutdown(&self) -> Result<usize, Status> {
1059-
let containers = self.list_managed_container_summaries().await?;
1060-
let targets = containers
1061-
.into_iter()
1062-
.filter_map(|container| {
1063-
let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY);
1064-
if container_state_needs_shutdown_stop(state) {
1065-
summary_container_target(&container)
1066-
} else {
1067-
None
1068-
}
1069-
})
1070-
.collect::<Vec<_>>();
1071-
let target_count = targets.len();
1072-
let mut stopped = 0usize;
1073-
let mut failures = Vec::new();
1074-
let stop_timeout_secs = self.config.stop_timeout_secs;
1075-
1076-
let mut stop_results = futures::stream::iter(targets.into_iter().map(|target| {
1077-
let docker = self.docker.clone();
1078-
async move {
1079-
let result = docker
1080-
.stop_container(
1081-
&target,
1082-
Some(
1083-
StopContainerOptionsBuilder::default()
1084-
.t(docker_stop_timeout_secs(stop_timeout_secs))
1085-
.build(),
1086-
),
1087-
)
1088-
.await;
1089-
(target, result)
1090-
}
1091-
}))
1092-
.buffer_unordered(16);
1093-
1094-
while let Some((target, result)) = stop_results.next().await {
1095-
match result {
1096-
Ok(()) => {
1097-
stopped += 1;
1098-
}
1099-
Err(err) if is_not_found_error(&err) || is_not_modified_error(&err) => {}
1100-
Err(err) => {
1101-
warn!(
1102-
container = %target,
1103-
error = %err,
1104-
"Failed to stop Docker sandbox container during shutdown"
1105-
);
1106-
failures.push(target);
1107-
}
1108-
}
1109-
}
1110-
1111-
if !failures.is_empty() {
1112-
return Err(Status::internal(format!(
1113-
"failed to stop {} of {target_count} Docker sandbox containers during shutdown",
1114-
failures.len()
1115-
)));
1116-
}
1117-
1118-
Ok(stopped)
1119-
}
1120-
11211058
async fn reserve_pending_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), Status> {
11221059
let mut pending = self.pending.lock().await;
11231060
if pending
@@ -3200,15 +3137,6 @@ fn summary_container_target(summary: &ContainerSummary) -> Option<String> {
32003137
.or_else(|| summary_container_name(summary))
32013138
}
32023139

3203-
fn container_state_needs_shutdown_stop(state: ContainerSummaryStateEnum) -> bool {
3204-
matches!(
3205-
state,
3206-
ContainerSummaryStateEnum::RUNNING
3207-
| ContainerSummaryStateEnum::RESTARTING
3208-
| ContainerSummaryStateEnum::PAUSED
3209-
)
3210-
}
3211-
32123140
/// States from which a managed container can be brought back to running by
32133141
/// `start_container`. Skip `Restarting` (already coming up), `Removing`,
32143142
/// `Dead` (terminal), `Paused` (needs `unpause`, not `start`), and

‎crates/openshell-driver-podman/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,12 @@ volume. Stopped managed containers remain visible through list and watch
2828
reconciliation. Delete remains responsible for removing the container,
2929
driver-owned secrets, and workspace volume.
3030

31+
Gateway shutdown does not stop running Podman containers. On startup, the
32+
gateway sends an idempotent `StartSandbox` request for each sandbox whose
33+
persisted phase requires running compute. This is a no-op for a container that
34+
survived and recovers a retained container stopped by a Podman machine or
35+
runtime restart. Explicitly stopped sandboxes are excluded.
36+
3137
## Architecture
3238

3339
The Podman driver communicates with the Podman daemon over a Unix socket and

‎crates/openshell-driver-vm/README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,11 @@ The driver also writes the accepted `DriverSandbox` launch request to
197197
new VM driver process; that process scans the sandbox state directories,
198198
restarts each persisted VM launcher, and preserves any existing `overlay.ext4`
199199
instead of cloning a fresh overlay template. If a restart happened before the
200-
overlay was created, the driver creates it during the start attempt.
200+
overlay was created, the driver creates it during the start attempt. After
201+
driver initialization, the gateway also sends the shared idempotent
202+
`StartSandbox` request for each persisted running-intent sandbox. Already
203+
restored VMs are unchanged, while stopped markers continue to exclude
204+
explicitly stopped sandboxes from startup.
201205
202206
Stop writes a marker in the sandbox state directory before terminating
203207
the launcher and releasing host GPU and network allocations. It retains

0 commit comments

Comments
 (0)