Skip to content

Commit 92a2029

Browse files
drewelezar
authored andcommitted
refactor(compute): unify gateway restart reconciliation
Remove the Docker-specific gateway shutdown cleanup and reconcile persisted running intent through ComputeDriver::StartSandbox for Docker, Podman, and VM drivers. Explicitly stopped sandboxes remain stopped. Refs #2417 Signed-off-by: Drew Newberry <[email protected]>
1 parent 701382d commit 92a2029

12 files changed

Lines changed: 261 additions & 259 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,15 @@ compute to zero, and VM retains its launch request and writable overlay beside
110110
a stop marker. Delete remains a separate operation that removes these
111111
resources.
112112

113+
Gateway shutdown does not stop Docker or Podman sandbox containers. The
114+
gateway-owned VM driver process exits, so its running VM launchers exit with
115+
it. On gateway startup, persisted running intent for Docker, Podman, and VM is
116+
reconciled through the shared idempotent `StartSandbox` RPC before watch
117+
processing begins. This restarts a retained local resource when necessary and
118+
is a no-op when it is already running. Explicitly `Stopped` sandboxes are
119+
excluded from the sweep. Kubernetes workloads are cluster-owned and continue
120+
running without a startup `StartSandbox` call.
121+
113122
## Deletion Lifecycle
114123

115124
Lifecycle requests use per-sandbox gates to serialize stop, start, and

‎crates/openshell-driver-docker/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ policy. Start starts that same container, so files in the resolved OCI
2626
workspace remain available. A durably stopped sandbox is excluded from
2727
gateway startup recovery and stays stopped across gateway restarts. Delete
2828
continues to force-remove the container and clean up driver-owned material.
29+
Gateway shutdown does not stop running Docker containers. On startup, the
30+
gateway sends an idempotent `StartSandbox` request for each sandbox whose
31+
persisted phase requires running compute, recovering a retained container if
32+
the Docker runtime stopped it while the gateway was offline.
2933

3034
Before creating the container, the driver inspects the final sandbox image and
3135
captures its immutable image ID, raw OCI `Config.User`, and OCI

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 0 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1055,69 +1055,6 @@ impl DockerComputeDriver {
10551055
}
10561056
}
10571057

1058-
pub async fn stop_managed_containers_on_shutdown(&self) -> Result<usize, Status> {
1059-
let containers = self.list_managed_container_summaries().await?;
1060-
let targets = containers
1061-
.into_iter()
1062-
.filter_map(|container| {
1063-
let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY);
1064-
if container_state_needs_shutdown_stop(state) {
1065-
summary_container_target(&container)
1066-
} else {
1067-
None
1068-
}
1069-
})
1070-
.collect::<Vec<_>>();
1071-
let target_count = targets.len();
1072-
let mut stopped = 0usize;
1073-
let mut failures = Vec::new();
1074-
let stop_timeout_secs = self.config.stop_timeout_secs;
1075-
1076-
let mut stop_results = futures::stream::iter(targets.into_iter().map(|target| {
1077-
let docker = self.docker.clone();
1078-
async move {
1079-
let result = docker
1080-
.stop_container(
1081-
&target,
1082-
Some(
1083-
StopContainerOptionsBuilder::default()
1084-
.t(docker_stop_timeout_secs(stop_timeout_secs))
1085-
.build(),
1086-
),
1087-
)
1088-
.await;
1089-
(target, result)
1090-
}
1091-
}))
1092-
.buffer_unordered(16);
1093-
1094-
while let Some((target, result)) = stop_results.next().await {
1095-
match result {
1096-
Ok(()) => {
1097-
stopped += 1;
1098-
}
1099-
Err(err) if is_not_found_error(&err) || is_not_modified_error(&err) => {}
1100-
Err(err) => {
1101-
warn!(
1102-
container = %target,
1103-
error = %err,
1104-
"Failed to stop Docker sandbox container during shutdown"
1105-
);
1106-
failures.push(target);
1107-
}
1108-
}
1109-
}
1110-
1111-
if !failures.is_empty() {
1112-
return Err(Status::internal(format!(
1113-
"failed to stop {} of {target_count} Docker sandbox containers during shutdown",
1114-
failures.len()
1115-
)));
1116-
}
1117-
1118-
Ok(stopped)
1119-
}
1120-
11211058
async fn reserve_pending_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), Status> {
11221059
let mut pending = self.pending.lock().await;
11231060
if pending
@@ -3200,15 +3137,6 @@ fn summary_container_target(summary: &ContainerSummary) -> Option<String> {
32003137
.or_else(|| summary_container_name(summary))
32013138
}
32023139

3203-
fn container_state_needs_shutdown_stop(state: ContainerSummaryStateEnum) -> bool {
3204-
matches!(
3205-
state,
3206-
ContainerSummaryStateEnum::RUNNING
3207-
| ContainerSummaryStateEnum::RESTARTING
3208-
| ContainerSummaryStateEnum::PAUSED
3209-
)
3210-
}
3211-
32123140
/// States from which a managed container can be brought back to running by
32133141
/// `start_container`. Skip `Restarting` (already coming up), `Removing`,
32143142
/// `Dead` (terminal), `Paused` (needs `unpause`, not `start`), and

‎crates/openshell-driver-podman/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,12 @@ The stop call waits until Podman reports the container as stopped or exited.
3232
This keeps an immediate start from racing a rootless Podman stop that is still
3333
finishing after its API request returns.
3434

35+
Gateway shutdown does not stop running Podman containers. On startup, the
36+
gateway sends an idempotent `StartSandbox` request for each sandbox whose
37+
persisted phase requires running compute. This is a no-op for a container that
38+
survived and recovers a retained container stopped by a Podman machine or
39+
runtime restart. Explicitly stopped sandboxes are excluded.
40+
3541
## Architecture
3642

3743
The Podman driver communicates with the Podman daemon over a Unix socket and

‎crates/openshell-driver-vm/README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,11 @@ The driver also writes the accepted `DriverSandbox` launch request to
197197
new VM driver process; that process scans the sandbox state directories,
198198
restarts each persisted VM launcher, and preserves any existing `overlay.ext4`
199199
instead of cloning a fresh overlay template. If a restart happened before the
200-
overlay was created, the driver creates it during the start attempt.
200+
overlay was created, the driver creates it during the start attempt. After
201+
driver initialization, the gateway also sends the shared idempotent
202+
`StartSandbox` request for each persisted running-intent sandbox. Already
203+
restored VMs are unchanged, while stopped markers continue to exclude
204+
explicitly stopped sandboxes from startup.
201205
202206
Stop writes a marker in the sandbox state directory before terminating
203207
the launcher and releasing host GPU and network allocations. It retains

0 commit comments

Comments
 (0)