Skip to content

Commit 428b859

Browse files
committed
fix(compute): recover Error-phase sandboxes on gateway startup
Sandboxes whose container exited on its own (e.g. SIGTERM from a Podman or Docker machine restart) were left stuck in the Error phase even though the container could be restarted. The startup sweep skipped every Error-phase sandbox unconditionally. Include Error-phase sandboxes in the startup sweep when their Ready condition reason marks a container exit or stop. If the driver restarts the container, move the sandbox back to Provisioning with a Resumed condition; if the container is gone or the start fails, leave the Error state untouched. Genuine (non-container-exit) errors are still skipped. Container-exit restart is handled by the drivers' existing idempotent start_sandbox path, which already restarts stopped/exited containers. The ContainerExited/ContainerStopped condition reasons are promoted to shared constants in openshell-core so the gateway and drivers agree on the recovery signal. Fixes NVIDIA#2179 Signed-off-by: Ian Miller <[email protected]>
1 parent de4c1fe commit 428b859

4 files changed

Lines changed: 264 additions & 22 deletions

File tree

‎crates/openshell-core/src/driver_utils.rs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,18 @@ pub fn openshell_sandbox_label_selector() -> String {
4545
format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE},{LABEL_SANDBOX_ID}")
4646
}
4747

48+
// ---------------------------------------------------------------------------
49+
// Sandbox condition reason strings set by compute drivers.
50+
// ---------------------------------------------------------------------------
51+
52+
/// Ready-condition reason when a container exits on its own (e.g. SIGTERM
53+
/// from a machine restart, OOM kill, application crash).
54+
pub const CONDITION_EXITED: &str = "ContainerExited";
55+
56+
/// Ready-condition reason when a container is explicitly stopped via the
57+
/// runtime API (e.g. `podman stop`, gateway-initiated shutdown).
58+
pub const CONDITION_STOPPED: &str = "ContainerStopped";
59+
4860
// ---------------------------------------------------------------------------
4961

5062
/// Path to the sandbox supervisor binary inside the container image.

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3116,7 +3116,12 @@ fn container_ready_condition(
31163116
("False", "ContainerPaused", "Container is paused", false)
31173117
}
31183118
ContainerSummaryStateEnum::EXITED => {
3119-
("False", "ContainerExited", "Container exited", false)
3119+
(
3120+
"False",
3121+
openshell_core::driver_utils::CONDITION_EXITED,
3122+
"Container exited",
3123+
false,
3124+
)
31203125
}
31213126
ContainerSummaryStateEnum::DEAD => ("False", "ContainerDead", "Container is dead", false),
31223127
}

‎crates/openshell-driver-podman/src/watcher.rs‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,7 @@ use tracing::{debug, info, warn};
2626
// Condition reason constants shared across event-building paths.
2727
const CONDITION_RUNNING: &str = "ContainerRunning";
2828
const CONDITION_STARTING: &str = "ContainerStarting";
29-
const CONDITION_EXITED: &str = "ContainerExited";
30-
const CONDITION_STOPPED: &str = "ContainerStopped";
29+
use openshell_core::driver_utils::{CONDITION_EXITED, CONDITION_STOPPED};
3130

3231
pub type WatchStream =
3332
Pin<Box<dyn Stream<Item = Result<WatchSandboxesEvent, ComputeDriverError>> + Send>>;

0 commit comments

Comments
 (0)