Skip to content

Commit bd649a6

Browse files
committed
fix(compute): recover Error-phase sandboxes on gateway startup
Podman machine restarts terminate sandbox containers out of band. The watcher observes the exit and drives the sandbox into the Error phase, but the container remains present and restartable, leaving the sandbox stuck with no path back to Ready. The gateway startup sweep now recovers sandboxes in Error whose reason is a recoverable exit or stop. Recovery is conservative and fails closed: only container exits confirmed to be caused by an external signal are recovered. - SIGTERM exits (143) are treated as external and recovered. - SIGKILL exits (137) are recovered only when inspect confirms the kill was not an OOM kill; where the OOM flag is unavailable (the container list path), 137 fails closed to CONDITION_FAILED. - Self-exits (including exit 0), OOM kills, and crashes stay in Error. - The Podman inspect path distinguishes stopped from exited so an explicit stop is recovered as CONDITION_STOPPED rather than misread. clear_recoverable_error() clears the error state via a compare-and-swap and refuses to act when the record has no resource version, avoiding a blind overwrite. Scope is limited to the Podman driver. Fixes NVIDIA#2179 Signed-off-by: Ian Miller <[email protected]>
1 parent 7909fb5 commit bd649a6

9 files changed

Lines changed: 930 additions & 52 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,18 @@ sandbox during gateway startup` in gateway logs. A sandbox explicitly stopped
201201
through the CLI remains stopped. Kubernetes sandboxes are cluster-owned and do
202202
not follow this local gateway lifecycle.
203203

204+
A sandbox whose container exits out of band (a Podman machine or daemon
205+
restart) lands in `Error`. On startup the sweep recovers these for Podman only —
206+
Docker and VM leave `Error` untouched. Recovery re-inspects the live backend
207+
rather than trusting the persisted reason: it restarts only when the live Ready
208+
reason is `ContainerExited` or `ContainerStopped`, and leaves `Error` when the
209+
live reason is `ContainerFailed`/`OOMKilled`, when the container is gone, or when
210+
the inspection fails. Look for `Started sandbox during gateway startup` with
211+
`recovered=true`, or `Failed to re-inspect sandbox during startup recovery` when
212+
a live lookup errored. A Podman sandbox stuck in `Error` after a machine restart
213+
usually means the container is missing or the live state classified as a
214+
non-recoverable failure.
215+
204216
### Step 5: Check Podman-Backed Gateways
205217

206218
```bash

‎architecture/compute-runtimes.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,39 @@ shared idempotent `StartSandbox` RPC before watch processing begins. Explicitly
119119
cluster-owned and continue running without gateway shutdown or startup
120120
lifecycle calls.
121121

122+
### Error-phase recovery on startup
123+
124+
A container that exits out of band — a Podman machine restart, a daemon
125+
restart — drives its sandbox into `Error`. The startup sweep recovers these,
126+
but only for Podman: Docker and VM leave `Error` rows untouched.
127+
128+
Recovery does not trust the persisted `Error` reason. The sweep runs before
129+
watchers reconcile persisted state, so the stored Ready reason can be stale
130+
(an older gateway persisted `ContainerExited` for every non-OOM exit, including
131+
a clean self-exit). Instead, the sweep re-inspects the live backend with
132+
`GetSandbox` and decides from the live Ready reason:
133+
134+
- `ContainerExited` / `ContainerStopped` — recoverable; the sweep restarts the
135+
container and clears the error.
136+
- `ContainerFailed` / `OOMKilled` — a genuine application or infrastructure
137+
failure; left in `Error`.
138+
- backend reports no container, or `GetSandbox` errors — left in `Error` for the
139+
reconcile loop to prune.
140+
141+
Classification is fail-closed and depends on the inspection path. The Podman
142+
list-entry path lacks the OOM flag, so it recovers only exit 143 (SIGTERM) and
143+
treats exit 137 as `ContainerFailed`. The inspect path sees the OOM flag and can
144+
additionally distinguish a non-OOM SIGKILL (137) restart from a true OOM kill.
145+
Podman `GetSandbox` therefore inspects exited and stopped containers, not just
146+
running ones, so live re-inspect classifies the real machine-restart case
147+
correctly.
148+
149+
An intentional stop is confirmed by any non-running terminal state —
150+
`ContainerExited`, `ContainerStopped`, `ContainerFailed`, or `OOMKilled`. A
151+
SIGTERM-trapping entrypoint that exits 0 during shutdown classifies as
152+
`ContainerFailed` under fail-closed rules; treating that as a confirmed stop
153+
keeps a graceful stop from regressing to `Error`.
154+
122155
## Deletion Lifecycle
123156

124157
Lifecycle requests use per-sandbox gates to serialize stop, start, and

‎crates/openshell-core/src/driver_utils.rs‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,54 @@ pub fn openshell_sandbox_label_selector() -> String {
4545
format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE},{LABEL_SANDBOX_ID}")
4646
}
4747

48+
// ---------------------------------------------------------------------------
49+
// Sandbox condition reason strings set by compute drivers.
50+
// ---------------------------------------------------------------------------
51+
52+
/// Ready-condition reason when a container is terminated by an external signal.
53+
///
54+
/// SIGTERM/SIGKILL — e.g. a machine or daemon restart. Eligible for startup
55+
/// recovery. OOM kills use the dedicated `OOMKilled` reason and a self-exit
56+
/// (crash or clean exit) uses [`CONDITION_FAILED`]; neither is recoverable.
57+
pub const CONDITION_EXITED: &str = "ContainerExited";
58+
59+
/// Ready-condition reason when a container is explicitly stopped via the
60+
/// runtime API (e.g. `podman stop`, gateway-initiated shutdown). Eligible for
61+
/// startup recovery.
62+
pub const CONDITION_STOPPED: &str = "ContainerStopped";
63+
64+
/// Ready-condition reason when a container exits on its own.
65+
///
66+
/// An application crash or a clean exit — rather than being terminated by an
67+
/// external signal. Not eligible for startup recovery: reviving a self-exited
68+
/// container would restart a process that intentionally or fatally stopped.
69+
pub const CONDITION_FAILED: &str = "ContainerFailed";
70+
71+
/// Container exit code for termination by SIGTERM (128 + 15). This is the
72+
/// graceful stop a container runtime issues to sandboxes on machine or daemon
73+
/// restart — the case startup recovery exists to repair.
74+
pub const EXIT_CODE_SIGTERM: i64 = 143;
75+
76+
/// Container exit code for termination by SIGKILL (128 + 9) — a hard stop.
77+
/// The OOM killer also uses SIGKILL, but drivers report OOM via a dedicated
78+
/// reason before this classifier runs.
79+
pub const EXIT_CODE_SIGKILL: i64 = 137;
80+
81+
/// Whether a container exit code indicates termination by an external signal.
82+
///
83+
/// SIGTERM/SIGKILL rather than the process exiting on its own. Only
84+
/// external-signal exits are eligible for startup recovery; a self-exit
85+
/// (crash, `exit 0/1/…`) leaves the sandbox in `Error`.
86+
///
87+
/// A 137 (SIGKILL) exit is also what the OOM killer produces, so callers must
88+
/// rule out an OOM kill (via the runtime's OOM flag) before treating a 137 as
89+
/// recoverable. Where the OOM flag is unavailable — e.g. a container-list
90+
/// summary — classify only 143 ([`EXIT_CODE_SIGTERM`]) as an external signal.
91+
#[must_use]
92+
pub fn is_external_signal_exit(exit_code: i64) -> bool {
93+
exit_code == EXIT_CODE_SIGTERM || exit_code == EXIT_CODE_SIGKILL
94+
}
95+
4896
// ---------------------------------------------------------------------------
4997

5098
/// Path to the sandbox supervisor binary inside the container image.

‎crates/openshell-driver-podman/src/client.rs‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,6 @@ pub struct ContainerListEntry {
196196
#[allow(dead_code)] // kept for podman API compat
197197
pub networks: Option<Vec<String>>,
198198
#[serde(default)]
199-
#[allow(dead_code)] // kept for podman API compat
200199
pub exit_code: i64,
201200
}
202201

‎crates/openshell-driver-podman/src/driver.rs‎

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1095,7 +1095,14 @@ impl PodmanComputeDriver {
10951095
let Some(entry) = entries.first() else {
10961096
return Ok(None);
10971097
};
1098-
if entry.state == "running" {
1098+
// Inspect running containers for health status and non-running
1099+
// (exited/stopped) containers for the OOM flag and exit code. The
1100+
// list entry lacks the OOM flag, so its exited classification is
1101+
// fail-closed (only SIGTERM is recoverable); the inspect path can
1102+
// additionally distinguish a non-OOM SIGKILL restart. Startup
1103+
// recovery relies on this snapshot, so prefer inspect and fall back
1104+
// to the list entry only when inspect fails.
1105+
if matches!(entry.state.as_str(), "running" | "exited" | "stopped") {
10991106
Ok(self
11001107
.client
11011108
.inspect_container(&entry.id)
@@ -1110,8 +1117,13 @@ impl PodmanComputeDriver {
11101117

11111118
/// List all managed sandboxes.
11121119
///
1113-
/// Only inspects running containers (to get health status). Non-running
1114-
/// containers are built directly from the list entry data.
1120+
/// Inspects running, exited, and stopped containers so their classification
1121+
/// matches `get_sandbox`: health status for running containers, and the OOM
1122+
/// flag plus exit code for terminal ones. The list entry carries no OOM
1123+
/// flag, so without inspecting terminal containers a non-OOM SIGKILL (137)
1124+
/// exit would fail-close to `ContainerFailed`, disagreeing with the
1125+
/// inspect-based watcher and startup recovery. `created`/`removing` entries
1126+
/// have no useful inspect data and are built directly from the list entry.
11151127
pub async fn list_sandboxes(&self) -> Result<Vec<DriverSandbox>, ComputeDriverError> {
11161128
let entries = self
11171129
.client
@@ -1121,8 +1133,9 @@ impl PodmanComputeDriver {
11211133

11221134
let mut sandboxes = Vec::with_capacity(entries.len());
11231135
for entry in &entries {
1124-
if entry.state == "running" {
1125-
// Running containers need inspect for health check status.
1136+
if matches!(entry.state.as_str(), "running" | "exited" | "stopped") {
1137+
// Inspect for health status (running) and the OOM flag / exit
1138+
// code (terminal states); fall back to the list entry on failure.
11261139
match self.client.inspect_container(&entry.id).await {
11271140
Ok(inspect) => {
11281141
if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) {
@@ -1135,12 +1148,12 @@ impl PodmanComputeDriver {
11351148
warn!(
11361149
container = %name,
11371150
error = %e,
1138-
"Failed to inspect running container during list, falling back to list entry"
1151+
"Failed to inspect container during list, falling back to list entry"
11391152
);
11401153
}
11411154
}
11421155
}
1143-
// Non-running containers (or inspect fallback): build from list data.
1156+
// created/removing entries, or inspect fallback: build from list data.
11441157
if let Some(sandbox) = driver_sandbox_from_list_entry(entry) {
11451158
sandboxes.push(sandbox);
11461159
}

‎crates/openshell-driver-podman/src/watcher.rs‎

Lines changed: 136 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,10 @@ use tracing::{debug, info, warn};
2626
// Condition reason constants shared across event-building paths.
2727
const CONDITION_RUNNING: &str = "ContainerRunning";
2828
const CONDITION_STARTING: &str = "ContainerStarting";
29-
const CONDITION_EXITED: &str = "ContainerExited";
30-
const CONDITION_STOPPED: &str = "ContainerStopped";
29+
use openshell_core::driver_utils::{
30+
CONDITION_EXITED, CONDITION_FAILED, CONDITION_STOPPED, EXIT_CODE_SIGTERM,
31+
is_external_signal_exit,
32+
};
3133

3234
pub type WatchStream =
3335
Pin<Box<dyn Stream<Item = Result<WatchSandboxesEvent, ComputeDriverError>> + Send>>;
@@ -143,10 +145,13 @@ pub async fn start_watch(
143145
let existing = client.list_containers(&[LABEL_MANAGED_FILTER]).await?;
144146

145147
for entry in &existing {
146-
// For running containers, use inspect to get full state including
147-
// health check status — matching the same condition derivation used
148-
// for live events.
149-
if entry.state == "running" {
148+
// Inspect running containers for health status, and exited/stopped
149+
// containers for the exit code and OOM flag — matching the condition
150+
// derivation used for live events. The list entry carries no OOM flag,
151+
// so an already-exited container observed only through the list path
152+
// cannot distinguish an OOM kill (137 + oom flag) from a SIGKILL
153+
// restart; inspecting here lets `condition_from_state` classify it.
154+
if matches!(entry.state.as_str(), "running" | "exited" | "stopped") {
150155
match client.inspect_container(&entry.id).await {
151156
Ok(inspect) => {
152157
if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) {
@@ -161,8 +166,9 @@ pub async fn start_watch(
161166
Err(e) => {
162167
warn!(
163168
container_id = %entry.id,
169+
state = %entry.state,
164170
error = %e,
165-
"Failed to inspect running container during initial sync, falling back to list entry"
171+
"Failed to inspect container during initial sync, falling back to list entry"
166172
);
167173
}
168174
}
@@ -405,7 +411,19 @@ pub fn driver_sandbox_from_list_entry(entry: &ContainerListEntry) -> Option<Driv
405411
"Container is running".to_string(),
406412
),
407413
"created" => ("ContainerCreated", "False", String::new()),
408-
"exited" => (CONDITION_EXITED, "False", String::new()),
414+
// The list entry carries no OOM flag, so a 137 (SIGKILL) exit cannot be
415+
// told apart from an OOM kill here. Fail closed: only a SIGTERM (143)
416+
// exit — which is never an OOM kill — is treated as a recoverable
417+
// external-signal exit from the list alone; every other exit becomes a
418+
// non-recoverable `CONDITION_FAILED`. Already-exited containers are
419+
// inspected during initial sync, so the inspect-based
420+
// `condition_from_state` path (which sees the OOM flag) classifies 137
421+
// exits and OOM kills correctly; this list path is the degraded
422+
// fallback used only when that inspect fails.
423+
"exited" if entry.exit_code == EXIT_CODE_SIGTERM => {
424+
(CONDITION_EXITED, "False", String::new())
425+
}
426+
"exited" => (CONDITION_FAILED, "False", String::new()),
409427
"stopped" => (CONDITION_STOPPED, "False", String::new()),
410428
"removing" => ("ContainerRemoving", "False", String::new()),
411429
_ => ("Unknown", "Unknown", String::new()),
@@ -444,16 +462,30 @@ fn condition_from_state(state: &ContainerState) -> DriverCondition {
444462
_ => ("False", CONDITION_STARTING, String::new()),
445463
},
446464
"created" => ("False", "ContainerCreated", String::new()),
447-
"exited" | "stopped" => {
465+
// An explicitly stopped container is recoverable regardless of its exit
466+
// code, matching the list-path classification and keeping the two paths
467+
// consistent.
468+
"stopped" => ("False", CONDITION_STOPPED, "Container stopped".to_string()),
469+
"exited" => {
448470
let msg = if state.oom_killed {
449471
"Container was killed by the OOM killer".to_string()
450472
} else {
451473
format!("Container exited with code {}", state.exit_code)
452474
};
475+
// This is the only path that sees Podman's OOM flag, so it — not the
476+
// list path — decides how a 137 exit is classified. Trust the flag:
477+
// a real OOM kill is non-recoverable `OOMKilled`, while an
478+
// external-signal exit (SIGTERM 143 / SIGKILL 137, e.g. a machine
479+
// restart) is a recoverable `CONDITION_EXITED`. If Podman ever
480+
// under-reports oom_killed, an OOM container would be restarted once
481+
// at the next gateway startup sweep — recovery is one-shot per
482+
// startup, not a restart loop.
453483
let reason = if state.oom_killed {
454484
"OOMKilled"
455-
} else {
485+
} else if is_external_signal_exit(state.exit_code) {
456486
CONDITION_EXITED
487+
} else {
488+
CONDITION_FAILED
457489
};
458490
("False", reason, msg)
459491
}
@@ -584,7 +616,9 @@ mod tests {
584616
}
585617

586618
#[test]
587-
fn condition_normal_exit() {
619+
fn condition_self_exit_is_failed() {
620+
// A small non-signal exit code is a self-exit (crash or clean exit),
621+
// not an external signal. Not eligible for startup recovery.
588622
let state = ContainerState {
589623
status: "exited".to_string(),
590624
running: false,
@@ -596,10 +630,31 @@ mod tests {
596630
};
597631
let cond = condition_from_state(&state);
598632
assert_eq!(cond.status, "False");
599-
assert_eq!(cond.reason, "ContainerExited");
633+
assert_eq!(cond.reason, "ContainerFailed");
600634
assert!(cond.message.contains("code 1"));
601635
}
602636

637+
#[test]
638+
fn condition_external_signal_exit_is_recoverable() {
639+
// SIGTERM (143) / SIGKILL (137) without OOM indicate an external signal
640+
// such as a machine or daemon restart. Eligible for startup recovery.
641+
for exit_code in [143, 137] {
642+
let state = ContainerState {
643+
status: "exited".to_string(),
644+
running: false,
645+
exit_code,
646+
oom_killed: false,
647+
health: None,
648+
started_at: None,
649+
finished_at: Some("2026-04-14T12:00:00Z".to_string()),
650+
};
651+
let cond = condition_from_state(&state);
652+
assert_eq!(cond.status, "False");
653+
assert_eq!(cond.reason, "ContainerExited");
654+
assert!(cond.message.contains(&format!("code {exit_code}")));
655+
}
656+
}
657+
603658
#[test]
604659
fn short_id_truncates() {
605660
assert_eq!(short_id("abc123def456789"), "abc123def456");
@@ -632,6 +687,75 @@ mod tests {
632687
assert!(!status.deleting);
633688
}
634689

690+
fn list_entry(state: &str, exit_code: i64) -> ContainerListEntry {
691+
let mut labels = HashMap::new();
692+
labels.insert(LABEL_SANDBOX_ID.to_string(), "test-id".to_string());
693+
labels.insert(LABEL_SANDBOX_NAME.to_string(), "test-name".to_string());
694+
labels.insert(LABEL_SANDBOX_WORKSPACE.to_string(), "default".to_string());
695+
ContainerListEntry {
696+
id: "abc123def456789".to_string(),
697+
names: vec!["openshell-sandbox-test-name".to_string()],
698+
state: state.to_string(),
699+
labels,
700+
ports: None,
701+
networks: None,
702+
exit_code,
703+
}
704+
}
705+
706+
fn list_entry_reason(state: &str, exit_code: i64) -> String {
707+
let sandbox = driver_sandbox_from_list_entry(&list_entry(state, exit_code))
708+
.expect("should produce a sandbox");
709+
sandbox.status.expect("should have status").conditions[0]
710+
.reason
711+
.clone()
712+
}
713+
714+
#[test]
715+
fn list_entry_sigterm_exit_is_recoverable() {
716+
// SIGTERM (143) is never an OOM kill, so it is safe to classify as a
717+
// recoverable external-signal exit even without the OOM flag.
718+
assert_eq!(list_entry_reason("exited", 143), CONDITION_EXITED);
719+
}
720+
721+
#[test]
722+
fn list_entry_sigkill_exit_fails_closed() {
723+
// A 137 exit is indistinguishable from an OOM kill without the OOM flag,
724+
// which the list entry does not carry. Fail closed to a non-recoverable
725+
// failure; the inspect path reclassifies genuine SIGKILL restarts.
726+
assert_eq!(list_entry_reason("exited", 137), CONDITION_FAILED);
727+
}
728+
729+
#[test]
730+
fn list_entry_self_exit_is_failed() {
731+
assert_eq!(list_entry_reason("exited", 1), CONDITION_FAILED);
732+
assert_eq!(list_entry_reason("exited", 0), CONDITION_FAILED);
733+
}
734+
735+
#[test]
736+
fn list_entry_stopped_is_recoverable() {
737+
assert_eq!(list_entry_reason("stopped", 0), CONDITION_STOPPED);
738+
}
739+
740+
#[test]
741+
fn condition_stopped_is_recoverable() {
742+
// The inspect path must yield `CONDITION_STOPPED` for a stopped
743+
// container, matching the list path rather than folding it into the
744+
// exited/failed classification.
745+
let state = ContainerState {
746+
status: "stopped".to_string(),
747+
running: false,
748+
exit_code: 0,
749+
oom_killed: false,
750+
health: None,
751+
started_at: None,
752+
finished_at: Some("2026-04-14T12:00:00Z".to_string()),
753+
};
754+
let cond = condition_from_state(&state);
755+
assert_eq!(cond.status, "False");
756+
assert_eq!(cond.reason, CONDITION_STOPPED);
757+
}
758+
635759
#[test]
636760
fn synthetic_inspect_failed_event_structure() {
637761
// Verify the structure of an inspect-failure event by constructing one

0 commit comments

Comments
 (0)