Skip to content

Commit acbac9c

Browse files
authored
feat(sandbox): add main restart policy (#2798)
* feat(sandbox): add main restart policy Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): harden policy-driven restarts Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): address restart review feedback Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): port restart policy to current runtime Signed-off-by: Drew Newberry <[email protected]> * perf(sandbox): restart promptly after terminal delivery Signed-off-by: Drew Newberry <[email protected]> --------- Signed-off-by: Drew Newberry <[email protected]> Signed-off-by: Drew Newberry <[email protected]>
1 parent 1358941 commit acbac9c

36 files changed

Lines changed: 3072 additions & 1002 deletions

File tree

‎.agents/skills/tui-development/SKILL.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ match app.screen {
102102
}
103103
```
104104

105-
Within the `Sandbox` screen, the top 20% renders sandbox metadata (`sandbox_detail`), and the bottom 80% dispatches based on focus and tab state:
105+
Within the `Sandbox` screen, `sandbox_detail::required_height` sizes the metadata and restart status pane to its contents. The remaining area dispatches based on focus and tab state:
106106

107107
```rust
108108
match app.focus {
@@ -262,7 +262,7 @@ The `Theme` struct has 16 `Style` fields, accessed at runtime via `app.theme`:
262262
| `border` | EVERGLADE fg | Light sage fg | Unfocused panel borders |
263263
| `border_focused` | NVIDIA_GREEN fg | NVIDIA_GREEN_DARK fg | Focused panel borders |
264264
| `status_ok` | NVIDIA_GREEN fg | NVIDIA_GREEN_DARK fg | Healthy, INFO, Ready |
265-
| `status_warn` | Yellow fg | Dark yellow fg | Degraded, WARN, Provisioning |
265+
| `status_warn` | Yellow fg | Dark yellow fg | Degraded, WARN, Provisioning, Starting |
266266
| `status_err` | Red fg | Dark red fg | Unhealthy, ERROR |
267267
| `key_hint` | NVIDIA_GREEN fg | NVIDIA_GREEN_DARK fg | Keyboard shortcut labels |
268268
| `log_cursor` | EVERGLADE bg | Light green bg | Selected log line highlight |
@@ -297,7 +297,7 @@ fn draw_detail_popup(frame: &mut Frame<'_>, data: &MyData, area: Rect, theme: &T
297297

298298
- **Selected row**: Green `▌` left-border marker on the selected row. Active gateway also gets a green `●` dot.
299299
- **Focused panel**: Border changes from `border` to `border_focused` style.
300-
- **Status indicators**: Green for healthy/ready/info, yellow for degraded/provisioning/warn, red for unhealthy/error.
300+
- **Status indicators**: Green for healthy/ready/info, yellow for degraded/provisioning/starting/warn, red for unhealthy/error.
301301
- **Separators**: Muted `│` characters between title bar segments and nav bar sections.
302302
- **Log source labels**: `"sandbox"` source renders in `accent` (green), `"gateway"` in `muted`.
303303

@@ -423,7 +423,7 @@ All actions are accessible via keyboard shortcuts displayed in the nav bar. The
423423
| `crates/openshell-tui/src/ui/providers.rs` | Provider list table with profile-aware columns: Name, Category, Type, Credentials, Workspace |
424424
| `crates/openshell-tui/src/ui/global_settings.rs` | Global settings table: Key, Type, Value. Includes edit overlay, confirm-set, and confirm-delete popups |
425425
| `crates/openshell-tui/src/ui/sandboxes.rs` | Reusable sandbox table widget with columns: Name, Status, Created, Age, Image, Workspace, Notes |
426-
| `crates/openshell-tui/src/ui/sandbox_detail.rs` | Sandbox metadata view — name, status, image, created, age, providers, policy version |
426+
| `crates/openshell-tui/src/ui/sandbox_detail.rs` | Sandbox metadata view — name, status, image, created, age, restart policy/status, providers, policy version |
427427
| `crates/openshell-tui/src/ui/sandbox_policy.rs` | Policy viewer — rendered policy lines with scroll support, tab title |
428428
| `crates/openshell-tui/src/ui/sandbox_settings.rs` | Sandbox settings table: Key, Type, Value, Scope. Includes edit overlay and confirm popups |
429429
| `crates/openshell-tui/src/ui/sandbox_logs.rs` | Structured log viewer — timestamp, source, level, target, message, key=value fields, scroll position, source filter, visual selection mode, clipboard copy |

‎architecture/compute-runtimes.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,14 @@ Canonical main-process support is part of the `ComputeDriver` contract. Every
7575
in-tree and extension driver must forward the exact specification; it is not an
7676
optional capability that drivers can omit or negotiate.
7777

78+
The gateway also owns canonical main-process restart policy. Drivers disable
79+
native container, pod, or VM restart behavior and implement the existing stop
80+
and start operations. When policy selects a restart, the gateway persists the
81+
`Starting` phase and backoff deadline, stops compute, starts it again after the
82+
deadline, and waits for a new supervisor session before returning to `Ready`.
83+
Restart count, exit code, and deadline distinguish this path from an explicit
84+
start. Driver snapshots cannot override that gateway-owned transition.
85+
7886
Drivers own runtime-specific platform event interpretation. When an event should
7987
drive client provisioning UI, the driver attaches the shared
8088
`openshell.progress.*` metadata defined in `openshell-core` instead of requiring
@@ -90,6 +98,9 @@ produce the public `SandboxPhase`:
9098
backend_phase = derive_phase(driver_status)
9199
92100
public_phase =
101+
if persisted_phase == Stopped: → Stopped
102+
if persisted_phase == Stopping and backend not stopped: → Stopping
103+
if persisted_phase == Starting and backend not Ready: → Starting
93104
if backend_phase in {Error, Deleting}: → pass through (terminal precedence)
94105
if driver_reports_runtime_readiness && backend_phase == Ready: → Ready
95106
if backend_phase == Ready && session connected: → Ready
@@ -193,6 +204,14 @@ restart a completed or failed canonical process. Before an explicit restart,
193204
the gateway disconnects the prior supervisor session and deletes its SSH
194205
sessions so credentials cannot cross runtime generations.
195206

207+
When the sandbox restart policy selects replacement, the gateway instead
208+
persists `Starting` with the previous exit result, restart count, and backoff
209+
deadline. The leader claims each due attempt before stopping compute, rotates
210+
launch credentials, starts the driver with the persisted generation, and waits
211+
for a different supervisor instance. Stop and delete intent fence late driver
212+
starts. A fresh supervisor clears the deadline and returns the sandbox to
213+
`Ready`; a successful main process exit under `OnFailure` remains `Completed`.
214+
196215
`StopSandbox` and `StartSandbox` are idempotent driver operations. Stop
197216
retains the driver resource and its persistent workspace boundary while making
198217
exec, SSH, forwarding, and exposed services unavailable. Start reactivates the

‎architecture/sandbox.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -919,9 +919,16 @@ the runtime policy it would enforce.
919919
for that finalized supervisor session to disconnect before deleting an
920920
ephemeral sandbox. Exit code 0 records
921921
`Completed/MainProcessCompleted`; nonzero and signal-normalized exits record
922-
`Error/MainProcessFailed`. Infrastructure failures also use `Error`, with a
923-
distinct condition reason and no fabricated canonical-process result. Runtime
924-
restart policies must not replace the canonical process.
922+
`Error/MainProcessFailed` when the gateway policy is `Never`. `OnFailure`
923+
replaces the runtime after a nonzero result; `Always` replaces it after any
924+
result. The gateway starts the first replacement after terminal delivery, then
925+
persists a 1-second doubling backoff for repeated quick exits, capped at 3
926+
minutes. A 10-second run resets the count. A bounded fallback recovers a
927+
restart when terminal finalization is lost. Each replacement starts a new
928+
runtime generation with fresh launch credentials. Infrastructure failures also
929+
use `Error`, with a distinct condition reason and no fabricated canonical-process
930+
result.
931+
Native runtime restart policies remain disabled.
925932

926933
## Shared Boundary Primitives
927934

‎crates/openshell-cli/src/main.rs‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1535,6 +1535,14 @@ enum SandboxCommands {
15351535
#[arg(long, conflicts_with = "editor")]
15361536
detach: bool,
15371537

1538+
/// Restart behavior after the canonical main process exits.
1539+
#[arg(
1540+
long,
1541+
value_parser = ["never", "on-failure", "always"],
1542+
default_value = "never"
1543+
)]
1544+
restart_policy: String,
1545+
15381546
/// Auto-create missing providers from local credentials.
15391547
///
15401548
/// Without this flag, an interactive prompt asks per-provider;
@@ -3314,6 +3322,7 @@ async fn run_async() -> Result<()> {
33143322
tty,
33153323
no_tty,
33163324
detach,
3325+
restart_policy,
33173326
auto_providers,
33183327
no_auto_providers,
33193328
labels,
@@ -3416,6 +3425,7 @@ async fn run_async() -> Result<()> {
34163425
output: output.as_str(),
34173426
detach,
34183427
suppress_credential_warnings: no_credential_warnings,
3428+
restart_policy: &restart_policy,
34193429
},
34203430
&cli.workspace,
34213431
&tls,
@@ -6008,6 +6018,51 @@ mod tests {
60086018
}
60096019
}
60106020

6021+
#[test]
6022+
fn sandbox_create_restart_policy_defaults_to_never() {
6023+
let cli = Cli::try_parse_from(["openshell", "sandbox", "create"]).unwrap();
6024+
match cli.command {
6025+
Some(Commands::Sandbox {
6026+
command: Some(SandboxCommands::Create { restart_policy, .. }),
6027+
..
6028+
}) => assert_eq!(restart_policy, "never"),
6029+
other => panic!("expected SandboxCommands::Create, got: {other:?}"),
6030+
}
6031+
}
6032+
6033+
#[test]
6034+
fn sandbox_create_restart_policy_accepts_on_failure() {
6035+
let cli = Cli::try_parse_from([
6036+
"openshell",
6037+
"sandbox",
6038+
"create",
6039+
"--restart-policy",
6040+
"on-failure",
6041+
])
6042+
.unwrap();
6043+
match cli.command {
6044+
Some(Commands::Sandbox {
6045+
command: Some(SandboxCommands::Create { restart_policy, .. }),
6046+
..
6047+
}) => assert_eq!(restart_policy, "on-failure"),
6048+
other => panic!("expected SandboxCommands::Create, got: {other:?}"),
6049+
}
6050+
}
6051+
6052+
#[test]
6053+
fn sandbox_create_restart_policy_rejects_unknown_value() {
6054+
assert!(
6055+
Cli::try_parse_from([
6056+
"openshell",
6057+
"sandbox",
6058+
"create",
6059+
"--restart-policy",
6060+
"unless-stopped",
6061+
])
6062+
.is_err()
6063+
);
6064+
}
6065+
60116066
#[test]
60126067
fn sandbox_create_detach_parses_with_main_command() {
60136068
let cli = Cli::try_parse_from([

‎crates/openshell-cli/src/run.rs‎

Lines changed: 111 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -56,11 +56,11 @@ use openshell_core::proto::{
5656
ListSandboxPoliciesRequest, ListSandboxTemplatesRequest, ListSandboxesRequest,
5757
ListServicesRequest, PolicySource, PolicyStatus, RejectDraftChunkRequest, ResourceRequirements,
5858
RevokeSshSessionRequest, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicy,
59-
SandboxResources, SandboxServiceExposure, SandboxServiceLevel, SandboxSpec, SandboxStartup,
60-
SandboxTemplate, SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec,
61-
ServiceEndpointResponse, SettingScope, StartSandboxRequest, StopSandboxRequest,
62-
TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, WatchSandboxRequest,
63-
exec_sandbox_event, tcp_forward_init,
59+
SandboxResources, SandboxRestartPolicy, SandboxServiceExposure, SandboxServiceLevel,
60+
SandboxSpec, SandboxStartup, SandboxTemplate, SandboxWorkloadConfig, SandboxWorkloadTemplate,
61+
SandboxWorkloadTemplateSpec, ServiceEndpointResponse, SettingScope, StartSandboxRequest,
62+
StopSandboxRequest, TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest,
63+
WatchSandboxRequest, exec_sandbox_event, tcp_forward_init,
6464
};
6565
use openshell_core::settings;
6666
use openshell_core::{ObjectId, ObjectName, ObjectWorkspace};
@@ -452,6 +452,7 @@ pub struct SandboxCreateConfig<'a> {
452452
pub output: &'a str,
453453
pub detach: bool,
454454
pub suppress_credential_warnings: bool,
455+
pub restart_policy: &'a str,
455456
}
456457

457458
impl Default for SandboxCreateConfig<'_> {
@@ -480,6 +481,7 @@ impl Default for SandboxCreateConfig<'_> {
480481
output: "table",
481482
detach: false,
482483
suppress_credential_warnings: false,
484+
restart_policy: "never",
483485
}
484486
}
485487
}
@@ -516,6 +518,7 @@ pub async fn sandbox_create(
516518
output,
517519
detach,
518520
suppress_credential_warnings,
521+
restart_policy,
519522
} = config;
520523

521524
if editor.is_some() && !command.is_empty() {
@@ -670,6 +673,12 @@ pub async fn sandbox_create(
670673
template: inline_template,
671674
command: main_command,
672675
tty: main_terminal,
676+
restart_policy: match restart_policy {
677+
"never" => SandboxRestartPolicy::Never as i32,
678+
"on-failure" => SandboxRestartPolicy::OnFailure as i32,
679+
"always" => SandboxRestartPolicy::Always as i32,
680+
value => return Err(miette::miette!("invalid restart policy '{value}'")),
681+
},
673682
..SandboxSpec::default()
674683
}),
675684
name: name.unwrap_or_default().to_string(),
@@ -1697,6 +1706,41 @@ where
16971706
"Resource version:".dimmed(),
16981707
sandbox.metadata.as_ref().map_or(0, |m| m.resource_version)
16991708
);
1709+
println!(
1710+
" {} {}",
1711+
"Restart policy:".dimmed(),
1712+
sandbox
1713+
.spec
1714+
.as_ref()
1715+
.map_or("never", |spec| { restart_policy_name(spec.restart_policy) })
1716+
);
1717+
if let Some(status) = sandbox.status.as_ref() {
1718+
println!(
1719+
" {} {}",
1720+
"Main process instance:".dimmed(),
1721+
if status.main_process_instance_id.is_empty() {
1722+
"-"
1723+
} else {
1724+
&status.main_process_instance_id
1725+
}
1726+
);
1727+
println!(
1728+
" {} {}",
1729+
"Last exit code:".dimmed(),
1730+
status
1731+
.exit_code
1732+
.map_or_else(|| "-".to_string(), |code| code.to_string())
1733+
);
1734+
println!(" {} {}", "Restart count:".dimmed(), status.restart_count);
1735+
println!(
1736+
" {} {}",
1737+
"Next restart:".dimmed(),
1738+
status.next_restart_time.as_ref().map_or_else(
1739+
|| "-".to_string(),
1740+
|time| format_epoch_ms(proto_timestamp_ms(Some(time))),
1741+
)
1742+
);
1743+
}
17001744

17011745
// Display labels if present
17021746
if let Some(metadata) = &sandbox.metadata
@@ -2846,6 +2890,14 @@ fn endpoint_status_display_lines(endpoint: &EndpointStatus) -> Vec<String> {
28462890
]
28472891
}
28482892

2893+
fn restart_policy_name(policy: i32) -> &'static str {
2894+
match SandboxRestartPolicy::try_from(policy) {
2895+
Ok(SandboxRestartPolicy::OnFailure) => "on-failure",
2896+
Ok(SandboxRestartPolicy::Always) => "always",
2897+
Ok(SandboxRestartPolicy::Unspecified | SandboxRestartPolicy::Never) | Err(_) => "never",
2898+
}
2899+
}
2900+
28492901
fn sandbox_detail_to_json(
28502902
sandbox: &Sandbox,
28512903
config: &GetSandboxConfigResponse,
@@ -2855,6 +2907,33 @@ fn sandbox_detail_to_json(
28552907
.as_object_mut()
28562908
.expect("sandbox_to_json returns object");
28572909

2910+
let restart_policy = sandbox
2911+
.spec
2912+
.as_ref()
2913+
.map_or("never", |spec| restart_policy_name(spec.restart_policy));
2914+
obj.insert("restart_policy".into(), serde_json::json!(restart_policy));
2915+
if let Some(status) = sandbox.status.as_ref() {
2916+
obj.insert(
2917+
"main_process_instance_id".into(),
2918+
serde_json::json!(status.main_process_instance_id),
2919+
);
2920+
obj.insert("exit_code".into(), serde_json::json!(status.exit_code));
2921+
obj.insert(
2922+
"restart_count".into(),
2923+
serde_json::json!(status.restart_count),
2924+
);
2925+
obj.insert(
2926+
"next_restart_at_ms".into(),
2927+
serde_json::json!(proto_timestamp_ms(status.next_restart_time.as_ref())),
2928+
);
2929+
obj.insert(
2930+
"main_process_started_at_ms".into(),
2931+
serde_json::json!(proto_timestamp_ms(
2932+
status.main_process_started_time.as_ref()
2933+
)),
2934+
);
2935+
}
2936+
28582937
let policy_source = if config.policy_source == PolicySource::Global as i32 {
28592938
"global"
28602939
} else {
@@ -6440,10 +6519,10 @@ mod tests {
64406519
use openshell_core::proto::{
64416520
EndpointResult, EndpointStatus, GetSandboxConfigResponse, GpuResourceRequirements,
64426521
PolicySource, PolicyStatus, ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase,
6443-
SandboxPolicy, SandboxPolicyRevision, SandboxResources, SandboxStatus,
6444-
SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateProvenance,
6445-
SandboxWorkloadTemplateSpec, ServiceEndpoint, ServiceEndpointResponse, WorkspaceMember,
6446-
WorkspaceRole, datamodel::v1::ObjectMeta,
6522+
SandboxPolicy, SandboxPolicyRevision, SandboxResources, SandboxRestartPolicy, SandboxSpec,
6523+
SandboxStatus, SandboxWorkloadConfig, SandboxWorkloadTemplate,
6524+
SandboxWorkloadTemplateProvenance, SandboxWorkloadTemplateSpec, ServiceEndpoint,
6525+
ServiceEndpointResponse, WorkspaceMember, WorkspaceRole, datamodel::v1::ObjectMeta,
64476526
};
64486527

64496528
#[test]
@@ -7717,6 +7796,22 @@ mod tests {
77177796
};
77187797
sandbox.set_phase(SandboxPhase::Ready as i32);
77197798
sandbox.set_current_policy_version(2);
7799+
sandbox.spec = Some(SandboxSpec {
7800+
restart_policy: SandboxRestartPolicy::OnFailure as i32,
7801+
..Default::default()
7802+
});
7803+
sandbox.status = Some(SandboxStatus {
7804+
phase: SandboxPhase::Starting as i32,
7805+
main_process_instance_id: "main-2".to_string(),
7806+
exit_code: Some(9),
7807+
restart_count: 2,
7808+
next_restart_time: openshell_core::time::timestamp_from_millis(1_700_000_000_000).ok(),
7809+
main_process_started_time: openshell_core::time::timestamp_from_millis(
7810+
1_699_999_000_000,
7811+
)
7812+
.ok(),
7813+
..Default::default()
7814+
});
77207815

77217816
let config = GetSandboxConfigResponse {
77227817
policy_source: PolicySource::Global as i32,
@@ -7728,7 +7823,13 @@ mod tests {
77287823

77297824
assert_eq!(json["id"], "sb-123");
77307825
assert_eq!(json["name"], "test-sb");
7731-
assert_eq!(json["phase"], "Ready");
7826+
assert_eq!(json["phase"], "Starting");
7827+
assert_eq!(json["restart_policy"], "on-failure");
7828+
assert_eq!(json["main_process_instance_id"], "main-2");
7829+
assert_eq!(json["exit_code"], 9);
7830+
assert_eq!(json["restart_count"], 2);
7831+
assert_eq!(json["next_restart_at_ms"], 1_700_000_000_000_i64);
7832+
assert_eq!(json["main_process_started_at_ms"], 1_699_999_000_000_i64);
77327833
assert_eq!(json["policy_source"], "global");
77337834
assert_eq!(json["revision"], 3);
77347835
assert!(json["policy"].is_null());

0 commit comments

Comments
 (0)