Skip to content

Commit c502be9

Browse files
authored
feat(api): return typed deletion outcomes with explicit missing-target semantics (NVIDIA#3317)
* feat(api): return typed deletion outcomes with explicit missing-target semantics Implement phase 2 of NVIDIA#3051 across the public gateway API and first-party SDKs. Preserve asynchronous sandbox deletion and observed resource identities, reserve legacy wire fields, and document the coordinated migration. Signed-off-by: Mrunal Patel <[email protected]> * fix(sdk): bind deletion waits to sandbox identity Track accepted deletions by original sandbox identity in Rust and TypeScript. Preserve name-only waits and cover replacement races and lookup failures. Merge the phase-one cleanup fix and adapt its interceptor regressions to typed deletion outcomes. Refs NVIDIA#3051. Signed-off-by: Mrunal Patel <[email protected]> * test(e2e): accept asynchronous stopped sandbox deletion Allow either completed or accepted deletion output, then continue polling for actual sandbox absence. This preserves the lifecycle assertion under the phase-two deletion contract. Refs NVIDIA#3051. Signed-off-by: Mrunal Patel <[email protected]> --------- Signed-off-by: Mrunal Patel <[email protected]>
1 parent d68b706 commit c502be9

93 files changed

Lines changed: 2863 additions & 1200 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/tui-development/SKILL.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -484,8 +484,12 @@ use openshell_core::proto::{
484484
let req = openshell_core::proto::DeleteSandboxRequest {
485485
name: sandbox_name,
486486
workspace_scope: Some(workspace_selector(workspace)),
487+
allow_missing: true,
487488
};
488489
```
490+
- Delete responses carry `DeletionOutcome`: distinguish `Accepted` (cleanup
491+
pending), `Completed`, and `AlreadyAbsent`. Treat unspecified or unknown
492+
outcomes as unconfirmed, not completed.
489493
- `WatchSandboxRequest` has extra fields beyond what you might need — always use `..Default::default()`:
490494
```rust
491495
let req = openshell_core::proto::WatchSandboxRequest {

‎architecture/gateway.md‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,8 +59,8 @@ write conflicts attach `google.rpc.ErrorInfo` with a stable reason and current
5959
version when available. `google.rpc.RetryInfo` expresses a minimum retry delay;
6060
it does not establish that a mutation is safe to repeat. SDKs retain the original
6161
transport status, metadata, and unknown details alongside decoded fields.
62-
Python cleanup inspects the original gRPC call beneath a typed error wrapper,
63-
preserving missing-resource handling without suppressing other failures.
62+
SDK deletion waits recognize missing-resource status through typed error wrappers
63+
without suppressing other failures.
6464

6565
The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
6666
The default local single-user deployment mode is mTLS user authentication:
@@ -376,6 +376,23 @@ than extending the frozen message.
376376

377377
The descriptor-derived test inventories the complete message and enum closure of the encoded durable roots and its intersection with the public RPC closure. The tables here record the reviewed roots and classifications.
378378

379+
Public delete, membership-removal, and SSH-revocation responses use
380+
`DeletionOutcome`, not a transport-success boolean. `COMPLETED` establishes
381+
logical gateway deletion or revocation; it does not guarantee that downstream
382+
platform garbage collection has finished. Sandbox deletion returns `ACCEPTED`
383+
while its captured object ID remains in the store, and returns that ID so callers
384+
can distinguish the original sandbox from a same-name replacement. Identity-aware
385+
SDK deletion waits complete on absence or a different observed ID; name-only waits
386+
continue until the name is absent. The existing owned deletion worker continues
387+
after request cancellation.
388+
389+
Missing targets return `NOT_FOUND` unless `allow_missing` explicitly requests
390+
`ALREADY_ABSENT`. Authorization, parent resolution, preconditions, and backend
391+
failures remain errors. Already-revoked sessions complete without another write
392+
after current authorization. The removed response booleans are reserved by name
393+
and number; this coordinated pre-1.0 API change does not alter durable schemas.
394+
It does not add request deduplication or identity preconditions for later retries.
395+
379396
| Dual-purpose encoded root | Current decision |
380397
|---|---|
381398
| `Sandbox` | Defer a storage twin; govern its complete dependency closure as durable. |

‎crates/openshell-cli/src/commands/provider.rs‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -676,6 +676,7 @@ async fn rollback_provider_create_after_gcloud_adc_failure(
676676
) -> Result<()> {
677677
match client
678678
.delete_provider(DeleteProviderRequest {
679+
allow_missing: true,
679680
name: provider_name.to_string(),
680681
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
681682
})
@@ -1713,6 +1714,7 @@ pub async fn provider_profile_delete(
17131714
for id in ids {
17141715
let response = match client
17151716
.delete_provider_profile(DeleteProviderProfileRequest {
1717+
allow_missing: true,
17161718
id: id.clone(),
17171719
workspace: workspace.to_string(),
17181720
})
@@ -1728,7 +1730,7 @@ pub async fn provider_profile_delete(
17281730
continue;
17291731
}
17301732
};
1731-
if response.deleted {
1733+
if crate::run::deletion_completed(response.outcome)? {
17321734
println!("{} Deleted provider profile {id}", "✓".green().bold());
17331735
} else {
17341736
println!("{} Provider profile {id} not found", "!".yellow());
@@ -1898,6 +1900,7 @@ pub async fn provider_refresh_delete(
18981900
let mut client = grpc_client(server, tls).await?;
18991901
let response = client
19001902
.delete_provider_refresh(DeleteProviderRefreshRequest {
1903+
allow_missing: true,
19011904
provider: name.to_string(),
19021905
credential_key: credential_key.to_string(),
19031906
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -1906,7 +1909,7 @@ pub async fn provider_refresh_delete(
19061909
.into_diagnostic()?
19071910
.into_inner();
19081911

1909-
if response.deleted {
1912+
if crate::run::deletion_completed(response.outcome)? {
19101913
println!(
19111914
"{} Deleted refresh config for {} {}",
19121915
"✓".green().bold(),
@@ -2396,6 +2399,7 @@ pub async fn provider_delete(
23962399
for name in names {
23972400
let response = match client
23982401
.delete_provider(DeleteProviderRequest {
2402+
allow_missing: true,
23992403
name: name.clone(),
24002404
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
24012405
})
@@ -2411,7 +2415,7 @@ pub async fn provider_delete(
24112415
continue;
24122416
}
24132417
};
2414-
if response.into_inner().deleted {
2418+
if crate::run::deletion_completed(response.into_inner().outcome)? {
24152419
println!("{} Deleted provider {name}", "✓".green().bold());
24162420
} else {
24172421
println!("{} Provider {name} not found", "!".yellow());

‎crates/openshell-cli/src/run.rs‎

Lines changed: 52 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -46,15 +46,16 @@ use openshell_core::proto::{
4646
ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, BeginRootfsTarStagingRequest,
4747
ClearDraftChunksRequest, CreateSandboxRequest, CreateSandboxTemplateRequest,
4848
CreateSshSessionRequest, DeleteSandboxRequest, DeleteSandboxTemplateRequest,
49-
DeleteServiceRequest, EndpointResult, EndpointStatus, ExecSandboxRequest, ExposeServiceRequest,
50-
GetCurrentUserRequest, GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest,
51-
GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxLogsRequest,
52-
GetSandboxPolicyStatusRequest, GetSandboxRequest, GetSandboxTemplateRequest, GetServiceRequest,
53-
GpuResourceRequirements, ListSandboxPoliciesRequest, ListSandboxTemplatesRequest,
54-
ListSandboxesRequest, ListServicesRequest, PolicySource, PolicyStatus, RejectDraftChunkRequest,
55-
ResourceRequirements, RevokeSshSessionRequest, Sandbox, SandboxCondition, SandboxPhase,
56-
SandboxPolicy, SandboxResources, SandboxServiceLevel, SandboxSpec, SandboxStartup,
57-
SandboxTemplate, SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec,
49+
DeleteServiceRequest, DeletionOutcome, EndpointResult, EndpointStatus, ExecSandboxRequest,
50+
ExposeServiceRequest, GetCurrentUserRequest, GetDraftHistoryRequest, GetDraftPolicyRequest,
51+
GetGatewayConfigRequest, GetSandboxConfigRequest, GetSandboxConfigResponse,
52+
GetSandboxLogsRequest, GetSandboxPolicyStatusRequest, GetSandboxRequest,
53+
GetSandboxTemplateRequest, GetServiceRequest, GpuResourceRequirements,
54+
ListSandboxPoliciesRequest, ListSandboxTemplatesRequest, ListSandboxesRequest,
55+
ListServicesRequest, PolicySource, PolicyStatus, RejectDraftChunkRequest, ResourceRequirements,
56+
RevokeSshSessionRequest, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicy,
57+
SandboxResources, SandboxServiceLevel, SandboxSpec, SandboxStartup, SandboxTemplate,
58+
SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec,
5859
ServiceEndpointResponse, SettingScope, StartSandboxRequest, StopSandboxRequest,
5960
TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, WatchSandboxRequest,
6061
exec_sandbox_event, tcp_forward_init,
@@ -1940,7 +1941,7 @@ pub async fn service_forward_tcp(
19401941
}
19411942
}
19421943
let _ = client
1943-
.revoke_ssh_session(RevokeSshSessionRequest { token })
1944+
.revoke_ssh_session(RevokeSshSessionRequest { allow_missing: true, token })
19441945
.await;
19451946
});
19461947
}
@@ -2850,6 +2851,17 @@ pub async fn sandbox_template_list(
28502851
Ok(())
28512852
}
28522853

2854+
pub(crate) fn deletion_completed(outcome: i32) -> Result<bool> {
2855+
use openshell_core::proto::DeletionOutcome;
2856+
match DeletionOutcome::try_from(outcome) {
2857+
Ok(DeletionOutcome::Completed) => Ok(true),
2858+
Ok(DeletionOutcome::AlreadyAbsent) => Ok(false),
2859+
_ => Err(miette!(
2860+
"gateway returned an unsupported deletion outcome: {outcome}"
2861+
)),
2862+
}
2863+
}
2864+
28532865
pub async fn sandbox_template_delete(
28542866
server: &str,
28552867
names: &[String],
@@ -2860,12 +2872,13 @@ pub async fn sandbox_template_delete(
28602872
for name in names {
28612873
let response = client
28622874
.delete_sandbox_template(DeleteSandboxTemplateRequest {
2875+
allow_missing: true,
28632876
name: name.clone(),
28642877
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
28652878
})
28662879
.await
28672880
.into_diagnostic()?;
2868-
if response.into_inner().deleted {
2881+
if deletion_completed(response.into_inner().outcome)? {
28692882
println!("{} Deleted sandbox template {name}", "✓".green().bold());
28702883
} else {
28712884
println!("Sandbox template {name} not found.");
@@ -3287,17 +3300,13 @@ pub async fn sandbox_delete(
32873300

32883301
let response = match client
32893302
.delete_sandbox(DeleteSandboxRequest {
3303+
allow_missing: true,
32903304
name: name.clone(),
32913305
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
32923306
})
32933307
.await
32943308
{
32953309
Ok(response) => response,
3296-
Err(status) if status.code() == Code::NotFound => {
3297-
clear_last_sandbox_if_matches(gateway, workspace, name);
3298-
println!("{} Sandbox {name} already deleted", "✓".green().bold());
3299-
continue;
3300-
}
33013310
Err(status) => {
33023311
eprintln!(
33033312
"{} Failed to delete sandbox {name}: {status}",
@@ -3308,13 +3317,25 @@ pub async fn sandbox_delete(
33083317
}
33093318
};
33103319

3311-
let deleted = response.into_inner().deleted;
3312-
if deleted {
3313-
clear_last_sandbox_if_matches(gateway, workspace, name);
3314-
println!("{} Deleted sandbox {name}", "✓".green().bold());
3315-
} else {
3316-
println!("{} Sandbox {name} not found", "!".yellow());
3320+
match response.into_inner().outcome() {
3321+
DeletionOutcome::Completed => println!("{} Deleted sandbox {name}", "✓".green().bold()),
3322+
DeletionOutcome::Accepted => println!(
3323+
"{} Sandbox {name} deletion accepted; cleanup is pending",
3324+
"✓".green().bold()
3325+
),
3326+
DeletionOutcome::AlreadyAbsent => {
3327+
println!("{} Sandbox {name} already deleted", "✓".green().bold());
3328+
}
3329+
DeletionOutcome::Unspecified => {
3330+
eprintln!(
3331+
"{} Unsupported deletion outcome for sandbox {name}",
3332+
"!".red().bold()
3333+
);
3334+
failures.push(name.clone());
3335+
continue;
3336+
}
33173337
}
3338+
clear_last_sandbox_if_matches(gateway, workspace, name);
33183339
}
33193340

33203341
aggregate_delete_failures("sandbox", &failures)
@@ -3586,6 +3607,7 @@ pub async fn service_delete(
35863607
let mut client = grpc_client(server, tls).await?;
35873608
let response = client
35883609
.delete_service(DeleteServiceRequest {
3610+
allow_missing: false,
35893611
sandbox: sandbox.to_string(),
35903612
service: service.to_string(),
35913613
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -3594,7 +3616,7 @@ pub async fn service_delete(
35943616
.map_err(|status| service_status_error("delete service", "sandbox:write", status))?
35953617
.into_inner();
35963618

3597-
if !response.deleted {
3619+
if !deletion_completed(response.outcome)? {
35983620
return Err(miette!("delete service failed: service endpoint not found"));
35993621
}
36003622

@@ -3958,10 +3980,13 @@ pub async fn workspace_delete(server: &str, names: &[String], tls: &TlsOptions)
39583980
let mut client = grpc_client(server, tls).await?;
39593981
for name in names {
39603982
let response = client
3961-
.delete_workspace(DeleteWorkspaceRequest { name: name.clone() })
3983+
.delete_workspace(DeleteWorkspaceRequest {
3984+
allow_missing: false,
3985+
name: name.clone(),
3986+
})
39623987
.await
39633988
.into_diagnostic()?;
3964-
if response.into_inner().deleted {
3989+
if deletion_completed(response.into_inner().outcome)? {
39653990
println!("{} Deleted workspace {name}", "✓".green().bold());
39663991
} else {
39673992
println!("{} Workspace {name} not found", "!".yellow());
@@ -4027,13 +4052,14 @@ pub async fn workspace_member_remove(
40274052
let mut client = grpc_client(server, tls).await?;
40284053
let response = client
40294054
.remove_workspace_member(RemoveWorkspaceMemberRequest {
4055+
allow_missing: true,
40304056
workspace: workspace.to_string(),
40314057
principal_subject: subject.to_string(),
40324058
})
40334059
.await
40344060
.into_diagnostic()?;
40354061

4036-
if response.into_inner().removed {
4062+
if deletion_completed(response.into_inner().outcome)? {
40374063
println!(
40384064
"{} Removed {} from workspace {}",
40394065
"✓".green().bold(),

‎crates/openshell-cli/tests/ensure_providers_integration.rs‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,10 @@ impl OpenShell for TestOpenShell {
195195
&self,
196196
_request: tonic::Request<DeleteSandboxRequest>,
197197
) -> Result<Response<DeleteSandboxResponse>, Status> {
198-
Ok(Response::new(DeleteSandboxResponse { deleted: true }))
198+
Ok(Response::new(DeleteSandboxResponse {
199+
sandbox_id: String::new(),
200+
outcome: openshell_core::proto::DeletionOutcome::Completed.into(),
201+
}))
199202
}
200203

201204
async fn get_sandbox_config(
@@ -496,7 +499,13 @@ impl OpenShell for TestOpenShell {
496499
) -> Result<Response<DeleteProviderResponse>, Status> {
497500
let name = request.into_inner().name;
498501
let deleted = self.state.providers.lock().await.remove(&name).is_some();
499-
Ok(Response::new(DeleteProviderResponse { deleted }))
502+
Ok(Response::new(DeleteProviderResponse {
503+
outcome: if deleted {
504+
openshell_core::proto::DeletionOutcome::Completed.into()
505+
} else {
506+
openshell_core::proto::DeletionOutcome::AlreadyAbsent.into()
507+
},
508+
}))
500509
}
501510

502511
type WatchSandboxStream =

‎crates/openshell-cli/tests/mtls_integration.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,10 @@ impl OpenShell for TestOpenShell {
161161
_request: tonic::Request<openshell_core::proto::DeleteSandboxRequest>,
162162
) -> Result<Response<openshell_core::proto::DeleteSandboxResponse>, Status> {
163163
Ok(Response::new(
164-
openshell_core::proto::DeleteSandboxResponse { deleted: true },
164+
openshell_core::proto::DeleteSandboxResponse {
165+
sandbox_id: String::new(),
166+
outcome: openshell_core::proto::DeletionOutcome::Completed.into(),
167+
},
165168
))
166169
}
167170

‎crates/openshell-cli/tests/provider_commands_integration.rs‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -331,7 +331,10 @@ impl OpenShell for TestOpenShell {
331331
&self,
332332
_request: tonic::Request<DeleteSandboxRequest>,
333333
) -> Result<Response<DeleteSandboxResponse>, Status> {
334-
Ok(Response::new(DeleteSandboxResponse { deleted: true }))
334+
Ok(Response::new(DeleteSandboxResponse {
335+
sandbox_id: String::new(),
336+
outcome: openshell_core::proto::DeletionOutcome::Completed.into(),
337+
}))
335338
}
336339

337340
async fn get_sandbox_config(
@@ -887,7 +890,13 @@ impl OpenShell for TestOpenShell {
887890
.await
888891
.remove(&(request.provider, request.credential_key))
889892
.is_some();
890-
Ok(Response::new(DeleteProviderRefreshResponse { deleted }))
893+
Ok(Response::new(DeleteProviderRefreshResponse {
894+
outcome: if deleted {
895+
openshell_core::proto::DeletionOutcome::Completed.into()
896+
} else {
897+
openshell_core::proto::DeletionOutcome::AlreadyAbsent.into()
898+
},
899+
}))
891900
}
892901

893902
async fn delete_provider(
@@ -905,7 +914,13 @@ impl OpenShell for TestOpenShell {
905914
return Err(Status::internal(message));
906915
}
907916
let deleted = self.state.providers.lock().await.remove(&name).is_some();
908-
Ok(Response::new(DeleteProviderResponse { deleted }))
917+
Ok(Response::new(DeleteProviderResponse {
918+
outcome: if deleted {
919+
openshell_core::proto::DeletionOutcome::Completed.into()
920+
} else {
921+
openshell_core::proto::DeletionOutcome::AlreadyAbsent.into()
922+
},
923+
}))
909924
}
910925

911926
async fn delete_provider_profile(
@@ -929,7 +944,13 @@ impl OpenShell for TestOpenShell {
929944
}
930945
let deleted = self.state.profiles.lock().await.remove(&id).is_some();
931946
Ok(Response::new(
932-
openshell_core::proto::DeleteProviderProfileResponse { deleted },
947+
openshell_core::proto::DeleteProviderProfileResponse {
948+
outcome: if deleted {
949+
openshell_core::proto::DeletionOutcome::Completed.into()
950+
} else {
951+
openshell_core::proto::DeletionOutcome::AlreadyAbsent.into()
952+
},
953+
},
933954
))
934955
}
935956

0 commit comments

Comments
 (0)