Skip to content

Commit 1f497db

Browse files
committed
fix(sdk): bind deletion waits to sandbox identity
Track accepted deletions by original sandbox identity in Rust and TypeScript. Preserve name-only waits and cover replacement races and lookup failures. Merge the phase-one cleanup fix and adapt its interceptor regressions to typed deletion outcomes. Refs #3051. Signed-off-by: Mrunal Patel <[email protected]>
1 parent 21a8e31 commit 1f497db

10 files changed

Lines changed: 326 additions & 31 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -59,8 +59,8 @@ write conflicts attach `google.rpc.ErrorInfo` with a stable reason and current
5959
version when available. `google.rpc.RetryInfo` expresses a minimum retry delay;
6060
it does not establish that a mutation is safe to repeat. SDKs retain the original
6161
transport status, metadata, and unknown details alongside decoded fields.
62-
Python cleanup inspects the original gRPC call beneath a typed error wrapper,
63-
preserving missing-resource handling without suppressing other failures.
62+
SDK deletion waits recognize missing-resource status through typed error wrappers
63+
without suppressing other failures.
6464

6565
The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
6666
The default local single-user deployment mode is mTLS user authentication:
@@ -373,8 +373,10 @@ Public delete, membership-removal, and SSH-revocation responses use
373373
logical gateway deletion or revocation; it does not guarantee that downstream
374374
platform garbage collection has finished. Sandbox deletion returns `ACCEPTED`
375375
while its captured object ID remains in the store, and returns that ID so callers
376-
can distinguish the original sandbox from a same-name replacement. The existing
377-
owned deletion worker continues after request cancellation.
376+
can distinguish the original sandbox from a same-name replacement. Identity-aware
377+
SDK deletion waits complete on absence or a different observed ID; name-only waits
378+
continue until the name is absent. The existing owned deletion worker continues
379+
after request cancellation.
378380

379381
Missing targets return `NOT_FOUND` unless `allow_missing` explicitly requests
380382
`ALREADY_ABSENT`. Authorization, parent resolution, preconditions, and backend

‎crates/openshell-sdk/README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,22 @@ Curated calls without a workspace argument explicitly select the `default`
5959
workspace. Cross-workspace listing uses the separate `*_all_workspaces`
6060
methods and requires Platform Admin access.
6161

62+
For an accepted sandbox deletion, pass its original ID to `wait_deleted` so a
63+
same-name replacement does not extend the wait. Both the default and
64+
workspace-scoped clients accept the optional third argument; pass `None` to wait
65+
for name absence instead.
66+
67+
```rust
68+
let deletion = client.delete_sandbox(name, openshell_sdk::DeleteOptions::default()).await?;
69+
if deletion.outcome == openshell_sdk::DeletionOutcome::Accepted {
70+
client.wait_deleted(
71+
name,
72+
std::time::Duration::from_secs(60),
73+
deletion.sandbox_id.as_deref(),
74+
).await?;
75+
}
76+
```
77+
6278
Curated `list_*` methods return a lazy `Pager<T>`. Each `next_page()` call
6379
issues at most one RPC and returns a `Page<T>` with its opaque continuation
6480
token. The explicit `list_all_*` conveniences exhaust that pager; `page_size`

‎crates/openshell-sdk/src/client.rs‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -417,16 +417,26 @@ impl OpenShellClient {
417417
.await
418418
}
419419

420-
/// Poll until the sandbox is gone (gRPC `NotFound`) or the `timeout`
421-
/// elapses.
422-
pub async fn wait_deleted(&self, name: &str, timeout: Duration) -> Result<()> {
420+
/// Poll until the sandbox is gone (gRPC `NotFound`) or the timeout elapses.
421+
///
422+
/// Pass the deletion result's `sandbox_id` as `expected_sandbox_id` to also
423+
/// complete when the name resolves to a different sandbox. With `None`,
424+
/// waits for the name to be absent, including any same-name replacement.
425+
pub async fn wait_deleted(
426+
&self,
427+
name: &str,
428+
timeout: Duration,
429+
expected_sandbox_id: Option<&str>,
430+
) -> Result<()> {
423431
let deadline = Instant::now() + timeout;
424432
let mut delay = Duration::from_millis(250);
425433
loop {
426434
match self.get_sandbox(name).await {
427435
Err(SdkError::NotFound { .. }) => return Ok(()),
428436
Err(other) => return Err(other),
429-
Ok(snapshot) if snapshot.phase == SandboxPhase::Deleting => {}
437+
Ok(snapshot) if expected_sandbox_id.is_some_and(|id| snapshot.id != id) => {
438+
return Ok(());
439+
}
430440
Ok(_) => {}
431441
}
432442
if Instant::now() >= deadline {
@@ -1009,13 +1019,25 @@ impl WorkspaceScopedClient {
10091019
}
10101020

10111021
/// Poll until the sandbox is gone (`NotFound`) or the timeout elapses.
1012-
pub async fn wait_deleted(&self, name: &str, timeout: Duration) -> Result<()> {
1022+
///
1023+
/// Pass the deletion result's `sandbox_id` as `expected_sandbox_id` to also
1024+
/// complete when the name resolves to a different sandbox. With `None`,
1025+
/// waits for the name to be absent, including any same-name replacement.
1026+
pub async fn wait_deleted(
1027+
&self,
1028+
name: &str,
1029+
timeout: Duration,
1030+
expected_sandbox_id: Option<&str>,
1031+
) -> Result<()> {
10131032
let deadline = Instant::now() + timeout;
10141033
let mut delay = Duration::from_millis(250);
10151034
loop {
10161035
match self.get_sandbox(name).await {
10171036
Err(SdkError::NotFound { .. }) => return Ok(()),
10181037
Err(other) => return Err(other),
1038+
Ok(snapshot) if expected_sandbox_id.is_some_and(|id| snapshot.id != id) => {
1039+
return Ok(());
1040+
}
10191041
Ok(_) => {}
10201042
}
10211043
if Instant::now() >= deadline {

‎crates/openshell-sdk/tests/client_mock.rs‎

Lines changed: 160 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ struct MockState {
6060
last_workspace_request: Mutex<Option<String>>,
6161
get_calls: AtomicU32,
6262
phase_sequence: Vec<proto::SandboxPhase>,
63+
get_sandbox_id: Option<String>,
64+
get_error: Option<Status>,
65+
delete_response: Option<proto::DeleteSandboxResponse>,
6366
get_returns_not_found: bool,
6467
not_found_after: Option<u32>,
6568
paginate_list: bool,
@@ -344,6 +347,9 @@ impl OpenShell for TestOpenShell {
344347
selected_workspace(&req.workspace_scope).map(str::to_string);
345348
let count = self.state.get_calls.fetch_add(1, Ordering::SeqCst);
346349

350+
if let Some(error) = &self.state.get_error {
351+
return Err(error.clone());
352+
}
347353
if self.state.get_returns_not_found {
348354
return Err(Status::not_found(format!("sandbox '{name}' not found")));
349355
}
@@ -361,8 +367,12 @@ impl OpenShell for TestOpenShell {
361367
.or_else(|| self.state.phase_sequence.last().copied())
362368
.unwrap_or(proto::SandboxPhase::Ready);
363369

370+
let mut sandbox = sandbox_with_phase(&name, phase);
371+
if let Some(id) = &self.state.get_sandbox_id {
372+
sandbox.metadata.as_mut().unwrap().id.clone_from(id);
373+
}
364374
Ok(Response::new(proto::SandboxResponse {
365-
sandbox: Some(sandbox_with_phase(&name, phase)),
375+
sandbox: Some(sandbox),
366376
}))
367377
}
368378

@@ -436,6 +446,9 @@ impl OpenShell for TestOpenShell {
436446
*self.state.last_delete_name.lock().await = Some(req.name);
437447
*self.state.last_delete_workspace.lock().await =
438448
selected_workspace(&req.workspace_scope).map(str::to_string);
449+
if let Some(response) = &self.state.delete_response {
450+
return Ok(Response::new(response.clone()));
451+
}
439452
Ok(Response::new(proto::DeleteSandboxResponse {
440453
sandbox_id: String::new(),
441454
outcome: proto::DeletionOutcome::Completed.into(),
@@ -1289,19 +1302,154 @@ async fn wait_ready_surfaces_error_phase() {
12891302

12901303
#[tokio::test]
12911304
async fn wait_deleted_returns_when_get_reports_not_found() {
1292-
let state = Arc::new(MockState {
1293-
phase_sequence: vec![proto::SandboxPhase::Deleting],
1294-
not_found_after: Some(2),
1295-
..Default::default()
1296-
});
1297-
let endpoint = start_mock(state.clone()).await;
1298-
let client = connect(&endpoint).await;
1305+
for scoped in [false, true] {
1306+
for expected_id in [None, Some("id-my-box")] {
1307+
let state = Arc::new(MockState {
1308+
phase_sequence: vec![proto::SandboxPhase::Deleting],
1309+
not_found_after: Some(1),
1310+
..Default::default()
1311+
});
1312+
let endpoint = start_mock(state.clone()).await;
1313+
let client = connect(&endpoint).await;
1314+
let timeout = std::time::Duration::from_secs(5);
1315+
if scoped {
1316+
client
1317+
.workspace("team")
1318+
.wait_deleted("my-box", timeout, expected_id)
1319+
.await
1320+
} else {
1321+
client.wait_deleted("my-box", timeout, expected_id).await
1322+
}
1323+
.unwrap();
1324+
assert_eq!(state.get_calls.load(Ordering::SeqCst), 2);
1325+
}
1326+
}
1327+
}
12991328

1300-
client
1301-
.wait_deleted("my-box", std::time::Duration::from_secs(5))
1302-
.await
1329+
#[tokio::test]
1330+
async fn wait_deleted_completes_on_replacement_after_accepted_deletion() {
1331+
for scoped in [false, true] {
1332+
let state = Arc::new(MockState {
1333+
get_sandbox_id: Some("replacement-id".to_string()),
1334+
delete_response: Some(proto::DeleteSandboxResponse {
1335+
outcome: proto::DeletionOutcome::Accepted.into(),
1336+
sandbox_id: "old-id".to_string(),
1337+
}),
1338+
..Default::default()
1339+
});
1340+
let endpoint = start_mock(state.clone()).await;
1341+
let client = connect(&endpoint).await;
1342+
let timeout = std::time::Duration::from_millis(50);
1343+
let result = if scoped {
1344+
client
1345+
.workspace("team")
1346+
.delete_sandbox("my-box", openshell_sdk::DeleteOptions::default())
1347+
.await
1348+
} else {
1349+
client
1350+
.delete_sandbox("my-box", openshell_sdk::DeleteOptions::default())
1351+
.await
1352+
}
13031353
.unwrap();
1304-
assert!(state.get_calls.load(Ordering::SeqCst) >= 3);
1354+
assert_eq!(result.outcome, openshell_sdk::DeletionOutcome::Accepted);
1355+
assert_eq!(result.sandbox_id.as_deref(), Some("old-id"));
1356+
if scoped {
1357+
client
1358+
.workspace("team")
1359+
.wait_deleted("my-box", timeout, result.sandbox_id.as_deref())
1360+
.await
1361+
} else {
1362+
client
1363+
.wait_deleted("my-box", timeout, result.sandbox_id.as_deref())
1364+
.await
1365+
}
1366+
.unwrap();
1367+
assert_eq!(state.get_calls.load(Ordering::SeqCst), 1);
1368+
let workspace = if scoped { "team" } else { "default" };
1369+
assert_eq!(
1370+
state.last_get_workspace.lock().await.as_deref(),
1371+
Some(workspace)
1372+
);
1373+
assert_eq!(
1374+
state.last_delete_workspace.lock().await.as_deref(),
1375+
Some(workspace)
1376+
);
1377+
}
1378+
}
1379+
1380+
#[tokio::test]
1381+
async fn wait_deleted_does_not_complete_while_expected_identity_remains() {
1382+
for scoped in [false, true] {
1383+
let state = Arc::new(MockState {
1384+
get_sandbox_id: Some("old-id".to_string()),
1385+
..Default::default()
1386+
});
1387+
let endpoint = start_mock(state.clone()).await;
1388+
let client = connect(&endpoint).await;
1389+
let timeout = std::time::Duration::ZERO;
1390+
let error = if scoped {
1391+
client
1392+
.workspace("team")
1393+
.wait_deleted("my-box", timeout, Some("old-id"))
1394+
.await
1395+
} else {
1396+
client.wait_deleted("my-box", timeout, Some("old-id")).await
1397+
}
1398+
.unwrap_err();
1399+
assert_eq!(error.code(), "connect");
1400+
assert!(error.to_string().contains("timed out waiting"));
1401+
assert_eq!(state.get_calls.load(Ordering::SeqCst), 1);
1402+
}
1403+
}
1404+
1405+
#[tokio::test]
1406+
async fn wait_deleted_without_identity_waits_for_replacement_to_disappear() {
1407+
for scoped in [false, true] {
1408+
let state = Arc::new(MockState {
1409+
get_sandbox_id: Some("replacement-id".to_string()),
1410+
not_found_after: Some(1),
1411+
..Default::default()
1412+
});
1413+
let endpoint = start_mock(state.clone()).await;
1414+
let client = connect(&endpoint).await;
1415+
let timeout = std::time::Duration::from_secs(5);
1416+
if scoped {
1417+
client
1418+
.workspace("team")
1419+
.wait_deleted("my-box", timeout, None)
1420+
.await
1421+
} else {
1422+
client.wait_deleted("my-box", timeout, None).await
1423+
}
1424+
.unwrap();
1425+
assert_eq!(state.get_calls.load(Ordering::SeqCst), 2);
1426+
}
1427+
}
1428+
1429+
#[tokio::test]
1430+
async fn wait_deleted_propagates_errors_other_than_not_found() {
1431+
for scoped in [false, true] {
1432+
for code in [tonic::Code::PermissionDenied, tonic::Code::Unavailable] {
1433+
let state = Arc::new(MockState {
1434+
get_error: Some(Status::new(code, "lookup failed")),
1435+
..Default::default()
1436+
});
1437+
let endpoint = start_mock(state.clone()).await;
1438+
let client = connect(&endpoint).await;
1439+
let timeout = std::time::Duration::from_secs(5);
1440+
let error = if scoped {
1441+
client
1442+
.workspace("team")
1443+
.wait_deleted("my-box", timeout, Some("old-id"))
1444+
.await
1445+
} else {
1446+
client.wait_deleted("my-box", timeout, Some("old-id")).await
1447+
}
1448+
.unwrap_err();
1449+
assert_eq!(error.grpc_status().unwrap().code(), code);
1450+
assert_eq!(state.get_calls.load(Ordering::SeqCst), 1);
1451+
}
1452+
}
13051453
}
13061454

13071455
#[tokio::test]

‎docs/reference/api-errors.mdx‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,9 +76,10 @@ Existing error codes and human-readable messages remain available. Python curate
7676
clients now raise `GatewayError`, which remains a `grpc.RpcError`; existing
7777
`except grpc.RpcError` handlers continue to work. `GatewayError` is not a
7878
`grpc.Call`. If your handler checks that interface, inspect `raw_error` before
79-
checking the status. The Python SDK's deletion wait and managed-sandbox cleanup
80-
use the original call to recognize `NOT_FOUND`; other failures still propagate.
81-
Rust error variants now retain
79+
checking the status, or call `GatewayError.code()` directly. Python deletion waits
80+
recognize `NOT_FOUND` through the wrapper; managed-sandbox cleanup requests
81+
`allow_missing=True` and handles the typed deletion outcome. Other failures still
82+
propagate. Rust error variants now retain
8283
status fields; use `..` when destructuring variants that do not need those fields.
8384
Go and TypeScript add typed detail fields to their existing error types.
8485

@@ -107,6 +108,19 @@ For an accepted deletion, poll or watch that identity. A same-name sandbox with
107108
different ID is not the sandbox being deleted. Cancellation or a disconnected
108109
client does not stop the gateway's already-started deletion worker.
109110

111+
Pass the deletion result's ID to the SDK wait helper to avoid following a
112+
same-name replacement.
113+
114+
| SDK | Identity-aware deletion wait |
115+
|---|---|
116+
| Rust | `client.wait_deleted(name, timeout, result.sandbox_id.as_deref()).await?` on either the default or workspace-scoped client. |
117+
| Python | `client.wait_deleted(name, workspace=workspace, expected_sandbox_id=result.sandbox_id)`. Managed-sandbox cleanup supplies this automatically. |
118+
| TypeScript | `client.sandbox.waitDeleted(name, timeoutSecs, { workspace, expectedSandboxId: result.sandboxId })`. |
119+
120+
These waits complete on `NOT_FOUND` or a different observed ID. Without an
121+
expected ID, they wait for the name to be absent. Existing Rust callers must add
122+
the third argument; use `None` to retain name-only behavior.
123+
110124
`allow_missing` is not request deduplication. A later request by name can delete
111125
a newly created resource with that name. Do not blindly repeat a timed-out
112126
deletion when names might be reused.

‎python/openshell/sandbox_cleanup_test.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,11 @@ def delete(request, context):
4545
state.calls.append("DeleteSandbox")
4646
assert request.name == "cleanup-test"
4747
assert request.workspace_scope.workspace == "default"
48+
assert request.allow_missing
49+
if state.code == grpc.StatusCode.NOT_FOUND:
50+
return openshell_pb2.DeleteSandboxResponse(
51+
outcome=openshell_pb2.DELETION_OUTCOME_ALREADY_ABSENT
52+
)
4853
fail(context)
4954

5055
server = grpc.server(futures.ThreadPoolExecutor(max_workers=1))
@@ -125,7 +130,9 @@ def test_wait_deleted_handles_intercepted_status(cleanup_client, code):
125130
grpc.StatusCode.UNAVAILABLE,
126131
],
127132
)
128-
def test_context_cleanup_handles_intercepted_status(cleanup_client, monkeypatch, code):
133+
def test_context_cleanup_handles_absence_and_intercepted_errors(
134+
cleanup_client, monkeypatch, code
135+
):
129136
client, state = cleanup_client
130137
state.code = code
131138
state.exists = True

‎sdk/typescript/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@ console.log(result.stdout.toString())
4343

4444
const deletion = await client.sandbox.delete(sandbox.name)
4545
console.log(deletion.outcome) // completed, accepted, or already_absent
46+
if (deletion.outcome === 'accepted') {
47+
await client.sandbox.waitDeleted(sandbox.name, 60, {
48+
expectedSandboxId: deletion.sandboxId,
49+
})
50+
}
4651
```
4752

4853
Deletion returns a typed result, not a boolean. `accepted` means cleanup is
@@ -52,6 +57,11 @@ you pass `{ allowMissing: true }`; this does not suppress missing parents or mak
5257
retries safe when names are reused. Upgrade gateway and SDK together for this
5358
pre-1.0 API change.
5459

60+
`waitDeleted` with `expectedSandboxId` completes when the name is absent or
61+
resolves to a different ID. Without that option, it waits for name absence,
62+
including any same-name replacement. Pass the same `workspace` to deletion and
63+
its wait when using a non-default workspace.
64+
5565
`connect()` constructs a lazy client; call `health()` when startup must verify
5666
gateway reachability. Static `oidcToken` and `edgeToken` values remain fixed for
5767
the client's lifetime. For long-running service automation, use the renewable

0 commit comments

Comments
 (0)