Skip to content

Commit a316fd7

Browse files
authored
feat(api): add durable workspace mutation admission and replay (NVIDIA#3321)
* feat(api): add durable workspace mutation admission and replay Part of NVIDIA#3051 (phase 3a). Preserve unresolved admissions, reauthorize replay, and protect same-name replacements. Signed-off-by: Mrunal Patel <[email protected]> * feat(api): extend mutation replay through gateway interceptors (NVIDIA#3323) Add typed durable replay receipts for 24 ordinary unary mutations, protect sensitive payload fingerprints, and revalidate intercepted retries without repeating post-commit observation. Part of NVIDIA#3051 (phase 3b). Signed-off-by: Mrunal Patel <[email protected]> * fix(api): scope workspace request IDs by target Include the requested workspace name in create/delete admission keys while leaving workspace UUID guards unset. Cover cross-target UUID reuse, replay, and missing targets with server and live gateway regressions. Merge the latest phase-two SDK fixes and preserve the approved interceptor replay changes. Refs NVIDIA#3051. Signed-off-by: Mrunal Patel <[email protected]> --------- Signed-off-by: Mrunal Patel <[email protected]>
1 parent 0a0a563 commit a316fd7

25 files changed

Lines changed: 5016 additions & 168 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,37 @@ transport status, metadata, and unknown details alongside decoded fields.
6262
SDK deletion waits recognize missing-resource status through typed error wrappers
6363
without suppressing other failures.
6464

65+
Ordinary user-callable unary mutations explicitly opt into durable request
66+
admission when the client supplies a UUID. Typed adapters
67+
check current authorization before looking up a caller/method/workspace-scoped
68+
key. The payload fingerprint excludes that UUID and canonicalizes protobuf maps.
69+
An atomic, quota-checked insert chooses one executor; owned execution survives
70+
client cancellation. Success is persisted before acknowledgment. Errors or
71+
interruption leave permanent unresolved claims, never stealable leases.
72+
73+
Admission rows live outside user workspace namespaces and are bounded per caller.
74+
Successes expire after 24 hours; cleanup uses the unique admission incarnation
75+
and version so an old cleaner cannot delete a new attempt. Replay stores only
76+
resource references and reviewed public scalar/diagnostic receipts, never
77+
credential-bearing response snapshots. It checks original identities and current
78+
authorization and never substitutes a same-name resource. Sandbox responses are
79+
live projections of the original UUID; normal status reconciliation does not
80+
invalidate replay. Refresh status additionally requires the original grant epoch
81+
and no deletion timestamp, including a timestamp at the Unix epoch.
82+
Other resource projections retain exact-version guards. Terminal delete receipts
83+
do not require the deleted target or parent to remain present.
84+
85+
Sandbox, service, provider/profile, and policy/config adapters use keyed payload
86+
fingerprints derived from existing gateway JWT or primary TLS private material.
87+
Replicas must share that material; missing keys or key changes fail closed without
88+
changing admission identity. Workspace/template adapters retain their original
89+
format. Intercepted requests carry the original decoded payload only in a private
90+
in-memory extension. Replay reauthorizes original and current effective scopes,
91+
requires the same effective payload, and reruns current interceptor validation.
92+
Interceptors cannot mutate the request UUID. Server-marked replay suppresses
93+
post-commit observation, which remains best-effort rather than an outbox.
94+
Credential capabilities and streaming execution require separate contracts.
95+
6596
The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
6697
The default local single-user deployment mode is mTLS user authentication:
6798
clients present a certificate signed by the local deployment CA, and the
@@ -391,7 +422,8 @@ Missing targets return `NOT_FOUND` unless `allow_missing` explicitly requests
391422
failures remain errors. Already-revoked sessions complete without another write
392423
after current authorization. The removed response booleans are reserved by name
393424
and number; this coordinated pre-1.0 API change does not alter durable schemas.
394-
It does not add request deduplication or identity preconditions for later retries.
425+
The outcome alone does not provide request deduplication. Opted-in unary methods
426+
require a request UUID for the admission contract.
395427

396428
| Dual-purpose encoded root | Current decision |
397429
|---|---|
@@ -442,6 +474,14 @@ populate `scope`, `version`, `status`, `dedup_key`, and `hit_count` so the
442474
gateway can efficiently fetch the latest policy, track load status, and manage
443475
advisor drafts without creating resource-specific tables.
444476

477+
Mutation admission uses a private, version-tagged JSON envelope in the same
478+
object store. Its identity namespace stays stable across format changes, and an
479+
unknown format fails closed. It contains explicit typed receipts, not arbitrary
480+
public response payloads, and is not part of the protobuf storage closure.
481+
Workspace create/delete admissions include the requested workspace name in the
482+
key, but omit a workspace UUID guard. Different names have independent request-ID
483+
namespaces; deletion receipts remain replayable after the target disappears.
484+
445485
Each sandbox policy revision stores the complete provenance annotation map
446486
supplied with that update. The revision payload is the authoritative immutable
447487
record; sandbox metadata receives the same annotations only as a convenience

‎crates/openshell-cli/src/commands/provider.rs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ pub async fn sandbox_provider_attach(
109109

110110
let response = match client
111111
.attach_sandbox_provider(AttachSandboxProviderRequest {
112+
request_id: String::new(),
112113
sandbox_name: name.to_string(),
113114
provider_name: provider.to_string(),
114115
expected_resource_version: resource_version,
@@ -165,6 +166,7 @@ pub async fn sandbox_provider_detach(
165166

166167
let response = match client
167168
.detach_sandbox_provider(DetachSandboxProviderRequest {
169+
request_id: String::new(),
168170
sandbox_name: name.to_string(),
169171
provider_name: provider.to_string(),
170172
expected_resource_version: resource_version,
@@ -477,6 +479,7 @@ async fn auto_create_provider(
477479
if let Some(exact_name) = preferred_name {
478480
// Explicit name: create with exactly that name, no retries.
479481
let request = CreateProviderRequest {
482+
request_id: String::new(),
480483
provider: Some(Provider {
481484
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
482485
id: String::new(),
@@ -525,6 +528,7 @@ async fn auto_create_provider(
525528
};
526529

527530
let request = CreateProviderRequest {
531+
request_id: String::new(),
528532
provider: Some(Provider {
529533
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
530534
id: String::new(),
@@ -676,6 +680,7 @@ async fn rollback_provider_create_after_gcloud_adc_failure(
676680
) -> Result<()> {
677681
match client
678682
.delete_provider(DeleteProviderRequest {
683+
request_id: String::new(),
679684
allow_missing: true,
680685
name: provider_name.to_string(),
681686
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -1107,6 +1112,7 @@ pub async fn provider_create_with_options(options: ProviderCreateOptions<'_>) ->
11071112

11081113
let response = client
11091114
.create_provider(CreateProviderRequest {
1115+
request_id: String::new(),
11101116
provider: Some(Provider {
11111117
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
11121118
id: String::new(),
@@ -1153,6 +1159,7 @@ pub async fn provider_create_with_options(options: ProviderCreateOptions<'_>) ->
11531159

11541160
if let Err(configure_err) = client
11551161
.configure_provider_refresh(ConfigureProviderRefreshRequest {
1162+
request_id: String::new(),
11561163
provider: provider_name.clone(),
11571164
credential_key: adc_credential_key.clone(),
11581165
strategy: ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32,
@@ -1178,6 +1185,7 @@ pub async fn provider_create_with_options(options: ProviderCreateOptions<'_>) ->
11781185

11791186
if let Err(rotate_err) = client
11801187
.rotate_provider_credential(RotateProviderCredentialRequest {
1188+
request_id: String::new(),
11811189
provider: provider_name.clone(),
11821190
credential_key: adc_credential_key,
11831191
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -1601,6 +1609,7 @@ pub async fn provider_profile_import(
16011609
if !items.is_empty() {
16021610
let response = client
16031611
.import_provider_profiles(ImportProviderProfilesRequest {
1612+
request_id: String::new(),
16041613
profiles: items,
16051614
workspace: workspace.to_string(),
16061615
})
@@ -1650,6 +1659,7 @@ pub async fn provider_profile_update(
16501659
.map_or(0, |profile| profile.resource_version);
16511660
let response = client
16521661
.update_provider_profiles(UpdateProviderProfilesRequest {
1662+
request_id: String::new(),
16531663
profile: Some(item),
16541664
expected_resource_version,
16551665
id: id.to_string(),
@@ -1714,6 +1724,7 @@ pub async fn provider_profile_delete(
17141724
for id in ids {
17151725
let response = match client
17161726
.delete_provider_profile(DeleteProviderProfileRequest {
1727+
request_id: String::new(),
17171728
allow_missing: true,
17181729
id: id.clone(),
17191730
workspace: workspace.to_string(),
@@ -1826,6 +1837,7 @@ pub async fn provider_refresh_config(
18261837
let mut client = grpc_client(server, tls).await?;
18271838
let status = client
18281839
.configure_provider_refresh(ConfigureProviderRefreshRequest {
1840+
request_id: String::new(),
18291841
provider: input.name.to_string(),
18301842
credential_key: input.credential_key.to_string(),
18311843
strategy: strategy as i32,
@@ -1863,6 +1875,7 @@ pub async fn provider_rotate(
18631875
let mut client = grpc_client(server, tls).await?;
18641876
let status = client
18651877
.rotate_provider_credential(RotateProviderCredentialRequest {
1878+
request_id: String::new(),
18661879
provider: name.to_string(),
18671880
credential_key: credential_key.to_string(),
18681881
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -1900,6 +1913,7 @@ pub async fn provider_refresh_delete(
19001913
let mut client = grpc_client(server, tls).await?;
19011914
let response = client
19021915
.delete_provider_refresh(DeleteProviderRefreshRequest {
1916+
request_id: String::new(),
19031917
allow_missing: true,
19041918
provider: name.to_string(),
19051919
credential_key: credential_key.to_string(),
@@ -2337,6 +2351,7 @@ pub async fn provider_update(options: ProviderUpdateOptions<'_>) -> Result<()> {
23372351

23382352
let response = client
23392353
.update_provider(UpdateProviderRequest {
2354+
request_id: String::new(),
23402355
provider: Some(Provider {
23412356
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
23422357
id: String::new(),
@@ -2399,6 +2414,7 @@ pub async fn provider_delete(
23992414
for name in names {
24002415
let response = match client
24012416
.delete_provider(DeleteProviderRequest {
2417+
request_id: String::new(),
24022418
allow_missing: true,
24032419
name: name.clone(),
24042420
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),

‎crates/openshell-cli/src/run.rs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -642,6 +642,7 @@ pub async fn sandbox_create(
642642
)])
643643
};
644644
let request = CreateSandboxRequest {
645+
request_id: String::new(),
645646
spec: Some(SandboxSpec {
646647
resource_requirements,
647648
environment: if template.is_none() {
@@ -2685,6 +2686,7 @@ pub async fn sandbox_template_create(
26852686
let mut client = grpc_client(server, tls).await?;
26862687
let response = client
26872688
.create_sandbox_template(CreateSandboxTemplateRequest {
2689+
request_id: String::new(),
26882690
template: Some(SandboxWorkloadTemplate {
26892691
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
26902692
id: String::new(),
@@ -2872,6 +2874,7 @@ pub async fn sandbox_template_delete(
28722874
for name in names {
28732875
let response = client
28742876
.delete_sandbox_template(DeleteSandboxTemplateRequest {
2877+
request_id: String::new(),
28752878
allow_missing: true,
28762879
name: name.clone(),
28772880
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -3300,6 +3303,7 @@ pub async fn sandbox_delete(
33003303

33013304
let response = match client
33023305
.delete_sandbox(DeleteSandboxRequest {
3306+
request_id: String::new(),
33033307
allow_missing: true,
33043308
name: name.clone(),
33053309
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -3360,6 +3364,7 @@ pub async fn sandbox_stop(
33603364
let mut client = grpc_client(server, tls).await?;
33613365
let sandbox = client
33623366
.stop_sandbox(StopSandboxRequest {
3367+
request_id: String::new(),
33633368
name: name.to_string(),
33643369
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
33653370
})
@@ -3383,6 +3388,7 @@ pub async fn sandbox_start(
33833388
let mut client = grpc_client(server, tls).await?;
33843389
let sandbox = client
33853390
.start_sandbox(StartSandboxRequest {
3391+
request_id: String::new(),
33863392
name: name.to_string(),
33873393
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
33883394
})
@@ -3481,6 +3487,7 @@ pub async fn service_expose(
34813487
let mut client = grpc_client(server, tls).await?;
34823488
let response = client
34833489
.expose_service(ExposeServiceRequest {
3490+
request_id: String::new(),
34843491
sandbox: sandbox.to_string(),
34853492
service: service.to_string(),
34863493
target_port: u32::from(target_port),
@@ -3607,6 +3614,7 @@ pub async fn service_delete(
36073614
let mut client = grpc_client(server, tls).await?;
36083615
let response = client
36093616
.delete_service(DeleteServiceRequest {
3617+
request_id: String::new(),
36103618
allow_missing: false,
36113619
sandbox: sandbox.to_string(),
36123620
service: service.to_string(),
@@ -3827,6 +3835,7 @@ pub async fn workspace_create(
38273835
let mut client = grpc_client(server, tls).await?;
38283836
let response = client
38293837
.create_workspace(CreateWorkspaceRequest {
3838+
request_id: String::new(),
38303839
name: name.to_string(),
38313840
labels,
38323841
})
@@ -3981,6 +3990,7 @@ pub async fn workspace_delete(server: &str, names: &[String], tls: &TlsOptions)
39813990
for name in names {
39823991
let response = client
39833992
.delete_workspace(DeleteWorkspaceRequest {
3993+
request_id: String::new(),
39843994
allow_missing: false,
39853995
name: name.clone(),
39863996
})
@@ -4018,6 +4028,7 @@ pub async fn workspace_member_add(
40184028
let mut client = grpc_client(server, tls).await?;
40194029
let response = client
40204030
.add_workspace_member(AddWorkspaceMemberRequest {
4031+
request_id: String::new(),
40214032
workspace: workspace.to_string(),
40224033
principal_subject: subject.to_string(),
40234034
role: role_val.into(),
@@ -4052,6 +4063,7 @@ pub async fn workspace_member_remove(
40524063
let mut client = grpc_client(server, tls).await?;
40534064
let response = client
40544065
.remove_workspace_member(RemoveWorkspaceMemberRequest {
4066+
request_id: String::new(),
40554067
allow_missing: true,
40564068
workspace: workspace.to_string(),
40574069
principal_subject: subject.to_string(),
@@ -5876,6 +5888,7 @@ pub async fn sandbox_draft_approve(
58765888

58775889
let response = client
58785890
.approve_draft_chunk(ApproveDraftChunkRequest {
5891+
request_id: String::new(),
58795892
name: name.to_string(),
58805893
chunk_id: chunk_id.to_string(),
58815894
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -5908,6 +5921,7 @@ pub async fn sandbox_draft_reject(
59085921

59095922
client
59105923
.reject_draft_chunk(RejectDraftChunkRequest {
5924+
request_id: String::new(),
59115925
name: name.to_string(),
59125926
chunk_id: chunk_id.to_string(),
59135927
reason: reason.to_string(),
@@ -5949,6 +5963,7 @@ pub async fn sandbox_draft_approve_all(
59495963

59505964
let response = client
59515965
.approve_all_draft_chunks(ApproveAllDraftChunksRequest {
5966+
request_id: String::new(),
59525967
name: name.to_string(),
59535968
include_security_flagged,
59545969
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
@@ -5980,6 +5995,7 @@ pub async fn sandbox_draft_clear(
59805995

59815996
let response = client
59825997
.clear_draft_chunks(ClearDraftChunksRequest {
5998+
request_id: String::new(),
59835999
name: name.to_string(),
59846000
workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
59856001
})

‎crates/openshell-gateway-interceptors/src/proto_json.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,7 @@ mod tests {
308308
let codec =
309309
ProtoJsonCodec::from_descriptor_set(openshell_core::FILE_DESCRIPTOR_SET).unwrap();
310310
let request = CreateSandboxRequest {
311+
request_id: String::new(),
311312
spec: Some(SandboxSpec {
312313
providers: vec!["github".to_string()],
313314
..SandboxSpec::default()
@@ -336,6 +337,7 @@ mod tests {
336337
fn interceptor_view_omits_nested_secrets_but_keeps_non_secret_fields() {
337338
let codec = ProtoJsonCodec::openshell().unwrap();
338339
let request = CreateProviderRequest {
340+
request_id: String::new(),
339341
provider: Some(Provider {
340342
r#type: "github".to_string(),
341343
credentials: HashMap::from([(
@@ -396,6 +398,7 @@ mod tests {
396398
fn generic_sandbox_environment_remains_visible() {
397399
let codec = ProtoJsonCodec::openshell().unwrap();
398400
let request = CreateSandboxRequest {
401+
request_id: String::new(),
399402
spec: Some(SandboxSpec {
400403
environment: HashMap::from([("FEATURE_FLAG".to_string(), "on".to_string())]),
401404
..SandboxSpec::default()

‎crates/openshell-gateway-interceptors/src/runtime.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -755,6 +755,7 @@ mod tests {
755755

756756
fn create_provider_operation(codec: &ProtoJsonCodec) -> ValidatedOperation {
757757
let request = CreateProviderRequest {
758+
request_id: String::new(),
758759
provider: Some(Provider {
759760
r#type: "github".to_string(),
760761
credentials: HashMap::from([(
@@ -1009,6 +1010,7 @@ mod tests {
10091010
codec: codec.clone(),
10101011
};
10111012
let request = UpdateConfigRequest {
1013+
request_id: String::new(),
10121014
name: "demo".to_string(),
10131015
expected_resource_version: u64::MAX - 1,
10141016
annotations: HashMap::from([
@@ -1049,6 +1051,7 @@ mod tests {
10491051
let codec =
10501052
ProtoJsonCodec::from_descriptor_set(openshell_core::FILE_DESCRIPTOR_SET).unwrap();
10511053
let request = CreateSandboxRequest {
1054+
request_id: String::new(),
10521055
spec: Some(SandboxSpec {
10531056
template: Some(SandboxTemplate {
10541057
resources: Some(

0 commit comments

Comments
 (0)