Skip to content

Commit e3d26dd

Browse files
authored
fix(policy): keep approved chunk when a mechanistic denial resubmits its endpoint (NVIDIA#2242)
* fix(policy): keep approved chunk when a mechanistic denial resubmits its endpoint A mechanistic denial flush for an endpoint already covered by an auto-approved mechanistic chunk flipped that chunk approved -> rejected with no human action. The dedup upsert in put_draft_chunk returns the existing row's id, which aliases onto the approved chunk; the self-reject scan then matched the row against itself and rejected it, while the merged rule stayed enforced — the governance ledger disagreed with the live policy. Guard self_reject_mechanistic_if_already_covered to act only on a still pending effective chunk, and exclude the incoming id from the covering scan. Add a regression test and document the dedup/self-reject invariant. Fixes NVIDIA#2165 Refs NVIDIA/NemoClaw#6329 Signed-off-by: Tinson Lai <[email protected]> * fix(policy): reject mechanistic chunk via atomic pending compare-and-set The self-reject-when-covered path read a chunk, confirmed it was pending, then issued an unconditional status update to rejected. An approval that committed between the read and the write flipped an already-approved chunk to rejected while its rule stayed merged, recreating the ledger/enforcement mismatch through a concurrent path. Add conditionally_reject_draft_chunk to the policy store: the pending->rejected transition carries a status = 'pending' predicate on the final write and reports whether a row changed. Zero changed rows is a benign no-op, meaning another operation already decided the chunk. Implemented for both SQLite and PostgreSQL. The pending pre-read and the self-exclusion guard stay as defense-in-depth. Adds persistence-level regression tests proving an approved row cannot be conditionally rejected and that an approval racing the reject wins. Signed-off-by: Tinson Lai <[email protected]> --------- Signed-off-by: Tinson Lai <[email protected]>
1 parent df06286 commit e3d26dd

5 files changed

Lines changed: 417 additions & 11 deletions

File tree

‎architecture/security-policy.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,19 @@ through the proposal loop instead of treating the denial as terminal.
128128
auto-rejects the older ones with reason `"superseded by chunk X"`. This
129129
gives the agent a refinement path (broad mechanistic L4 → narrow agent
130130
L7) without an explicit `supersedes_chunk_id` field.
131-
5. **Escalation.** Anything else lands in `pending` for human review.
131+
5. **Mechanistic dedup and self-reject.** Mechanistic submissions dedup on
132+
`(host, port, binary)`: a repeat denial for an endpoint that already has a
133+
draft row folds into that row (bumping `hit_count`) instead of creating a
134+
new chunk. When the endpoint is already covered by a *different* approved
135+
chunk, the redundant mechanistic submission self-rejects on arrival with
136+
reason `"already covered by approved chunk X"`. This only ever acts on a
137+
genuinely fresh, still-`pending` submission; it never rewrites the status
138+
of an already-decided chunk. A dedup hit that resolves to an approved
139+
row's own id leaves that chunk `approved` and merged — the self-reject
140+
path does not un-merge a rule, so flipping an approved chunk to `rejected`
141+
would leave the governance ledger disagreeing with the still-enforced
142+
policy.
143+
6. **Escalation.** Anything else lands in `pending` for human review.
132144

133145
## What the prover decides
134146

‎crates/openshell-server/src/grpc/policy.rs‎

Lines changed: 301 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -778,6 +778,29 @@ async fn self_reject_mechanistic_if_already_covered(
778778
return;
779779
}
780780

781+
// `put_draft_chunk` dedups mechanistic submissions on `(host, port,
782+
// binary)` and returns the id of whatever row now owns that key. On a
783+
// dedup hit that id belongs to a pre-existing row, which may already be
784+
// `approved`. Self-reject must only fire on a genuinely fresh, still
785+
// `pending` submission: flipping a non-pending row here would move an
786+
// already-decided chunk to `rejected` with no human action, and because
787+
// this path never un-merges the rule (unlike the human reject handler),
788+
// the live policy would keep enforcing an access the ledger now reports as
789+
// revoked.
790+
match state.store.get_draft_chunk(new_chunk_id).await {
791+
Ok(Some(chunk)) if chunk.status == "pending" => {}
792+
Ok(_) => return,
793+
Err(err) => {
794+
warn!(
795+
sandbox_id = %sandbox_id,
796+
chunk_id = %new_chunk_id,
797+
error = %err,
798+
"self-reject status check failed; mechanistic chunk remains pending"
799+
);
800+
return;
801+
}
802+
}
803+
781804
let approved = match state
782805
.store
783806
.list_draft_chunks(sandbox_id, Some("approved"))
@@ -794,11 +817,14 @@ async fn self_reject_mechanistic_if_already_covered(
794817
}
795818
};
796819

797-
// If any approved chunk for this sandbox already targets the same
798-
// (host, port, binary), the mechanistic submission is redundant.
820+
// If any *other* approved chunk for this sandbox already targets the same
821+
// (host, port, binary), the mechanistic submission is redundant. Excluding
822+
// `new_chunk_id` is load-bearing: the dedup upsert can alias the incoming
823+
// id onto an approved row for this endpoint, and without the exclusion that
824+
// row would be found as its own "covering" chunk and reject itself.
799825
let covered_by = approved
800826
.iter()
801-
.find(|c| c.host == host && c.port == port && c.binary == binary);
827+
.find(|c| c.id != new_chunk_id && c.host == host && c.port == port && c.binary == binary);
802828
let Some(covering) = covered_by else {
803829
return;
804830
};
@@ -809,15 +835,10 @@ async fn self_reject_mechanistic_if_already_covered(
809835
);
810836
match state
811837
.store
812-
.update_draft_chunk_status(
813-
new_chunk_id,
814-
"rejected",
815-
Some(current_time_ms()),
816-
Some(&reason),
817-
)
838+
.conditionally_reject_draft_chunk(new_chunk_id, current_time_ms(), &reason)
818839
.await
819840
{
820-
Ok(_) => {
841+
Ok(true) => {
821842
info!(
822843
sandbox_id = %sandbox_id,
823844
chunk_id = %new_chunk_id,
@@ -828,6 +849,13 @@ async fn self_reject_mechanistic_if_already_covered(
828849
"Auto-rejected incoming mechanistic chunk: endpoint already covered by an approved chunk"
829850
);
830851
}
852+
Ok(false) => {
853+
info!(
854+
sandbox_id = %sandbox_id,
855+
chunk_id = %new_chunk_id,
856+
"mechanistic self-reject skipped: chunk no longer pending (decided concurrently)"
857+
);
858+
}
831859
Err(err) => {
832860
warn!(
833861
chunk_id = %new_chunk_id,
@@ -7988,6 +8016,269 @@ mod tests {
79888016
);
79898017
}
79908018

8019+
/// Regression: a mechanistic denial flush for an endpoint already covered
8020+
/// by an auto-approved mechanistic chunk must NOT flip that chunk to
8021+
/// `rejected`. The dedup upsert returns the approved row's own id; before
8022+
/// the self-exclusion + pending guard the self-reject scan matched the row
8023+
/// against itself and rejected it, corrupting the governance ledger (status
8024+
/// read `rejected`) while the merged rule stayed enforced.
8025+
#[tokio::test]
8026+
async fn resubmitted_mechanistic_endpoint_keeps_approved_chunk() {
8027+
use openshell_core::proto::{
8028+
FilesystemPolicy, NetworkBinary, NetworkEndpoint, SandboxPhase, SandboxPolicy,
8029+
SandboxSpec,
8030+
};
8031+
8032+
let state = test_server_state().await;
8033+
let sandbox_name = "mechanistic-reapprove".to_string();
8034+
let sandbox_id = "sb-mechanistic-reapprove";
8035+
let mut sandbox = Sandbox {
8036+
metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
8037+
id: sandbox_id.to_string(),
8038+
name: sandbox_name.clone(),
8039+
created_at_ms: 1_000_000,
8040+
labels: std::collections::HashMap::new(),
8041+
resource_version: 0,
8042+
}),
8043+
spec: Some(SandboxSpec {
8044+
policy: Some(SandboxPolicy {
8045+
version: 1,
8046+
filesystem: Some(FilesystemPolicy {
8047+
read_write: vec!["/sandbox".to_string()],
8048+
..Default::default()
8049+
}),
8050+
..Default::default()
8051+
}),
8052+
..Default::default()
8053+
}),
8054+
..Default::default()
8055+
};
8056+
sandbox.set_phase(SandboxPhase::Ready as i32);
8057+
state.store.put_message(&sandbox).await.unwrap();
8058+
// No providers in scope + auto mode → empty-delta prover verdict → the
8059+
// first mechanistic submit auto-approves and merges the rule.
8060+
seed_sandbox_approval_mode(&state, &sandbox_name, "auto").await;
8061+
8062+
let proposed_rule = NetworkPolicyRule {
8063+
name: "allow_10_0_0_5_8080".to_string(),
8064+
endpoints: vec![NetworkEndpoint {
8065+
host: "10.0.0.5".to_string(),
8066+
port: 8080,
8067+
..Default::default()
8068+
}],
8069+
binaries: vec![NetworkBinary {
8070+
path: "/usr/bin/curl".to_string(),
8071+
..Default::default()
8072+
}],
8073+
};
8074+
let submit_one = || {
8075+
let state = state.clone();
8076+
let sandbox_name = sandbox_name.clone();
8077+
let rule = proposed_rule.clone();
8078+
async move {
8079+
handle_submit_policy_analysis(
8080+
&state,
8081+
with_user(Request::new(SubmitPolicyAnalysisRequest {
8082+
name: sandbox_name,
8083+
analysis_mode: "mechanistic".to_string(),
8084+
proposed_chunks: vec![PolicyChunk {
8085+
rule_name: "allow_10_0_0_5_8080".to_string(),
8086+
proposed_rule: Some(rule),
8087+
..Default::default()
8088+
}],
8089+
..Default::default()
8090+
})),
8091+
)
8092+
.await
8093+
.unwrap()
8094+
.into_inner()
8095+
}
8096+
};
8097+
8098+
// First submit: auto-approves the endpoint.
8099+
submit_one().await;
8100+
let after_first = handle_get_draft_policy(
8101+
&state,
8102+
with_user(Request::new(GetDraftPolicyRequest {
8103+
name: sandbox_name.clone(),
8104+
status_filter: String::new(),
8105+
})),
8106+
)
8107+
.await
8108+
.unwrap()
8109+
.into_inner();
8110+
assert_eq!(after_first.chunks.len(), 1);
8111+
assert_eq!(
8112+
after_first.chunks[0].status, "approved",
8113+
"first mechanistic submit under auto mode must auto-approve; got {}",
8114+
after_first.chunks[0].status
8115+
);
8116+
8117+
// Second submit for the SAME endpoint: the dedup upsert returns the
8118+
// approved row's own id. The chunk must stay approved, not self-reject.
8119+
submit_one().await;
8120+
let after_second = handle_get_draft_policy(
8121+
&state,
8122+
with_user(Request::new(GetDraftPolicyRequest {
8123+
name: sandbox_name.clone(),
8124+
status_filter: String::new(),
8125+
})),
8126+
)
8127+
.await
8128+
.unwrap()
8129+
.into_inner();
8130+
assert_eq!(
8131+
after_second.chunks.len(),
8132+
1,
8133+
"resubmit must dedup into the one existing row"
8134+
);
8135+
let chunk = &after_second.chunks[0];
8136+
assert_eq!(
8137+
chunk.status, "approved",
8138+
"resubmitting a mechanistic denial for an already-approved endpoint must not flip \
8139+
the approved chunk to rejected; got status {}",
8140+
chunk.status
8141+
);
8142+
assert!(
8143+
chunk.rejection_reason.is_empty(),
8144+
"approved chunk must carry no rejection reason; got {:?}",
8145+
chunk.rejection_reason
8146+
);
8147+
assert_eq!(
8148+
chunk.hit_count, 2,
8149+
"the redundant denial must fold into hit_count, not a status flip"
8150+
);
8151+
8152+
// The rule must remain merged in the active policy — the ledger and the
8153+
// enforced policy must agree.
8154+
let latest = state
8155+
.store
8156+
.get_latest_policy(sandbox_id)
8157+
.await
8158+
.unwrap()
8159+
.expect("auto-approve must have persisted a policy revision");
8160+
let policy = SandboxPolicy::decode(latest.policy_payload.as_slice()).unwrap();
8161+
assert!(
8162+
policy.network_policies.contains_key("allow_10_0_0_5_8080"),
8163+
"approved rule must stay merged after resubmit; keys: {:?}",
8164+
policy.network_policies.keys().collect::<Vec<_>>()
8165+
);
8166+
}
8167+
8168+
fn pending_draft_chunk(id: &str, sandbox_id: &str) -> DraftChunkRecord {
8169+
DraftChunkRecord {
8170+
id: id.to_string(),
8171+
sandbox_id: sandbox_id.to_string(),
8172+
draft_version: 1,
8173+
status: "pending".to_string(),
8174+
rule_name: "allow_endpoint".to_string(),
8175+
proposed_rule: Vec::new(),
8176+
rationale: String::new(),
8177+
security_notes: String::new(),
8178+
confidence: 0.5,
8179+
created_at_ms: 1_000,
8180+
decided_at_ms: None,
8181+
host: "10.0.0.5".to_string(),
8182+
port: 8080,
8183+
binary: "/usr/bin/curl".to_string(),
8184+
hit_count: 1,
8185+
first_seen_ms: 1_000,
8186+
last_seen_ms: 1_000,
8187+
validation_result: String::new(),
8188+
rejection_reason: String::new(),
8189+
}
8190+
}
8191+
8192+
#[tokio::test]
8193+
async fn conditionally_reject_transitions_pending_chunk() {
8194+
let store = test_store().await;
8195+
store
8196+
.put_draft_chunk(&pending_draft_chunk("cas-pending", "sb-cas-pending"), None)
8197+
.await
8198+
.unwrap();
8199+
8200+
let changed = store
8201+
.conditionally_reject_draft_chunk("cas-pending", 5, "covered by approved chunk cover-1")
8202+
.await
8203+
.unwrap();
8204+
assert!(changed, "a pending chunk must transition to rejected");
8205+
8206+
let stored = store.get_draft_chunk("cas-pending").await.unwrap().unwrap();
8207+
assert_eq!(stored.status, "rejected");
8208+
assert_eq!(stored.decided_at_ms, Some(5));
8209+
assert_eq!(stored.rejection_reason, "covered by approved chunk cover-1");
8210+
}
8211+
8212+
#[tokio::test]
8213+
async fn conditionally_reject_leaves_approved_chunk_untouched() {
8214+
let store = test_store().await;
8215+
store
8216+
.put_draft_chunk(
8217+
&pending_draft_chunk("cas-approved", "sb-cas-approved"),
8218+
None,
8219+
)
8220+
.await
8221+
.unwrap();
8222+
assert!(
8223+
store
8224+
.update_draft_chunk_status("cas-approved", "approved", Some(1), None)
8225+
.await
8226+
.unwrap()
8227+
);
8228+
8229+
let changed = store
8230+
.conditionally_reject_draft_chunk("cas-approved", 2, "covered")
8231+
.await
8232+
.unwrap();
8233+
assert!(
8234+
!changed,
8235+
"an approved chunk must not be conditionally rejected"
8236+
);
8237+
8238+
let stored = store
8239+
.get_draft_chunk("cas-approved")
8240+
.await
8241+
.unwrap()
8242+
.unwrap();
8243+
assert_eq!(
8244+
stored.status, "approved",
8245+
"the approved status must survive a losing conditional reject"
8246+
);
8247+
assert!(stored.rejection_reason.is_empty());
8248+
}
8249+
8250+
#[tokio::test]
8251+
async fn conditionally_reject_loses_race_to_approval() {
8252+
let store = test_store().await;
8253+
store
8254+
.put_draft_chunk(&pending_draft_chunk("cas-race", "sb-cas-race"), None)
8255+
.await
8256+
.unwrap();
8257+
8258+
let observed = store.get_draft_chunk("cas-race").await.unwrap().unwrap();
8259+
assert_eq!(observed.status, "pending");
8260+
8261+
assert!(
8262+
store
8263+
.update_draft_chunk_status("cas-race", "approved", Some(10), None)
8264+
.await
8265+
.unwrap()
8266+
);
8267+
8268+
let changed = store
8269+
.conditionally_reject_draft_chunk("cas-race", 11, "covered")
8270+
.await
8271+
.unwrap();
8272+
assert!(!changed);
8273+
8274+
let stored = store.get_draft_chunk("cas-race").await.unwrap().unwrap();
8275+
assert_eq!(
8276+
stored.status, "approved",
8277+
"an approval that commits before the conditional reject must win; the ledger must \
8278+
never read rejected while the merged rule stays enforced"
8279+
);
8280+
}
8281+
79918282
/// Undo of an approve must clear any `rejection_reason` left over from a
79928283
/// prior reject. Without this, the in-sandbox agent reading chunks via
79938284
/// `policy.local` cannot tell "pending and never rejected" from "pending

0 commit comments

Comments
 (0)