Skip to content

feat(policy): lint authored policy files offline - #5358

Open
shiju-nv wants to merge 2 commits into
NVIDIA:mainfrom
shiju-nv:feat/4035-offline-policy-lint/shiju-nv
Open

shiju-nv wants to merge 2 commits into
NVIDIA:mainfrom
shiju-nv:feat/4035-offline-policy-lint/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add openshell policy lint <file> to validate authored policies without a gateway. After parsing, it reports file-only semantic violations together and identifies deferred live checks.

Related Issue

Closes #4035

Changes

  • Reuse shared safety, authoring, ambiguity and built-in middleware validation.
  • Dispatch lint before gateway selection and authentication.
  • Cover offline operation, multiple violations and gateway rejection parity.

Release Notes

openshell policy lint policy.yaml validates authored files offline. Success does not establish that a sandbox can start.

Testing

  • Checks appropriate to the affected code pass: two policy unit functions, the CLI integration table, two in-process gateway functions and production Clippy with -D warnings.
  • Unit and integration coverage added. The real Codex review policy also passed offline lint with an unreachable gateway.
  • E2E tests added/updated (not applicable; no full E2E run).

Checklist

  • Follows Conventional Commits.
  • Commit signed and signed off for DCO.
  • Architecture docs updated (not applicable; command guide and policy/CLI skills updated).

@shiju-nv shiju-nv added area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. rust Pull requests that update rust code labels Nov 7, 2026
Validate authored files before gateway selection or authentication. Share creation safety checks and report file-only violations together while deferring checks that require live gateway state.

Closes NVIDIA#4035

Signed-off-by: Shiju <[email protected]>
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

CreateSandbox size rejections after MCP defaulting previously included BadRequest(spec.policy) and ErrorInfo(INVALID_ARGUMENT), but now return bare INVALID_ARGUMENT.

  • Impact: SDK consumers handling policies that cross 262,144 bytes during MCP defaulting lose the structured field and reason needed for automated error handling.
  • Action: For v0.1.8, preserve BadRequest(spec.policy) and ErrorInfo(INVALID_ARGUMENT) by retaining the existing gateway size-check path or returning the equivalent structured error.
  • PR: feat(policy): lint authored policy files offline #5358
Review details
  • Execution: complete
  • Candidate: PR #5358 (4afdab630a7d3dfc4c2176bf892815e453513935)
  • Baseline: af8182f7b63da6dd8bbea8b3c313227764ee0dea (af8182f7b63da6dd8bbea8b3c313227764ee0dea)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shiju-nv
shiju-nv force-pushed the feat/4035-offline-policy-lint/shiju-nv branch from 74dcaf5 to b49c040 Compare November 7, 2026 09:18
@shiju-nv shiju-nv removed the area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. label Nov 7, 2026

@shvma shvma left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran 11 authored policies through openshell policy lint and through CreateSandbox on the in-process gateway from openshell_server::test_support. For the four built-in middleware cases I compared against MiddlewareRegistry::validate_policy_configs with the built-in services instead (see the note at the end). Lint agreed everywhere except the external middleware case, which lint skips by design, as the docs say. Two small things, both with patches below.

  1. The pair order in endpoint ambiguity messages isn't stable. With two conflicting rules a and b, openshell policy lint printed 'a' … and 'b' in 10 of 20 runs and 'b' … and 'a' in the other 10. find_endpoint_ambiguities walks the network_policies map without sorting. The code predates this PR, and the gateway message has the same problem. Lint makes it more visible because its output goes to CI logs and scripts. The crate already states the rule in lib.rs: "Sort rule keys so callers receive stable validation errors for equivalent policy inputs." Patch 1 sorts the keys the same way and adds a unit test that fails without the change (one run got [("mid", "alpha"), ("alpha", "zeta")]) and passes with it. cargo test -p openshell-policy --lib is 300/300 and clippy with -D warnings is clean. Fine as a follow-up if you'd rather keep this PR to lint.

  2. authored_policy_lint_matches_gateway_file_validation covers identity, reserved keys and credential bindings. policy_lint_integration.rs covers ambiguity on the lint side only, and I didn't find a test that CreateSandbox rejects the same authored file. The gateway rejects an ambiguous file on CreateSandbox even without providers, with FailedPrecondition (policy.rs), so it doesn't fit the existing loop that expects InvalidArgument. Patch 2 adds a separate test on the same harness. It uses one rule with two endpoints, so it doesn't depend on the ordering in point 1 and works with or without Patch 1. It passes, and clippy is clean.

Not for this PR, just context for why the parity test can't cover built-in middleware: gateway_service_with_driver doesn't register the built-ins that the real gateway does (openshell-server lib.rs), so openshell/regex is always "not registered" there.

Patch 1: sort ambiguity pairs by rule key
diff --git a/crates/openshell-policy/src/ambiguity.rs b/crates/openshell-policy/src/ambiguity.rs
index efb846a..0eb9739 100644
--- a/crates/openshell-policy/src/ambiguity.rs
+++ b/crates/openshell-policy/src/ambiguity.rs
@@ -58,9 +58,12 @@ struct EndpointRef<'a> {
 /// request-specific metadata) path selectors can match the same request.
 #[must_use]
 pub fn find_endpoint_ambiguities(policy: &SandboxPolicy) -> Vec<EndpointAmbiguity> {
-    let endpoints = policy
-        .network_policies
-        .iter()
+    // Protobuf maps do not preserve iteration order. Sort rule keys so callers
+    // receive the same ambiguity pairs, in the same order, for equivalent input.
+    let mut network_policies: Vec<_> = policy.network_policies.iter().collect();
+    network_policies.sort_by_key(|(key, _)| *key);
+    let endpoints = network_policies
+        .into_iter()
         .flat_map(|(key, rule)| {
             let policy_name = if rule.name.is_empty() {
                 key.as_str()
@@ -775,6 +778,43 @@ mod tests {
         assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
     }
 
+    #[test]
+    fn ambiguities_are_reported_in_rule_key_order() {
+        let pairs = |policy: &SandboxPolicy| {
+            find_endpoint_ambiguities(policy)
+                .into_iter()
+                .map(|ambiguity| (ambiguity.left_policy, ambiguity.right_policy))
+                .collect::<Vec<_>>()
+        };
+        let build = || {
+            let mut policy = SandboxPolicy::default();
+            for (key, tls) in [
+                ("zeta", NetworkTlsMode::Skip),
+                ("alpha", NetworkTlsMode::Unspecified),
+                ("mid", NetworkTlsMode::Skip),
+            ] {
+                let mut endpoint = endpoint("api.example.com", 443);
+                endpoint.tls = tls as i32;
+                policy.network_policies.insert(
+                    key.to_string(),
+                    NetworkPolicyRule {
+                        endpoints: vec![endpoint],
+                        ..Default::default()
+                    },
+                );
+            }
+            policy
+        };
+        let expected = vec![
+            ("alpha".to_string(), "mid".to_string()),
+            ("alpha".to_string(), "zeta".to_string()),
+        ];
+        // Each map gets its own hash seed, so repeat to catch unordered iteration.
+        for _ in 0..32 {
+            assert_eq!(pairs(&build()), expected);
+        }
+    }
+
     #[test]
     fn compatible_request_rules_may_overlap() {
         let mut left = endpoint("api.example.com", 443);
Patch 2: parity test for endpoint ambiguity
diff --git a/crates/openshell-cli/tests/policy_only_process_identity_integration.rs b/crates/openshell-cli/tests/policy_only_process_identity_integration.rs
index 559534e..90b77ca 100644
--- a/crates/openshell-cli/tests/policy_only_process_identity_integration.rs
+++ b/crates/openshell-cli/tests/policy_only_process_identity_integration.rs
@@ -183,3 +183,35 @@ async fn authored_policy_lint_matches_gateway_file_validation() {
         }
     }
 }
+
+#[tokio::test]
+async fn authored_policy_lint_matches_gateway_endpoint_ambiguity() {
+    let server = run_server().await;
+    let policy = openshell_policy::parse_sandbox_policy(
+        "version: 1\nnetwork_policies:\n  conflict:\n    endpoints:\n      - {host: example.com, port: 443}\n      - {host: example.com, port: 443, tls: skip}\n",
+    )
+    .unwrap();
+    let ambiguities = openshell_policy::find_endpoint_ambiguities(&policy);
+    assert_eq!(ambiguities.len(), 1, "{ambiguities:?}");
+    let mut client = grpc_client(&server.endpoint, &server.tls).await.unwrap();
+    let error = client
+        .create_sandbox(CreateSandboxRequest {
+            name: "ambiguity".to_string(),
+            spec: Some(SandboxSpec {
+                policy: Some(policy),
+                ..Default::default()
+            }),
+            workspace_scope: Some(openshell_core::proto::workspace_selector(
+                "default".to_string(),
+            )),
+            ..Default::default()
+        })
+        .await
+        .expect_err("an ambiguous authored file must be rejected by the gateway too");
+    assert_eq!(error.code(), tonic::Code::FailedPrecondition);
+    assert!(
+        error.message().contains(&ambiguities[0].to_string()),
+        "{error} did not report {}",
+        ambiguities[0]
+    );
+}

@purp purp added topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:ux CLI commands/flags, help, errors/exit codes, human-readable output, TUI, and first-run experience. labels Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. rust Pull requests that update rust code topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:ux CLI commands/flags, help, errors/exit codes, human-readable output, TUI, and first-run experience.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add a command to check a policy file before creating a sandbox

3 participants