Skip to content

Commit 2afbb3c

Browse files
committed
feat(observability): record generation on mapping denials and middleware reloads
Two events still lacked the generation after the main change. A transparent TCP connection that passes policy evaluation but then fails the DNS mapping check, including when the policy changes between the decision and the mapping lookup, now records the decision's generation in firewall_rule.version. Opens that fail the mapping check before evaluation leave it unset. Installing a new middleware registry advances the policy engine generation. replace_middleware_registry now returns that generation, and the "Supervisor middleware registry reloaded" event records it as unmapped.active_generation, like the other activation events. Refs NVIDIA#3840 Signed-off-by: Rakesh Gohel <[email protected]>
1 parent 73b367d commit 2afbb3c

4 files changed

Lines changed: 138 additions & 36 deletions

File tree

‎crates/openshell-supervisor-network/src/opa.rs‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -892,7 +892,9 @@ impl OpaEngine {
892892

893893
/// Replace the complete middleware service registry and invalidate
894894
/// existing tunnels so subsequent requests use the new service set.
895-
pub fn replace_middleware_registry(&self, registry: MiddlewareRegistry) -> Result<()> {
895+
/// Install a new middleware registry and return the policy generation it
896+
/// activated.
897+
pub fn replace_middleware_registry(&self, registry: MiddlewareRegistry) -> Result<u64> {
896898
// Generation changes serialize through the engine lock so guarded
897899
// publication cannot overlap any runtime generation transition.
898900
let _engine = self
@@ -904,8 +906,7 @@ impl OpaEngine {
904906
.write()
905907
.map_err(|_| miette::miette!("middleware runner lock poisoned"))?;
906908
*runner = runner.with_replacement_registry(registry);
907-
self.advance_generation();
908-
Ok(())
909+
Ok(self.advance_generation())
909910
}
910911

911912
pub(crate) fn middleware_runner(&self) -> Result<ChainRunner> {
@@ -8539,6 +8540,17 @@ network_policies:
85398540
assert_eq!(val, regorus::Value::from(true));
85408541
}
85418542

8543+
#[test]
8544+
fn replacing_the_middleware_registry_returns_the_activated_generation() {
8545+
let engine = test_engine();
8546+
let before = engine.current_generation();
8547+
let activated = engine
8548+
.replace_middleware_registry(MiddlewareRegistry::default())
8549+
.unwrap();
8550+
assert_eq!(activated, before + 1);
8551+
assert_eq!(engine.current_generation(), activated);
8552+
}
8553+
85428554
#[test]
85438555
fn policy_generation_starts_at_zero_and_increments_on_successful_reload() {
85448556
let engine = l7_engine();

‎crates/openshell-supervisor-network/src/proxy.rs‎

Lines changed: 84 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1251,7 +1251,7 @@ async fn handle_transparent_tcp_connection(
12511251
) {
12521252
Ok(mapping) => mapping,
12531253
Err(error) => {
1254-
emit_transparent_mapping_denial(workload_addr, original, None, error);
1254+
emit_transparent_mapping_denial(workload_addr, original, None, error, None);
12551255
emit_activity(&activity_tx, true, "transparent_tcp_mapping");
12561256
return Ok(());
12571257
}
@@ -1295,6 +1295,7 @@ async fn handle_transparent_tcp_connection(
12951295
original,
12961296
Some(&host),
12971297
MappingLookupError::EndpointMismatch,
1298+
Some(decision.policy_generation),
12981299
);
12991300
emit_activity(&activity_tx, true, "transparent_tcp_mapping");
13001301
return Ok(());
@@ -1313,6 +1314,7 @@ async fn handle_transparent_tcp_connection(
13131314
original,
13141315
Some(&host),
13151316
MappingLookupError::StalePolicy,
1317+
Some(decision.policy_generation),
13161318
);
13171319
emit_activity(&activity_tx, true, "transparent_tcp_mapping");
13181320
return Ok(());
@@ -1325,7 +1327,13 @@ async fn handle_transparent_tcp_connection(
13251327
) {
13261328
Ok(mapping) => mapping,
13271329
Err(error) => {
1328-
emit_transparent_mapping_denial(workload_addr, original, Some(&host), error);
1330+
emit_transparent_mapping_denial(
1331+
workload_addr,
1332+
original,
1333+
Some(&host),
1334+
error,
1335+
Some(decision.policy_generation),
1336+
);
13291337
emit_activity(&activity_tx, true, "transparent_tcp_mapping");
13301338
return Ok(());
13311339
}
@@ -1579,7 +1587,28 @@ fn emit_transparent_mapping_denial(
15791587
original: SocketAddr,
15801588
mapped_host: Option<&str>,
15811589
error: MappingLookupError,
1590+
policy_generation: Option<u64>,
15821591
) {
1592+
egress_emit!(build_transparent_mapping_denial_event(
1593+
workload,
1594+
original,
1595+
mapped_host,
1596+
error,
1597+
policy_generation,
1598+
));
1599+
}
1600+
1601+
/// `policy_generation` is the generation of the egress decision made before
1602+
/// the mapping check, or `None` when the open failed before policy
1603+
/// evaluation.
1604+
#[cfg(target_os = "linux")]
1605+
fn build_transparent_mapping_denial_event(
1606+
workload: SocketAddr,
1607+
original: SocketAddr,
1608+
mapped_host: Option<&str>,
1609+
error: MappingLookupError,
1610+
policy_generation: Option<u64>,
1611+
) -> openshell_ocsf::OcsfEvent {
15831612
let detail = match error {
15841613
MappingLookupError::Missing => "transparent_tcp_mapping_missing",
15851614
MappingLookupError::Expired => "transparent_tcp_mapping_expired",
@@ -1589,19 +1618,21 @@ fn emit_transparent_mapping_denial(
15891618
| MappingLookupError::InvalidMapping
15901619
| MappingLookupError::LockPoisoned => "transparent_tcp_destination_denied",
15911620
};
1592-
egress_emit!(
1593-
NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx())
1594-
.activity(ActivityId::Open)
1595-
.action(ActionId::Denied)
1596-
.disposition(DispositionId::Blocked)
1597-
.severity(SeverityId::Medium)
1598-
.status(StatusId::Failure)
1599-
.dst_endpoint(transparent_destination_endpoint(original, mapped_host))
1600-
.src_endpoint_addr(workload.ip(), workload.port())
1601-
.message(format!("Transparent TCP denied: {error}"))
1602-
.status_detail(detail)
1603-
.build()
1604-
);
1621+
let mut builder = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx())
1622+
.activity(ActivityId::Open)
1623+
.action(ActionId::Denied)
1624+
.disposition(DispositionId::Blocked)
1625+
.severity(SeverityId::Medium)
1626+
.status(StatusId::Failure)
1627+
.dst_endpoint(transparent_destination_endpoint(original, mapped_host))
1628+
.src_endpoint_addr(workload.ip(), workload.port());
1629+
if let Some(generation) = policy_generation {
1630+
builder = builder.firewall_rule_at_generation("-", "opa", generation);
1631+
}
1632+
builder
1633+
.message(format!("Transparent TCP denied: {error}"))
1634+
.status_detail(detail)
1635+
.build()
16051636
}
16061637

16071638
#[cfg(target_os = "linux")]
@@ -8537,6 +8568,44 @@ process: {{ run_as_user: sandbox, run_as_group: sandbox }}
85378568
);
85388569
}
85398570

8571+
#[cfg(target_os = "linux")]
8572+
#[test]
8573+
fn mapping_denial_after_evaluation_records_the_decision_generation() {
8574+
let workload: SocketAddr = "10.0.0.5:40000".parse().unwrap();
8575+
let original: SocketAddr = "198.18.0.2:443".parse().unwrap();
8576+
8577+
let stale = build_transparent_mapping_denial_event(
8578+
workload,
8579+
original,
8580+
Some("api.example.test"),
8581+
MappingLookupError::StalePolicy,
8582+
Some(5),
8583+
);
8584+
let stale = serde_json::to_value(stale).unwrap();
8585+
assert_eq!(
8586+
stale["status_detail"],
8587+
"transparent_tcp_mapping_stale_policy"
8588+
);
8589+
assert_eq!(
8590+
stale["firewall_rule"],
8591+
serde_json::json!({"name": "-", "type": "opa", "version": "5"})
8592+
);
8593+
8594+
let before_evaluation = build_transparent_mapping_denial_event(
8595+
workload,
8596+
original,
8597+
None,
8598+
MappingLookupError::Missing,
8599+
None,
8600+
);
8601+
let before_evaluation = serde_json::to_value(before_evaluation).unwrap();
8602+
assert_eq!(
8603+
before_evaluation["status_detail"],
8604+
"transparent_tcp_mapping_missing"
8605+
);
8606+
assert!(before_evaluation.get("firewall_rule").is_none());
8607+
}
8608+
85408609
#[test]
85418610
fn staged_denial_names_the_mapped_host_and_keeps_the_synthetic_address() {
85428611
let (open, _) = staged_curl_open("198.18.0.2:80".parse().unwrap(), 1);

‎crates/openshell-supervisor/src/lib.rs‎

Lines changed: 38 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4441,7 +4441,7 @@ async fn install_builtin_middleware_registry(opa_engine: &OpaEngine) -> Result<(
44414441
Vec::new(),
44424442
)
44434443
.await?;
4444-
opa_engine.replace_middleware_registry(registry)
4444+
opa_engine.replace_middleware_registry(registry).map(|_| ())
44454445
}
44464446

44474447
/// Wait the configured poll interval, but never past the point at which an
@@ -4487,6 +4487,25 @@ struct MiddlewareRegistryReconciliation<'a> {
44874487
status: &'a mut MiddlewareRegistryStatus,
44884488
}
44894489

4490+
/// Build the CONFIG event recorded when a new middleware registry is
4491+
/// installed. Installing it advances the policy engine generation, so the
4492+
/// event records the generation it activated.
4493+
fn middleware_registry_reloaded_event(service_count: usize, active_generation: u64) -> OcsfEvent {
4494+
ConfigStateChangeBuilder::new(ocsf_ctx())
4495+
.severity(SeverityId::Informational)
4496+
.status(StatusId::Success)
4497+
.state(StateId::Enabled, "loaded")
4498+
.unmapped(
4499+
"supervisor_middleware_service_count",
4500+
serde_json::json!(service_count),
4501+
)
4502+
.unmapped("active_generation", serde_json::json!(active_generation))
4503+
.message(format!(
4504+
"Supervisor middleware registry reloaded [service_count:{service_count}]"
4505+
))
4506+
.build()
4507+
}
4508+
44904509
async fn reconcile_middleware_registry(
44914510
opa_engine: &OpaEngine,
44924511
middleware_connector: &MiddlewareConnector,
@@ -4503,7 +4522,7 @@ async fn reconcile_middleware_registry(
45034522
.await
45044523
.and_then(|registry| opa_engine.replace_middleware_registry(registry))
45054524
{
4506-
Ok(()) => {
4525+
Ok(active_generation) => {
45074526
retain_extension_credentials(
45084527
reconciliation.extension_credentials,
45094528
reconciliation.desired_services,
@@ -4514,21 +4533,10 @@ async fn reconcile_middleware_registry(
45144533
.current_services
45154534
.extend_from_slice(reconciliation.desired_services);
45164535
*reconciliation.status = MiddlewareRegistryStatus::Synchronized;
4517-
ocsf_emit!(
4518-
ConfigStateChangeBuilder::new(ocsf_ctx())
4519-
.severity(SeverityId::Informational)
4520-
.status(StatusId::Success)
4521-
.state(StateId::Enabled, "loaded")
4522-
.unmapped(
4523-
"supervisor_middleware_service_count",
4524-
serde_json::json!(reconciliation.current_services.len())
4525-
)
4526-
.message(format!(
4527-
"Supervisor middleware registry reloaded [service_count:{}]",
4528-
reconciliation.current_services.len()
4529-
))
4530-
.build()
4531-
);
4536+
ocsf_emit!(middleware_registry_reloaded_event(
4537+
reconciliation.current_services.len(),
4538+
active_generation,
4539+
));
45324540
}
45334541
Err(error) => {
45344542
// Emit only on the transition into the failed state to avoid
@@ -7818,6 +7826,19 @@ network_policies:
78187826
assert_eq!(json["unmapped"]["policy_hash"], "sha256:same");
78197827
assert!(json["unmapped"].get("active_generation").is_none());
78207828
}
7829+
7830+
#[test]
7831+
fn middleware_registry_reload_records_active_generation() {
7832+
let json = middleware_registry_reloaded_event(2, 7).to_json().unwrap();
7833+
assert_eq!(json["unmapped"]["supervisor_middleware_service_count"], 2);
7834+
assert_eq!(json["unmapped"]["active_generation"], 7);
7835+
assert!(
7836+
json["message"]
7837+
.as_str()
7838+
.unwrap()
7839+
.contains("Supervisor middleware registry reloaded [service_count:2]")
7840+
);
7841+
}
78217842
}
78227843

78237844
#[cfg(test)]

‎docs/observability/ocsf-json-export.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -346,7 +346,7 @@ Policy validation failures emit a `CONFIG` event with `active_generation` and `c
346346

347347
The generation starts at zero when the supervisor starts. Use the `LIFECYCLE` start event as the boundary between supervisor lifetimes.
348348

349-
Layer 4 decisions (CONNECT, forward proxy, and transparent TCP) and layer 7 decisions for REST, JSON-RPC, MCP, GraphQL, and inspected WebSocket messages carry the generation. Denials made without matching a rule, such as when the calling process cannot be identified, carry the generation that was active when the denial was made. Events that are not policy decisions, such as MCP protocol version rejections and credential binding failures, leave `firewall_rule.version` unset. Policy DNS refusals also leave it unset. When the workload then connects anyway, the transparent TCP denial that follows carries the generation. Connections closed because the policy changed carry the generation they were authorized under.
349+
Layer 4 decisions (CONNECT, forward proxy, and transparent TCP) and layer 7 decisions for REST, JSON-RPC, MCP, GraphQL, and inspected WebSocket messages carry the generation. Denials made without matching a rule, such as when the calling process cannot be identified, carry the generation that was active when the denial was made. Events that are not policy decisions, such as MCP protocol version rejections and credential binding failures, leave `firewall_rule.version` unset. Policy DNS refusals also leave it unset. When the workload then connects anyway, the transparent TCP denial that follows carries the generation. A transparent TCP connection that passes policy but fails the DNS mapping check, for example because the policy changed in between, carries the generation of that policy decision. Connections closed because the policy changed carry the generation they were authorized under.
350350

351351
## OCSF Event Classes in JSON
352352

0 commit comments

Comments
 (0)