Skip to content

Commit 11030e5

Browse files
committed
feat(observability): correlate HTTP receipts, audit records and traces
Closes NVIDIA#5393 Refs NVIDIA#5210 Signed-off-by: Alex Fournier <[email protected]>
1 parent 576d08f commit 11030e5

33 files changed

Lines changed: 8346 additions & 638 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ ocsf_emit!(event);
153153

154154
- `crate::ocsf_ctx()` returns the process-wide `EventContext`. It is always available (falls back to defaults in tests).
155155
- `ocsf_emit!()` is non-blocking and cannot panic. It stores the event in a thread-local and emits via `tracing::info!()`.
156-
- In `openshell-supervisor-network`, emit with `crate::telemetry::egress_emit!` instead, or `crate::telemetry::emit_decision` for the records of one decision, so denied HTTP decisions get a deny span. Clippy rejects `ocsf_emit!` there.
156+
- In `openshell-supervisor-network`, route inspected request records through `RequestCorrelation`, which uses their existing request span without creating a denial-only span. Other egress decisions use `crate::telemetry::egress_emit!` or `crate::telemetry::emit_decision`. Clippy rejects direct `ocsf_emit!` calls there.
157157
- The shorthand layer and JSONL layer extract the event from the thread-local. The shorthand format is derived automatically from the builder fields.
158158
- For security findings, **dual-emit**: one domain event (e.g., `SshActivityBuilder`) AND one `DetectionFindingBuilder` for the same incident.
159159
- Never log secrets, credentials, or query parameters in OCSF messages. The OCSF JSONL file may be shipped to external systems.

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/openshell-ocsf/src/events/mod.rs‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,11 @@ impl OcsfEvent {
169169
}
170170
}
171171

172+
/// Associate this event with other events produced for the same operation.
173+
pub fn set_correlation_uid(&mut self, uid: impl Into<String>) {
174+
self.base_mut().metadata.correlation_uid = Some(uid.into());
175+
}
176+
172177
/// Returns mutable base data common to all event classes.
173178
pub(crate) fn base_mut(&mut self) -> &mut BaseEventData {
174179
match self {
@@ -281,6 +286,29 @@ mod tests {
281286
assert_eq!(deserialized.class_uid(), 2004);
282287
}
283288

289+
#[test]
290+
fn correlation_uid_can_be_set_on_any_event_class() {
291+
let ctx = test_sandbox_context();
292+
let mut event = AuthenticationBuilder::new(
293+
&ctx,
294+
User::other("user-1", "User"),
295+
Service::new("gateway"),
296+
)
297+
.status(StatusId::Success)
298+
.build();
299+
300+
event.set_correlation_uid("request-123");
301+
302+
assert_eq!(
303+
event.base().metadata.correlation_uid.as_deref(),
304+
Some("request-123")
305+
);
306+
assert_eq!(
307+
serde_json::to_value(&event).unwrap()["metadata"]["correlation_uid"],
308+
"request-123"
309+
);
310+
}
311+
284312
#[test]
285313
fn test_roundtrip_application_lifecycle() {
286314
let ctx = test_sandbox_context();

‎crates/openshell-otel/src/ocsf.rs‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ const MAX_EXPORT_ATTEMPTS: usize = 4;
3434
const INITIAL_BACKOFF: Duration = Duration::from_millis(250);
3535
const MAX_BACKOFF: Duration = Duration::from_secs(2);
3636
const GAP_EVENT: &str = "openshell.ocsf.gap";
37+
const CORRELATION_UID_ATTRIBUTE: &str = "ocsf.metadata.correlation_uid";
3738

3839
/// Exports OCSF events to an OTLP collector as log records.
3940
pub struct OcsfOtlp {
@@ -391,6 +392,9 @@ fn log_record(
391392
if let Some(uid) = &base.metadata.uid {
392393
record.add_attribute("ocsf.metadata.uid", uid.clone());
393394
}
395+
if let Some(uid) = &base.metadata.correlation_uid {
396+
record.add_attribute(CORRELATION_UID_ATTRIBUTE, uid.clone());
397+
}
394398
if let Some(sandbox_id) = event.sandbox_id() {
395399
record.add_attribute("openshell.sandbox.id", sandbox_id.to_string());
396400
}
@@ -629,6 +633,16 @@ mod tests {
629633
assert_eq!(native(&event)["resources"][0]["uid"], "requested-sandbox");
630634
}
631635

636+
#[test]
637+
fn correlation_uid_is_a_stable_otlp_attribute() {
638+
let mut event = event("sandbox-1", SeverityId::Informational);
639+
event.set_correlation_uid("request-123");
640+
assert_eq!(
641+
attribute(&record(&event), CORRELATION_UID_ATTRIBUTE),
642+
Some(AnyValue::String("request-123".into()))
643+
);
644+
}
645+
632646
#[test]
633647
fn events_without_a_sandbox_carry_no_sandbox_attribute() {
634648
let record = record(&event("", SeverityId::Informational));
@@ -788,12 +802,23 @@ mod tests {
788802
.with(otlp.layer());
789803
let span_context = tracing::subscriber::with_default(subscriber, || {
790804
let span = tracing::info_span!("decision");
791-
span.in_scope(|| ocsf_emit!(event("sandbox-1", SeverityId::Medium)));
805+
let mut event = event("sandbox-1", SeverityId::Medium);
806+
event.set_correlation_uid("request-with-or-without-sampling");
807+
span.in_scope(|| ocsf_emit!(event));
792808
span.context().span().span_context().clone()
793809
});
794810
otlp.shutdown().await;
795811

796812
let records = exporter.exported();
813+
assert_eq!(records.len(), 1);
814+
assert_eq!(
815+
attribute(&records[0].record, CORRELATION_UID_ATTRIBUTE),
816+
Some(AnyValue::String("request-with-or-without-sampling".into()))
817+
);
818+
assert_eq!(
819+
body(&records[0].record)["metadata"]["correlation_uid"],
820+
"request-with-or-without-sampling"
821+
);
797822
let trace = records[0].record.trace_context();
798823
if span_context.is_sampled() {
799824
let trace = trace.expect("sampled span context is attached");

‎crates/openshell-supervisor-middleware/src/headers.rs‎

Lines changed: 60 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -366,25 +366,31 @@ fn is_safe_value(value: &str) -> bool {
366366
}
367367

368368
fn is_request_protected(name: &str) -> bool {
369-
matches!(
370-
name,
371-
"authorization"
372-
| "proxy-authorization"
373-
| "proxy-authenticate"
374-
| "cookie"
375-
| "host"
376-
| "content-length"
377-
| "transfer-encoding"
378-
| "connection"
379-
| "proxy-connection"
380-
| "keep-alive"
381-
| "te"
382-
| "trailer"
383-
| "upgrade"
384-
) || name.starts_with("x-amz-")
369+
is_platform_correlation_header(name)
370+
|| matches!(
371+
name,
372+
"authorization"
373+
| "proxy-authorization"
374+
| "proxy-authenticate"
375+
| "cookie"
376+
| "host"
377+
| "content-length"
378+
| "transfer-encoding"
379+
| "connection"
380+
| "proxy-connection"
381+
| "keep-alive"
382+
| "te"
383+
| "trailer"
384+
| "upgrade"
385+
)
386+
|| name.starts_with("x-amz-")
385387
|| name.starts_with("x-openshell-credential")
386388
}
387389

390+
fn is_platform_correlation_header(name: &str) -> bool {
391+
name == "x-openshell-request-id"
392+
}
393+
388394
/// Return whether a response header can carry authentication material and must
389395
/// never be exposed to middleware.
390396
#[must_use]
@@ -402,7 +408,8 @@ pub fn is_response_credential_header(name: &str) -> bool {
402408
}
403409

404410
fn is_response_protected(name: &str) -> bool {
405-
is_response_credential_header(name)
411+
is_platform_correlation_header(name)
412+
|| is_response_credential_header(name)
406413
|| matches!(
407414
name,
408415
"connection"
@@ -488,6 +495,42 @@ mod tests {
488495
);
489496
}
490497

498+
#[test]
499+
fn platform_request_id_is_immutable_for_all_http_phases() {
500+
for authority in [
501+
HeaderAuthority::Request,
502+
HeaderAuthority::HookRequest,
503+
HeaderAuthority::RequestTrailers,
504+
HeaderAuthority::Response,
505+
HeaderAuthority::HookResponse,
506+
HeaderAuthority::ResponseTrailers,
507+
] {
508+
let name = "X-OpenShell-Request-ID";
509+
let write_error = apply(
510+
authority,
511+
&[],
512+
&[],
513+
&[write(name, "forged", ExistingHeaderAction::Overwrite)],
514+
)
515+
.expect_err("platform request ID write");
516+
assert_eq!(
517+
write_error,
518+
HeaderMutationError::Protected {
519+
name: name.to_string()
520+
}
521+
);
522+
523+
let remove_error = apply(authority, &[], &[], &[remove(name)])
524+
.expect_err("platform request ID removal");
525+
assert_eq!(
526+
remove_error,
527+
HeaderMutationError::Protected {
528+
name: name.to_string()
529+
}
530+
);
531+
}
532+
}
533+
491534
#[test]
492535
fn unsafe_header_value_is_rejected() {
493536
let error = apply(

‎crates/openshell-supervisor-network/Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ hex = "0.4"
3838
hickory-proto = "0.26.1"
3939
ipnet = "2"
4040
miette = { workspace = true }
41+
opentelemetry = { workspace = true }
42+
opentelemetry_sdk = { workspace = true }
4143
prost-types = { workspace = true }
4244
rcgen = { workspace = true }
4345
regorus = { version = "0.9", default-features = false, features = ["std", "arc", "glob"] }
@@ -56,6 +58,7 @@ tokio-rustls = { workspace = true }
5658
tonic = { workspace = true }
5759
tower-mcp-types = { workspace = true }
5860
tracing = { workspace = true }
61+
tracing-opentelemetry = { workspace = true }
5962
uuid = { workspace = true }
6063
webpki-roots = { workspace = true, optional = true }
6164

‎crates/openshell-supervisor-network/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,31 @@ domain event and a detection finding without credential values or placeholders.
5555
Explicit body and WebSocket
5656
rewrite flags authorize those locations for ordinary static credentials. SigV4
5757
signing retains the endpoint-scoped resolver's existing trusted access.
58+
59+
## HTTP enforcement correlation
60+
61+
Each inspected HTTP request has one UUID shared by middleware, OCSF
62+
`metadata.correlation_uid`, and the protected `x-openshell-request-id` response
63+
header. Application correlation must use another header. The receipt is only a
64+
join key: it does not establish identity, authorization or complete delivery.
65+
L4 and passthrough traffic do not provide this header guarantee.
66+
67+
The audit state commits one primary HTTP Activity record when the final response
68+
head is written, with an early-termination fallback. It retains the original
69+
request and process context. Middleware invocation records remain separate,
70+
request-correlated observations, including after the primary is committed. They
71+
retain attribution and transformation details without changing the final
72+
response status or consuming the late-failure budget. Security findings remain
73+
separate. A process crash or indefinitely hung request cannot be finalized by
74+
the fallback.
75+
76+
Request IDs and audit outcomes do not depend on OpenTelemetry, sampling or
77+
collector availability. Inspected requests have a supervisor-owned request span
78+
and children around stages that actually run, with valid workload W3C context
79+
only as an untrusted link. Their records never create an extra denial-only span,
80+
even if the request span is disabled or unsampled. Other egress paths retain the
81+
legacy bounded deny-span behavior. End HTTP tracing before an upgraded session.
82+
The request span ends after its owned stages, not when response headers are logged.
83+
The `openshell.upstream.send` stage measures request writes and flush; response
84+
handling is measured separately. Receipt filtering preserves raw HTTP bytes
85+
and borrows unchanged header blocks.

0 commit comments

Comments
 (0)