Skip to content

Commit b595d70

Browse files
committed
feat(observability): restore supervisor-owned HTTP request traces
Signed-off-by: Alex Fournier <[email protected]>
1 parent 81c9f05 commit b595d70

21 files changed

Lines changed: 2049 additions & 133 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ ocsf_emit!(event);
153153

154154
- `crate::ocsf_ctx()` returns the process-wide `EventContext`. It is always available (falls back to defaults in tests).
155155
- `ocsf_emit!()` is non-blocking and cannot panic. It stores the event in a thread-local and emits via `tracing::info!()`.
156-
- In `openshell-supervisor-network`, emit with `crate::telemetry::egress_emit!` instead, or `crate::telemetry::emit_decision` for the records of one decision, so denied HTTP decisions get a deny span. Clippy rejects `ocsf_emit!` there.
156+
- In `openshell-supervisor-network`, route inspected request records through `RequestCorrelation`, which uses their existing request span without creating a denial-only span. Other egress decisions use `crate::telemetry::egress_emit!` or `crate::telemetry::emit_decision`. Clippy rejects direct `ocsf_emit!` calls there.
157157
- The shorthand layer and JSONL layer extract the event from the thread-local. The shorthand format is derived automatically from the builder fields.
158158
- For security findings, **dual-emit**: one domain event (e.g., `SshActivityBuilder`) AND one `DetectionFindingBuilder` for the same incident.
159159
- Never log secrets, credentials, or query parameters in OCSF messages. The OCSF JSONL file may be shipped to external systems.

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/openshell-supervisor-network/Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ hex = "0.4"
3838
hickory-proto = "0.26.1"
3939
ipnet = "2"
4040
miette = { workspace = true }
41+
opentelemetry = { workspace = true }
42+
opentelemetry_sdk = { workspace = true }
4143
prost-types = { workspace = true }
4244
rcgen = { workspace = true }
4345
regorus = { version = "0.9", default-features = false, features = ["std", "arc", "glob"] }
@@ -56,6 +58,7 @@ tokio-rustls = { workspace = true }
5658
tonic = { workspace = true }
5759
tower-mcp-types = { workspace = true }
5860
tracing = { workspace = true }
61+
tracing-opentelemetry = { workspace = true }
5962
uuid = { workspace = true }
6063
webpki-roots = { workspace = true, optional = true }
6164

‎crates/openshell-supervisor-network/README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,9 @@ outcomes. Routine outcomes do not add records; existing security findings and
6767
post-header failure records remain distinct. A process crash or indefinitely
6868
hung request cannot be finalized by the fallback.
6969

70-
Request IDs do not depend on OpenTelemetry or collector availability. This layer
71-
adds no request spans or workload-controlled supervisor trace context and keeps
72-
the existing bounded deny-span path. Request-stage tracing is a separate layer
73-
that requires operator-controlled supervisor sampling.
70+
Request IDs and audit outcomes do not depend on OpenTelemetry, sampling or
71+
collector availability. Inspected requests have a supervisor-owned request span
72+
and children around stages that actually run, with valid workload W3C context
73+
only as an untrusted link. Their records never create an extra denial-only span,
74+
even if the request span is disabled or unsampled. Other egress paths retain the
75+
legacy bounded deny-span behavior. End HTTP tracing before an upgraded session.

‎crates/openshell-supervisor-network/src/l7/correlation.rs‎

Lines changed: 81 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
1717

1818
use crate::l7::provider::L7Request;
1919
use crate::l7::relay::L7EvalContext;
20+
pub use crate::l7::request_trace::RequestStage;
21+
use crate::l7::request_trace::RequestTrace;
2022

2123
#[derive(Debug)]
2224
pub struct RequestCorrelation {
@@ -28,6 +30,7 @@ pub struct RequestCorrelation {
2830

2931
#[derive(Debug, Default)]
3032
struct RequestState {
33+
trace: Option<RequestTrace>,
3134
primary: Option<OcsfEvent>,
3235
seeded: bool,
3336
status: Option<u16>,
@@ -72,10 +75,18 @@ impl RequestCorrelation {
7275
.dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port))
7376
.message("HTTP request ended before a response was selected")
7477
.build();
78+
let trace = RequestTrace::new(
79+
&request_id,
80+
&request.action,
81+
&ctx.host,
82+
ctx.port,
83+
&request.raw_header,
84+
);
7585
Self {
7686
request_id,
7787
inspected: true,
7888
state: Box::new(Mutex::new(RequestState {
89+
trace: Some(trace),
7990
primary: Some(fallback),
8091
..RequestState::default()
8192
})),
@@ -87,6 +98,41 @@ impl RequestCorrelation {
8798
&self.request_id
8899
}
89100

101+
pub(crate) fn stage(&self, stage: RequestStage) -> tracing::Span {
102+
self.state
103+
.lock()
104+
.unwrap_or_else(std::sync::PoisonError::into_inner)
105+
.trace
106+
.as_ref()
107+
.map_or_else(tracing::Span::none, |trace| trace.stage(stage))
108+
}
109+
110+
#[cfg(test)]
111+
pub(crate) fn trace_span(&self) -> tracing::Span {
112+
self.state
113+
.lock()
114+
.unwrap_or_else(std::sync::PoisonError::into_inner)
115+
.trace
116+
.as_ref()
117+
.map_or_else(tracing::Span::none, RequestTrace::span)
118+
}
119+
120+
pub(crate) fn finish_http_trace(&self) {
121+
let (mut trace, status) = {
122+
let mut state = self
123+
.state
124+
.lock()
125+
.unwrap_or_else(std::sync::PoisonError::into_inner);
126+
(state.trace.take(), state.status)
127+
};
128+
if status == Some(101)
129+
&& let Some(trace) = &mut trace
130+
{
131+
trace.record_outcome("upgraded");
132+
}
133+
drop(trace);
134+
}
135+
90136
pub(crate) fn observe_response<'a, C>(&'a self, client: &'a mut C) -> AuditedClient<'a, C> {
91137
AuditedClient {
92138
inner: client,
@@ -103,17 +149,25 @@ impl RequestCorrelation {
103149
}
104150

105151
pub(crate) fn record_final_http_outcome(&self, outcome: &'static str) {
106-
self.state
152+
let mut state = self
153+
.state
107154
.lock()
108-
.unwrap_or_else(std::sync::PoisonError::into_inner)
109-
.outcome = Some(outcome);
155+
.unwrap_or_else(std::sync::PoisonError::into_inner);
156+
state.outcome = Some(outcome);
157+
if let Some(trace) = &mut state.trace {
158+
trace.record_outcome(outcome);
159+
}
110160
}
111161

112162
pub(crate) fn record_response_status(&self, status: u16) {
113-
self.state
163+
let mut state = self
164+
.state
114165
.lock()
115-
.unwrap_or_else(std::sync::PoisonError::into_inner)
116-
.status = Some(status);
166+
.unwrap_or_else(std::sync::PoisonError::into_inner);
167+
state.status = Some(status);
168+
if let Some(trace) = &state.trace {
169+
trace.record_status(status);
170+
}
117171
}
118172

119173
pub(crate) fn record_upstream_status(&self, status: u16) {
@@ -124,10 +178,14 @@ impl RequestCorrelation {
124178
}
125179

126180
pub(crate) fn record_failure_reason(&self, reason: &'static str) {
127-
self.state
181+
let mut state = self
182+
.state
128183
.lock()
129-
.unwrap_or_else(std::sync::PoisonError::into_inner)
130-
.failure_reason = Some(reason);
184+
.unwrap_or_else(std::sync::PoisonError::into_inner);
185+
state.failure_reason = Some(reason);
186+
if let Some(trace) = &state.trace {
187+
trace.record_failure_reason(reason);
188+
}
131189
}
132190

133191
pub(crate) fn emit(&self, event: OcsfEvent) {
@@ -271,7 +329,19 @@ impl RequestCorrelation {
271329

272330
fn emit_correlated(&self, mut event: OcsfEvent) {
273331
event.set_correlation_uid(self.request_id.clone());
274-
crate::telemetry::emit_decision([event], None);
332+
let parent = {
333+
let mut state = self
334+
.state
335+
.lock()
336+
.unwrap_or_else(std::sync::PoisonError::into_inner);
337+
state.trace.as_mut().map(|trace| {
338+
if let OcsfEvent::HttpActivity(http) = &event {
339+
trace.observe_http(http);
340+
}
341+
trace.span()
342+
})
343+
};
344+
crate::telemetry::emit_inspected_decision([event], parent.as_ref());
275345
}
276346
}
277347

@@ -451,6 +521,7 @@ impl<C: AsyncWrite + Unpin> AsyncWrite for AuditedClient<'_, C> {
451521
impl Drop for RequestCorrelation {
452522
fn drop(&mut self) {
453523
self.emit_primary(true);
524+
self.finish_http_trace();
454525
}
455526
}
456527

‎crates/openshell-supervisor-network/src/l7/correlation/tests.rs‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,135 @@ fn http_events(capture: &openshell_ocsf::test_support::OcsfCapture) -> Vec<serde
6262
.collect()
6363
}
6464

65+
#[test]
66+
fn audit_receipt_and_denial_are_independent_of_sampling_or_exporter() {
67+
use opentelemetry::trace::TracerProvider as _;
68+
use opentelemetry_sdk::trace::{InMemorySpanExporter, Sampler, SdkTracerProvider};
69+
use tracing_subscriber::layer::SubscriberExt as _;
70+
71+
crate::telemetry::tests::init_trace_extractor();
72+
for sampler in [Some(Sampler::AlwaysOn), Some(Sampler::AlwaysOff), None] {
73+
let sampled = matches!(sampler, Some(Sampler::AlwaysOn));
74+
let exporter = InMemorySpanExporter::default();
75+
let provider = sampler.map(|sampler| {
76+
SdkTracerProvider::builder()
77+
.with_sampler(sampler)
78+
.with_simple_exporter(exporter.clone())
79+
.build()
80+
});
81+
let layer = provider.as_ref().map(|provider| {
82+
tracing_opentelemetry::layer().with_tracer(provider.tracer("request-audit-test"))
83+
});
84+
let capture = openshell_ocsf::test_support::OcsfCapture::default();
85+
let subscriber = tracing_subscriber::registry()
86+
.with(capture.clone())
87+
.with(layer);
88+
tracing::subscriber::with_default(subscriber, || {
89+
let correlation = RequestCorrelation::with_request_id(
90+
&request(),
91+
&context(),
92+
"00000000-0000-4000-8000-000000000001".into(),
93+
);
94+
correlation.emit(allowed());
95+
correlation.emit(denied());
96+
correlation.record_final_http_outcome("denied");
97+
correlation.response_selected(403);
98+
});
99+
let events = http_events(&capture);
100+
assert_eq!(events.len(), 1);
101+
let event = &events[0];
102+
assert_eq!(
103+
event["metadata"]["correlation_uid"],
104+
"00000000-0000-4000-8000-000000000001"
105+
);
106+
assert_eq!(event["action"], "Denied");
107+
assert_eq!(event["status"], "Failure");
108+
assert_eq!(event["http_response"]["code"], 403);
109+
assert_eq!(event["http_request"]["url"]["path"], "/data");
110+
assert_eq!(event["trace"]["uid"].is_string(), sampled);
111+
if let Some(provider) = provider {
112+
provider.force_flush().unwrap();
113+
let spans = exporter.get_finished_spans().unwrap();
114+
assert_eq!(spans.len(), usize::from(sampled));
115+
if sampled {
116+
assert_eq!(spans[0].name, "openshell.http.request");
117+
assert_eq!(
118+
event["trace"]["uid"],
119+
spans[0].span_context.trace_id().to_string()
120+
);
121+
}
122+
provider.shutdown().unwrap();
123+
}
124+
}
125+
}
126+
127+
#[test]
128+
fn finished_http_trace_is_not_retained_by_an_upgraded_session() {
129+
use opentelemetry::trace::TracerProvider as _;
130+
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider};
131+
use tracing_subscriber::layer::SubscriberExt as _;
132+
133+
let exporter = InMemorySpanExporter::default();
134+
let provider = SdkTracerProvider::builder()
135+
.with_simple_exporter(exporter.clone())
136+
.build();
137+
let subscriber = tracing_subscriber::registry()
138+
.with(tracing_opentelemetry::layer().with_tracer(provider.tracer("request-upgrade-test")));
139+
tracing::subscriber::with_default(subscriber, || {
140+
let correlation = RequestCorrelation::new(&request(), &context());
141+
correlation.emit(allowed());
142+
correlation.response_selected(101);
143+
correlation.finish_http_trace();
144+
assert_eq!(exporter.get_finished_spans().unwrap().len(), 1);
145+
assert!(correlation.trace_span().is_disabled());
146+
assert!(
147+
correlation
148+
.stage(RequestStage::UpstreamExchange)
149+
.is_disabled()
150+
);
151+
correlation.finish_http_trace();
152+
drop(correlation);
153+
assert_eq!(exporter.get_finished_spans().unwrap().len(), 1);
154+
});
155+
provider.shutdown().unwrap();
156+
}
157+
158+
#[test]
159+
fn a_provider_without_an_exporter_still_emits_the_request_id_join() {
160+
use opentelemetry::trace::TracerProvider as _;
161+
use opentelemetry_sdk::trace::{Sampler, SdkTracerProvider};
162+
use tracing_subscriber::layer::SubscriberExt as _;
163+
164+
crate::telemetry::tests::init_trace_extractor();
165+
let provider = SdkTracerProvider::builder()
166+
.with_sampler(Sampler::AlwaysOn)
167+
.build();
168+
let capture = openshell_ocsf::test_support::OcsfCapture::default();
169+
let subscriber = tracing_subscriber::registry()
170+
.with(capture.clone())
171+
.with(tracing_opentelemetry::layer().with_tracer(provider.tracer("no-exporter-test")));
172+
let mut id = None;
173+
tracing::subscriber::with_default(subscriber, || {
174+
let correlation = RequestCorrelation::new(&request(), &context());
175+
id = Some(correlation.request_id().to_owned());
176+
assert_eq!(
177+
correlation
178+
.stage(RequestStage::PolicyEvaluate)
179+
.in_scope(|| 42),
180+
42
181+
);
182+
correlation.emit(allowed());
183+
correlation.record_final_http_outcome("allowed");
184+
correlation.response_selected(200);
185+
});
186+
let events = http_events(&capture);
187+
assert_eq!(events.len(), 1);
188+
assert_eq!(events[0]["metadata"]["correlation_uid"], id.unwrap());
189+
assert_eq!(events[0]["http_response"]["code"], 200);
190+
assert_eq!(events[0]["action"], "Allowed");
191+
provider.shutdown().unwrap();
192+
}
193+
65194
#[test]
66195
fn request_ids_are_unique_canonical_uuids() {
67196
let capture = openshell_ocsf::test_support::OcsfCapture::default();

0 commit comments

Comments
 (0)