Skip to content

Commit c4c020b

Browse files
committed
feat(gateway): accept relayed OTLP export data from supervisors
Add the gateway half of the supervisor OTLP telemetry relay, rebuilt on top of the RFC 0012 split-boundary architecture (NVIDIA#2942). Protocol: `SupervisorHello.capabilities` and `SessionAccepted.capabilities` negotiate optional supervisor features; `OtelExportData` is a new `SupervisorMessage` payload carrying OTLP trace bytes and OCSF events. Both repeated fields default to empty and the new oneof case is ignored by older peers, so the change is wire-compatible in both directions. Gateway: `confirm_capabilities` grants `otel_export` only when `[openshell.gateway.otlp]` produced a relay exporter. `handle_otel_export` forwards trace data through a dedicated `OtelRelayExporter` that bypasses the gateway's own tracer provider so supervisor-enriched resource attributes survive, and re-emits bounded, well-formed OCSF events on the `ocsf_relay` target. The capability selection and OCSF acceptance checks are pure functions with unit tests; the exporter has decode and URI tests that need no collector. The public RPC schema inventory grows by one message; the fingerprint and architecture/gateway.md are updated accordingly. The supervisor sends an empty capability list until its relay lands in a later commit. Refs NVIDIA#2641 Signed-off-by: Roland Huß <[email protected]>
1 parent 7e7a8d5 commit c4c020b

16 files changed

Lines changed: 494 additions & 8 deletions

File tree

‎Cargo.lock‎

Lines changed: 46 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ tracing-appender = "0.2"
7171
opentelemetry = "0.32"
7272
opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] }
7373
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["grpc-tonic", "trace"] }
74+
opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace", "with-serde"] }
7475
tracing-opentelemetry = { version = "0.33", default-features = false, features = ["tracing-log"] }
7576

7677
# Metrics

‎architecture/gateway.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -352,6 +352,8 @@ Public RPC contracts and durable protobuf formats have separate ownership. The `
352352

353353
`ReportEndpointStatus` is a sandbox-authenticated public gateway RPC. Its request, response, and `EndpointObservation` messages belong only to the public closure. `EndpointStatus` and `EndpointResult` also belong to the durable closure because `Sandbox.status.endpoint_statuses` persists them. The repeated status field uses a new wire tag; stored sandboxes without it decode with an empty endpoint list and retain their lifecycle fields. A fixed payload encoded with the earlier sandbox schema verifies that no database rewrite is required.
354354

355+
The `ConnectSupervisor` stream negotiates optional capabilities. `SupervisorHello.capabilities` advertises what a supervisor can do and `SessionAccepted.capabilities` echoes the subset the gateway will serve; both are new repeated fields that older peers leave empty. `OtelExportData` is a new `SupervisorMessage` payload case carrying relayed OTLP trace bytes and OCSF events, and the gateway only acts on it after confirming `otel_export`. All three belong only to the public closure; nothing on this stream is persisted.
356+
355357
The removed `NetworkBinary.harness` field remains reserved by number and name,
356358
so protobuf implementations cannot reuse its wire slot or source identifier.
357359
The durable-policy compatibility decoder reads the former boolean before Prost

‎crates/openshell-ocsf/src/lib.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ pub use builders::{
6464

6565
// --- Tracing layers ---
6666
pub use tracing_layers::{
67-
OCSF_TARGET, OcsfJsonlLayer, OcsfShorthandLayer, clear_current_event, clone_current_event,
68-
emit_ocsf_event, emit_ocsf_event_routed, set_current_event,
67+
OCSF_TARGET, OcsfJsonlLayer, OcsfRelayLayer, OcsfRelaySink, OcsfShorthandLayer,
68+
clear_current_event, clone_current_event, emit_ocsf_event, emit_ocsf_event_routed,
69+
set_current_event,
6970
};

‎crates/openshell-ocsf/src/tracing_layers/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@
99
1010
pub(crate) mod event_bridge;
1111
mod jsonl_layer;
12+
mod relay_layer;
1213
mod shorthand_layer;
1314

1415
pub use event_bridge::{
1516
OCSF_TARGET, clear_current_event, clone_current_event, emit_ocsf_event, emit_ocsf_event_routed,
1617
set_current_event,
1718
};
1819
pub use jsonl_layer::OcsfJsonlLayer;
20+
pub use relay_layer::{OcsfRelayLayer, OcsfRelaySink};
1921
pub use shorthand_layer::OcsfShorthandLayer;
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//! Tracing layer that captures OCSF events and forwards them as JSON bytes
5+
//! through a telemetry buffer sender for relay to the gateway.
6+
7+
use std::sync::Arc;
8+
9+
use tracing::Subscriber;
10+
use tracing_subscriber::Layer;
11+
use tracing_subscriber::layer::Context;
12+
13+
use super::event_bridge::{OCSF_TARGET, clone_current_event};
14+
15+
/// Callback trait for delivering serialized OCSF events.
16+
pub trait OcsfRelaySink: Send + Sync + 'static {
17+
fn send(&self, json_bytes: Vec<u8>);
18+
}
19+
20+
/// A tracing layer that captures OCSF events and serializes them to JSON
21+
/// for relay through the telemetry transport.
22+
pub struct OcsfRelayLayer {
23+
sink: Arc<dyn OcsfRelaySink>,
24+
}
25+
26+
impl OcsfRelayLayer {
27+
pub fn new(sink: Arc<dyn OcsfRelaySink>) -> Self {
28+
Self { sink }
29+
}
30+
}
31+
32+
impl<S: Subscriber> Layer<S> for OcsfRelayLayer {
33+
fn on_event(&self, event: &tracing::Event<'_>, _ctx: Context<'_, S>) {
34+
if event.metadata().target() != OCSF_TARGET {
35+
return;
36+
}
37+
38+
let Some(ocsf_event) = clone_current_event() else {
39+
return;
40+
};
41+
42+
if let Ok(json) = serde_json::to_vec(&ocsf_event) {
43+
self.sink.send(json);
44+
}
45+
}
46+
}

‎crates/openshell-otel/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
66
mod driver;
77
mod grpc;
8-
mod propagation;
8+
pub mod propagation;
99

1010
pub use driver::{
1111
BoxGrpcStream, ComputeDriverTracing, DriverTracingConfig, DriverTracingHandle,

‎crates/openshell-otel/src/propagation.rs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,22 @@ impl Injector for MetadataMapInjector<'_> {
5454
}
5555
}
5656

57+
/// Writes OpenTelemetry propagation fields to HTTP headers.
58+
#[derive(Debug)]
59+
pub struct HeaderMapInjector<'a>(pub &'a mut HeaderMap);
60+
61+
impl Injector for HeaderMapInjector<'_> {
62+
fn set(&mut self, key: &str, value: String) {
63+
let Ok(key) = key.parse::<http::header::HeaderName>() else {
64+
return;
65+
};
66+
let Ok(value) = value.parse() else {
67+
return;
68+
};
69+
self.0.insert(key, value);
70+
}
71+
}
72+
5773
#[derive(Debug)]
5874
struct TraceContextMapInjector<'a>(&'a mut BTreeMap<String, String>);
5975

@@ -75,6 +91,15 @@ pub fn current_trace_context_carrier() -> Option<BTreeMap<String, String>> {
7591
carrier.contains_key("traceparent").then_some(carrier)
7692
}
7793

94+
/// Inject W3C traceparent into HTTP headers if not already present.
95+
pub fn inject_traceparent_if_missing(headers: &mut HeaderMap) {
96+
if headers.contains_key("traceparent") {
97+
return;
98+
}
99+
let context = tracing::Span::current().context();
100+
TraceContextPropagator::new().inject_context(&context, &mut HeaderMapInjector(headers));
101+
}
102+
78103
/// Injects the active W3C trace context into an outbound tonic request.
79104
#[derive(Debug, Clone, Copy)]
80105
pub struct TraceContextInterceptor;

‎crates/openshell-server/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ tracing-appender = { workspace = true }
7575
# OpenTelemetry (OTLP trace export, opt-in via [openshell.gateway.otlp])
7676
opentelemetry = { workspace = true }
7777
opentelemetry_sdk = { workspace = true }
78+
opentelemetry-proto = { workspace = true }
7879
tracing-opentelemetry = { workspace = true }
7980

8081
# Metrics

‎crates/openshell-server/src/lib.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ mod grpc;
2525
mod http;
2626
mod middleware;
2727
mod multiplex;
28+
pub(crate) mod otel_relay;
2829
mod otel_tracing;
2930
mod pagination;
3031
mod persistence;
@@ -346,6 +347,10 @@ pub struct ServerState {
346347
/// Empty when OIDC is not configured — `authorize_workspace()` treats
347348
/// every authenticated user as Platform Admin in that case.
348349
pub admin_role: String,
350+
351+
/// Dedicated OTLP exporter for relayed telemetry from supervisors.
352+
/// `None` when the gateway has no OTLP endpoint configured.
353+
pub otel_relay_exporter: Option<Arc<otel_relay::OtelRelayExporter>>,
349354
}
350355

351356
fn is_benign_tls_handshake_failure(error: &std::io::Error) -> bool {
@@ -434,6 +439,7 @@ impl ServerState {
434439
provider_profile_sources:
435440
provider_profile_sources::ProviderProfileSources::with_default_sources(),
436441
admin_role,
442+
otel_relay_exporter: None,
437443
}
438444
}
439445
}
@@ -682,6 +688,7 @@ pub(crate) async fn run_server(
682688
state.middleware_registry = middleware_registry;
683689
state.gateway_interceptors = gateway_interceptors;
684690
state.provider_profile_sources = provider_profile_sources;
691+
state.otel_relay_exporter = otel_relay::try_create_exporter(config_file.as_ref()).await;
685692
state.sandbox_jwt_issuer = sandbox_jwt_issuer.clone();
686693
state.sandbox_jwt_authenticator = sandbox_jwt_authenticator;
687694
state.sandbox_session_jwt_authority = sandbox_session_jwt_authority;

0 commit comments

Comments
 (0)