Skip to content

Commit f5784ab

Browse files
committed
feat(observability): supervisor OTLP telemetry relay
Add an OTLP relay to the sandbox supervisor that accepts agent-emitted traces on 127.0.0.1:4318, enriches spans with sandbox resource attributes, and forwards them to the gateway over the existing session protocol. The relay enables zero-config observability for OTel-instrumented agents inside network-isolated sandboxes. Agents export to the standard OTLP endpoint; the supervisor handles enrichment, buffering, and forwarding without requiring egress policy exceptions. Key components: - OTLP HTTP receiver (protobuf + JSON) with per-driver binding (netns for Docker/Podman, direct for K8s/VM) - Span enrichment with sandbox identity (sandbox_id, workspace_id, policy, user, image, driver) and telemetry source marker - Bounded buffer (4096 slots) with non-blocking try_send forwarding - Gateway-side TelemetryRelayExporter with dedicated gRPC client that preserves supervisor-enriched resource attributes - Capability negotiation: supervisor advertises telemetry_relay, gateway confirms only when OTLP export is configured - Graceful shutdown ordering: relay drains before network teardown Proto: TelemetryData message, capabilities on SupervisorHello and SessionAccepted. Closes: NVIDIA#2641 Signed-off-by: Roland Huß <[email protected]>
1 parent 592df3e commit f5784ab

30 files changed

Lines changed: 2156 additions & 1571 deletions

File tree

‎Cargo.lock‎

Lines changed: 864 additions & 974 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ tracing-appender = "0.2"
6666
opentelemetry = "0.32"
6767
opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] }
6868
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["grpc-tonic", "trace"] }
69+
opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace", "with-serde"] }
6970
tracing-opentelemetry = { version = "0.33", default-features = false, features = ["tracing-log"] }
7071

7172
# Metrics

‎architecture/sandbox.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -462,6 +462,72 @@ Sandbox logs are emitted locally and can also be pushed back to the gateway.
462462
Security-relevant sandbox behavior uses OCSF structured events; internal
463463
diagnostics use ordinary tracing.
464464

465+
## Telemetry Relay
466+
467+
The supervisor can relay OpenTelemetry trace data and OCSF events from agent
468+
processes to the gateway over the session protocol. This gives OTel-instrumented
469+
agents (LangChain, CrewAI, etc.) a zero-configuration path to an external
470+
collector without requiring direct egress from the sandbox.
471+
472+
### Data Flow
473+
474+
```
475+
Agent process --> OTLP HTTP (127.0.0.1:4318) --> Supervisor receiver
476+
--> Enrichment (sandbox resource attributes)
477+
--> Bounded buffer (4096 slots, shared traces + OCSF)
478+
--> Forwarder --> Session channel (TelemetryData message)
479+
--> Gateway --> Dedicated SpanExporter --> External OTLP collector
480+
```
481+
482+
### Receiver Binding
483+
484+
The OTLP HTTP receiver always binds to `127.0.0.1:4318`. For Docker/Podman
485+
drivers where the supervisor creates a network namespace, the bind happens
486+
inside the namespace via `bind_tcp_in_netns()`. For Kubernetes and VM drivers,
487+
the supervisor and agent share the same network namespace, so a direct bind
488+
suffices. The receiver accepts both `application/x-protobuf` and
489+
`application/json` content types.
490+
491+
### Span Enrichment
492+
493+
Forwarded spans are enriched with sandbox resource attributes:
494+
`openshell.sandbox.id`, `openshell.workspace.id`, `openshell.sandbox.policy`,
495+
`openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`.
496+
The `openshell.telemetry.source` attribute (fixed value `"agent"`) is always
497+
injected regardless of the enrichment toggle so collectors can filter agent
498+
spans from gateway infrastructure spans. Enrichment can be disabled for
499+
pass-through forwarding.
500+
501+
### Activation and Capability Negotiation
502+
503+
The relay is opt-in. It starts only when the gateway has `[openshell.gateway.otlp]`
504+
configured. The supervisor advertises `"telemetry_relay"` in
505+
`SupervisorHello.capabilities`. The gateway confirms via `SessionAccepted.capabilities`.
506+
The supervisor gates `TelemetryData` sending on this confirmation. When the
507+
relay is active, the supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT` and
508+
`OTEL_EXPORTER_OTLP_PROTOCOL` in agent child processes via `child_env.rs`.
509+
510+
### Non-Interference
511+
512+
The buffer uses `try_send` (non-blocking) on the session channel. When the
513+
buffer reaches capacity, the oldest entries are dropped and a counter records
514+
each drop. A queue depth gauge tracks buffer pressure. This ensures telemetry
515+
cannot block or degrade sandbox control operations.
516+
517+
### OCSF Event Relay
518+
519+
OCSF events generated inside the sandbox (e.g., network deny events) can also
520+
be forwarded through the same transport. A per-sandbox token bucket rate limiter
521+
controls the OCSF event rate, with configurable rate and drop counter.
522+
523+
### Gateway-Side Handling
524+
525+
The gateway receives `TelemetryData` messages and exports trace data through a
526+
dedicated `TelemetryRelayExporter` that connects directly to the configured OTLP
527+
collector. This bypasses the gateway's own `SdkTracerProvider` to preserve the
528+
supervisor-enriched resource attributes. OCSF events are emitted via
529+
`tracing::info!` on the `ocsf_relay` target.
530+
465531
## Policy Proposals
466532

467533
When an L4 CONNECT is denied, the proxy emits a `DenialEvent`. The denial

‎crates/openshell-core/src/sandbox_env.rs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,18 @@ pub const SANDBOX_GID: &str = "OPENSHELL_SANDBOX_GID";
235235
/// OCI only for the former contract.
236236
pub const OCI_IMAGE_USER: &str = "OPENSHELL_OCI_IMAGE_USER";
237237

238+
/// Standard OpenTelemetry environment variable for the OTLP exporter endpoint.
239+
///
240+
/// Set conditionally by the telemetry relay when the gateway has an OTLP
241+
/// endpoint configured. Points agent SDKs at the supervisor's local OTLP
242+
/// HTTP receiver.
243+
pub const OTEL_EXPORTER_OTLP_ENDPOINT: &str = "OTEL_EXPORTER_OTLP_ENDPOINT";
244+
245+
/// Standard OpenTelemetry environment variable for the OTLP exporter protocol.
246+
///
247+
/// Set to `http/protobuf` when the telemetry relay is active.
248+
pub const OTEL_EXPORTER_OTLP_PROTOCOL: &str = "OTEL_EXPORTER_OTLP_PROTOCOL";
249+
238250
// The corporate upstream-proxy configuration deliberately has no reserved
239251
// environment variables: it travels on the supervisor's argv
240252
// (`--upstream-proxy` and friends), which a sandbox image cannot forge the

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4341,6 +4341,16 @@ fn build_env_list(
43414341
tls_enabled,
43424342
provider_spiffe_socket_path,
43434343
);
4344+
upsert_env(
4345+
&mut env,
4346+
openshell_core::sandbox_env::OTEL_EXPORTER_OTLP_ENDPOINT,
4347+
"http://127.0.0.1:4318",
4348+
);
4349+
upsert_env(
4350+
&mut env,
4351+
openshell_core::sandbox_env::OTEL_EXPORTER_OTLP_PROTOCOL,
4352+
"http/protobuf",
4353+
);
43444354
env
43454355
}
43464356

0 commit comments

Comments
 (0)