Skip to content

Commit c6e9dc7

Browse files
committed
docs(observability): describe the OTLP relay on the split boundary
Document the relay as it works on the RFC 0012 architecture: the reserved unroutable destination, the seccomp-staged path into the supervisor, the gateway-side environment injection, capability gating, buffering, drain bounds, and the enriched resource attributes. The Kubernetes runtime page notes that agent telemetry crosses the Pod boundary over the existing channel with no NetworkPolicy change, and the gateway config reference explains what `[openshell.gateway.otlp]` now enables for sandboxes. The AGENTS.md row for openshell-supervisor-process reflects what the crate owns after the isolation split. Refs NVIDIA#2641 Signed-off-by: Roland Huß <[email protected]>
1 parent 3e51e8f commit c6e9dc7

4 files changed

Lines changed: 78 additions & 1 deletion

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
6969
| `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |
7070
| `crates/openshell-supervisor-middleware-builtins/` | Built-in middleware | First-party in-process middleware implementations |
7171
| `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and provider credential injection |
72-
| `crates/openshell-supervisor-process/` | Process supervisor | Process lifecycle, namespace, and bypass monitoring |
72+
| `crates/openshell-supervisor-process/` | Process supervisor | SSH access, gateway session, log forwarding, and the OTLP telemetry relay |
7373
| `crates/openshell-vfio/` | VFIO support | PCI and GPU passthrough preparation and lifecycle |
7474
| `python/openshell/` | Python SDK | Python bindings and CLI packaging |
7575
| `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |

‎architecture/sandbox.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -582,6 +582,66 @@ connection context use Network Activity. Producer regression tests validate
582582
required fields and `at_least_one` constraints against the vendored OCSF 1.8
583583
schemas.
584584

585+
## Telemetry Relay
586+
587+
The supervisor relays OpenTelemetry trace data from agent processes to the
588+
gateway over the session stream, so OTel-instrumented agents reach an
589+
external collector without any sandbox egress.
590+
591+
The relay is opt-in on the gateway. When `[openshell.gateway.otlp]` is
592+
configured, `CreateSandbox` sets `OTEL_EXPORTER_OTLP_ENDPOINT` to
593+
`http://192.0.0.8:4318` and `OTEL_EXPORTER_OTLP_PROTOCOL` to `http/protobuf`
594+
in the sandbox environment unless the caller already set an endpoint. The
595+
values persist in the stored spec, so restarts inherit them.
596+
597+
`192.0.0.8` is the RFC 7600 dummy address and is never routed. A workload
598+
connect to any non-loopback address is intercepted by the sandbox seccomp
599+
broker and staged for the supervisor as a `PendingTcpOpen`, so the address is
600+
a label the supervisor switches on. The proxy recognises it ahead of host
601+
mapping, policy, and SSRF validation and hands the staged stream to the OTLP
602+
receiver instead of dialing upstream. The receiver speaks HTTP/1.1 on that
603+
stream and accepts `POST /v1/traces` as protobuf or JSON. No socket is bound,
604+
the driver outer fence is untouched, and the isolation contract and boundary
605+
protocol are unchanged. Loopback cannot serve this purpose because the broker
606+
completes loopback connects locally without mediation.
607+
608+
```
609+
Agent process --> connect(192.0.0.8:4318) --> seccomp broker stages the open
610+
--> supervisor proxy, reserved destination --> OTLP receiver
611+
--> enrichment (openshell.sandbox.* resource attributes)
612+
--> bounded buffer (4096 items, newest dropped when full)
613+
--> supervisor session (OtelExportData) --> gateway
614+
--> dedicated OtelRelayExporter --> external OTLP collector
615+
```
616+
617+
The relay starts before networking, so the first staged stream finds it. The
618+
supervisor advertises `otel_export` in `SupervisorHello`; the gateway confirms
619+
it in `SessionAccepted` only when it has a relay exporter. Forwarding is gated
620+
per session on that confirmation. Items received before a confirming session,
621+
or while a session declines, wait in the bounded buffer. The receiver serves
622+
at most 64 concurrent connections and refuses further opens before the
623+
sandbox commits the socket, which the agent observes as `EAGAIN`.
624+
625+
Forwarded spans gain `openshell.sandbox.id`, `openshell.workspace.id`,
626+
`openshell.sandbox.policy`, `openshell.sandbox.user`,
627+
`openshell.sandbox.image`, and `openshell.sandbox.driver`, and always
628+
`openshell.telemetry.source=agent` so collectors can separate agent spans
629+
from gateway spans. Agent-supplied values for these keys are replaced.
630+
631+
Both hops are non-blocking. The receiver uses `try_send` into the buffer and
632+
the session uses `try_send` into its outbound channel, so telemetry can never
633+
stall control traffic. After the main process exits and before the exit is
634+
reported, the supervisor asks the session to stop the receiver (no new
635+
streams, keep-alive disabled, two seconds for in-flight requests, then
636+
stragglers cut) and flush the buffer onto the stream. The whole drain is
637+
bounded at three seconds so an unreachable gateway cannot delay the report.
638+
639+
The gateway forwards trace bytes through a dedicated `OtelRelayExporter`
640+
rather than its own tracer provider, so the supervisor's resource attributes
641+
survive. `OtelExportData` also carries OCSF events, which the gateway
642+
re-emits on the `ocsf_relay` target; the supervisor-side OCSF sink is not
643+
installed yet.
644+
585645
## Policy Proposals
586646

587647
When an L4 CONNECT is denied, the proxy emits a `DenialEvent`. The denial

‎docs/kubernetes/sandbox-runtime.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,17 @@ OpenShell labels, or read bootstrap Secrets. Set
8585
`supervisor.sandboxRuntime.networkPolicyEnforced: true` only after you verify that the
8686
cluster CNI enforces both ingress and egress policies for these namespaces.
8787

88+
## Relay Agent Telemetry
89+
90+
When the gateway has `[openshell.gateway.otlp]` configured, agents in the
91+
workload Pod export OpenTelemetry traces to `http://192.0.0.8:4318`. That
92+
address is never routed. The sandbox seccomp broker stages the connection for
93+
the supervisor over the existing boundary channel, and the supervisor serves
94+
the OTLP receiver on that stream, adds sandbox attributes to the spans, and
95+
forwards them to the collector through the gateway. The workload egress fence
96+
stays empty and no NetworkPolicy change is needed. See
97+
[OTLP Export](/reference/gateway-config#otlp-export).
98+
8899
## Bootstrap a Sandbox
89100

90101
The driver creates each sandbox generation in a fail-closed order:

‎docs/reference/gateway-config.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,12 @@ Only OpenTelemetry traces are exported. Inbound gRPC and HTTP requests produce s
297297

298298
The gateway forwards the OTLP configuration, configured gateway name, configured compute driver, and W3C trace context to managed external drivers. Built-in drivers also export their spans to the same collector through dedicated in-process providers. Driver spans retain the gateway trace context, use a distinct service name such as `openshell-driver-docker` or `openshell-driver-podman`, and carry the same `openshell.gateway.name` and `openshell.gateway.compute_driver` resource attributes as gateway spans. Compute-driver client and server spans use the same fully qualified protobuf operation name, such as `openshell.compute.v1.ComputeDriver/CreateSandbox`, in both the span name and `rpc.method`. The service name and span kind distinguish each side. Backend-prefixed child spans identify implementation work. A streaming watch records a terminal status when observed; consumer teardown without a terminal status leaves the span status unset. Operator-run external drivers own their own telemetry configuration.
299299

300+
### Agent Telemetry Relay
301+
302+
Configuring this table also enables the sandbox telemetry relay. `CreateSandbox` sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://192.0.0.8:4318` and `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` in each new sandbox's environment unless the caller already supplied an endpoint. The values persist in the sandbox spec and survive restarts. OTel SDKs inside the sandbox export to that address over HTTP/protobuf or HTTP/JSON.
303+
304+
The address is never routed. The sandbox stages the connection for the supervisor, which serves the OTLP receiver itself, adds `openshell.sandbox.id`, `openshell.workspace.id`, `openshell.sandbox.policy`, `openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`, and `openshell.telemetry.source=agent`, and forwards the spans to the gateway. The gateway exports them to the same `endpoint` through a dedicated exporter, so agent spans arrive with the supervisor's resource attributes rather than the gateway's. Agent traces need no sandbox egress and no network policy entry. Telemetry never blocks sandbox control traffic: the relay buffers up to 4096 items per sandbox and drops the newest on overflow.
305+
300306
For Helm deployments, set `server.otlp.endpoint` to render this table. The
301307
optional `server.otlp.serviceName` value overrides the gateway service name;
302308
driver service names remain fixed.

0 commit comments

Comments
 (0)