Skip to content

Commit 8ed8bf0

Browse files
authored
docs(observability): show where each OTLP signal goes, including relayed agent traces (#5153)
The observability overview had no picture of which component exports which signal to which collector, agent traces appeared only in the gateway configuration reference, and the Traces page did not say that the agent's own spans travel separately. Add an OTLP Collectors section to the overview with a diagram, drawn in the style of the architecture images, of every lane: gateway and driver spans, supervisor spans and OCSF events, and agent traces relayed through the supervisor. Summarize the per-collector protocol, header, and TLS settings, add an Agent Traces section with an MLflow example, and link to the overview from the Traces page and the configuration reference. Signed-off-by: Kris Hicks <[email protected]>
1 parent 4b8aab5 commit 8ed8bf0

4 files changed

Lines changed: 132 additions & 3 deletions

File tree

‎docs/how-it-works/gateways/configuration.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -600,7 +600,7 @@ Allow enough retention and disk space for shipper outages. Graceful shutdown all
600600

601601
## OTLP Export
602602

603-
`[openshell.gateway.otlp]` enables OpenTelemetry export over OTLP. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and the [agent trace relay](#agent-trace-relay) is off until `[openshell.gateway.otlp.agent]` sets `enabled = true`. Each can use its own collector.
603+
`[openshell.gateway.otlp]` enables OpenTelemetry export over OTLP. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and the [agent trace relay](#agent-trace-relay) is off until `[openshell.gateway.otlp.agent]` sets `enabled = true`. Each can use its own collector. [OTLP Collectors](/observability/overview#otlp-collectors) shows where each signal goes.
604604

605605
The gateway already uses the Rust `tracing` framework for structured logs sent to stdout and the sandbox log stream. Enabling this section adds an OpenTelemetry layer to the same tracing subscriber. It exports span trees to an OTLP collector without exporting, replacing, or redirecting the existing diagnostic log events.
606606

Lines changed: 98 additions & 0 deletions
Loading

‎docs/observability/overview.mdx‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
title: "Observability Overview"
55
sidebar-title: "Overview"
66
description: "The traces, metrics, and OCSF events that OpenShell produces, where each one goes, and how to enable it."
7-
keywords: "Generative AI, Cybersecurity, Observability, OpenTelemetry, OTLP, Traces, Metrics, Prometheus, OCSF, SIEM"
7+
keywords: "Generative AI, Cybersecurity, Observability, OpenTelemetry, OTLP, Traces, Agent Traces, MLflow, Metrics, Prometheus, OCSF, SIEM"
88
position: 1
99
---
1010

@@ -13,14 +13,29 @@ OpenShell produces three kinds of observability signals: traces, metrics, and OC
1313
| Signal | Produced by | Destination |
1414
|---|---|---|
1515
| Traces | Gateway, compute drivers, sandbox supervisors | OTLP collector |
16+
| Agent traces | Agents, relayed by sandbox supervisors | OTLP collector |
1617
| Metrics | Gateway | Prometheus scrape |
1718
| OCSF events, shorthand | Gateway, sandbox supervisors | Gateway console, `openshell logs`, supervisor log file |
1819
| OCSF events, JSONL | Gateway, sandbox supervisors | Local file |
1920
| OCSF events, OTLP | Gateway, sandbox supervisors | OTLP collector |
2021

2122
The examples on this page are `gateway.toml` tables. In the Helm chart, set each table under `gatewayConfig`, as described in [Configure Chart Values](/kubernetes/setup#configure-chart-values).
2223

23-
OTLP export uses gRPC by default, or HTTP/protobuf for collectors that set it. Configure it in `gateway.toml`, and the gateway passes the settings to the sandbox supervisors it starts. Supervisors connect to the collector directly, so the collector must be reachable from sandboxes as well as from the gateway. The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. Refer to [Tuning](/how-it-works/gateways/configuration#tuning) for the full list.
24+
## OTLP Collectors
25+
26+
Configure OTLP export in `gateway.toml`. The gateway exports its own signals and passes each collector's settings to the compute drivers and sandbox supervisors it starts. Supervisors export their own signals and relay the agent's traces directly, so each collector must be reachable from sandboxes as well as from the gateway.
27+
28+
![OpenShell OTLP collectors. The gateway and its compute drivers export spans to the trace collector, and the gateway exports its OCSF events to the OCSF collector. The gateway gives each sandbox supervisor its collector settings with the launch credentials. The supervisor exports its own spans and OCSF events to the same collectors. The network-isolated agent sends OTLP/HTTP to the supervisor's agent trace relay, which tags the spans with the sandbox ID and forwards them to the agent trace collector.](../images/openshell-otlp-collectors.svg)
29+
30+
`[openshell.gateway.otlp]` sets one collector for every signal. `[openshell.gateway.otlp.traces]`, `[openshell.gateway.otlp.ocsf]`, and `[openshell.gateway.otlp.agent]` can each send their signal to a different collector instead, such as a SIEM for OCSF events or an MLflow tracking server for agent traces. Each collector has its own transport settings:
31+
32+
- OTLP over gRPC, the default, or HTTP/protobuf.
33+
- Request headers, such as an authorization token, set inline or read from a file.
34+
- TLS, verified against the system trust roots or a CA file, with an optional client certificate.
35+
36+
Supervisors receive collector headers and TLS material with their launch credentials, never through the sandbox environment. The agent reaches the relay through the sandbox's mediated channel, so it needs no egress rule and never sees the agent trace collector's address or credentials.
37+
38+
The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. For every setting, refer to [OTLP Export](/how-it-works/gateways/configuration#otlp-export) and [Tuning](/how-it-works/gateways/configuration#tuning).
2439

2540
## Traces
2641

@@ -35,6 +50,20 @@ endpoint = "https://backend:4317"
3550

3651
For span names, resource attributes, and gateway names, refer to [Traces](/observability/traces).
3752

53+
## Agent Traces
54+
55+
Agent traces are the spans that the agent itself produces, such as its model calls and tool use when it is instrumented with OpenTelemetry. Each sandbox supervisor relays them to the collector you choose. The relay is off by default. To send agent traces to MLflow, which needs OTLP/HTTP and an experiment header:
56+
57+
```toml
58+
[openshell.gateway.otlp.agent]
59+
enabled = true
60+
endpoint = "http://mlflow.mlops.svc:5000"
61+
protocol = "http/protobuf"
62+
headers = { x-mlflow-experiment-id = "0" }
63+
```
64+
65+
While the relay is on, every workload process gets `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at it, so an agent that uses a standard OpenTelemetry SDK exports without further configuration. Each relayed trace carries `openshell.telemetry.source=agent` and the sandbox's `openshell.sandbox.id`, so you can separate agent traces from OpenShell's own and join them with the sandbox's supervisor spans. For the workload environment, request limits, and delivery guarantees, refer to [Agent Trace Relay](/how-it-works/gateways/configuration#agent-trace-relay).
66+
3867
## Metrics
3968

4069
The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity, on a listener separate from its API. Use them to alert on saturation and to choose autoscaling signals.

‎docs/observability/traces.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ In the Helm chart, set the table under `gatewayConfig`. The gateway passes the e
3030
| Compute drivers | Child spans for each compute-driver operation the gateway performs, from both the gateway's side and the driver's side. |
3131
| Sandbox supervisors | Supervisor startup, such as loading policy. |
3232

33+
These are OpenShell's own spans. Spans that the agent produces travel separately, through the [agent trace relay](/observability/overview#agent-traces).
34+
3335
Spans for routine or high-volume operations use DEBUG level, and the default INFO level does not export them. These include store operations, health checks, the RPCs that supervisors call on a timer (`GetSandboxConfig` and `ReportProviderReadiness`), and how supervisors handle egress connections. Set the gateway or sandbox log level to `debug` to include them.
3436

3537
## Span Names

0 commit comments

Comments
 (0)