You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8ed8bf0
Browse filesBrowse the repository at this point in the historyBrowse files
docs(observability): show where each OTLP signal goes, including relayed agent traces (#5153)
The observability overview had no picture of which component exports
which signal to which collector, agent traces appeared only in the
gateway configuration reference, and the Traces page did not say that
the agent's own spans travel separately.
Add an OTLP Collectors section to the overview with a diagram, drawn in
the style of the architecture images, of every lane: gateway and driver
spans, supervisor spans and OCSF events, and agent traces relayed
through the supervisor. Summarize the per-collector protocol, header,
and TLS settings, add an Agent Traces section with an MLflow example,
and link to the overview from the Traces page and the configuration
reference.
Signed-off-by: Kris Hicks <[email protected]>
Copy file name to clipboardExpand all lines: docs/how-it-works/gateways/configuration.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -600,7 +600,7 @@ Allow enough retention and disk space for shipper outages. Graceful shutdown all
600
600
601
601
## OTLP Export
602
602
603
-
`[openshell.gateway.otlp]`enables OpenTelemetry export over OTLP. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and the [agent trace relay](#agent-trace-relay) is off until `[openshell.gateway.otlp.agent]` sets `enabled = true`. Each can use its own collector.
603
+
`[openshell.gateway.otlp]`enables OpenTelemetry export over OTLP. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and the [agent trace relay](#agent-trace-relay) is off until `[openshell.gateway.otlp.agent]` sets `enabled = true`. Each can use its own collector. [OTLP Collectors](/observability/overview#otlp-collectors) shows where each signal goes.
604
604
605
605
The gateway already uses the Rust `tracing` framework for structured logs sent to stdout and the sandbox log stream. Enabling this section adds an OpenTelemetry layer to the same tracing subscriber. It exports span trees to an OTLP collector without exporting, replacing, or redirecting the existing diagnostic log events.
The examples on this page are `gateway.toml` tables. In the Helm chart, set each table under `gatewayConfig`, as described in [Configure Chart Values](/kubernetes/setup#configure-chart-values).
22
23
23
-
OTLP export uses gRPC by default, or HTTP/protobuf for collectors that set it. Configure it in `gateway.toml`, and the gateway passes the settings to the sandbox supervisors it starts. Supervisors connect to the collector directly, so the collector must be reachable from sandboxes as well as from the gateway. The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. Refer to [Tuning](/how-it-works/gateways/configuration#tuning) for the full list.
24
+
## OTLP Collectors
25
+
26
+
Configure OTLP export in `gateway.toml`. The gateway exports its own signals and passes each collector's settings to the compute drivers and sandbox supervisors it starts. Supervisors export their own signals and relay the agent's traces directly, so each collector must be reachable from sandboxes as well as from the gateway.
27
+
28
+

29
+
30
+
`[openshell.gateway.otlp]` sets one collector for every signal. `[openshell.gateway.otlp.traces]`, `[openshell.gateway.otlp.ocsf]`, and `[openshell.gateway.otlp.agent]` can each send their signal to a different collector instead, such as a SIEM for OCSF events or an MLflow tracking server for agent traces. Each collector has its own transport settings:
31
+
32
+
- OTLP over gRPC, the default, or HTTP/protobuf.
33
+
- Request headers, such as an authorization token, set inline or read from a file.
34
+
- TLS, verified against the system trust roots or a CA file, with an optional client certificate.
35
+
36
+
Supervisors receive collector headers and TLS material with their launch credentials, never through the sandbox environment. The agent reaches the relay through the sandbox's mediated channel, so it needs no egress rule and never sees the agent trace collector's address or credentials.
37
+
38
+
The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. For every setting, refer to [OTLP Export](/how-it-works/gateways/configuration#otlp-export) and [Tuning](/how-it-works/gateways/configuration#tuning).
For span names, resource attributes, and gateway names, refer to [Traces](/observability/traces).
37
52
53
+
## Agent Traces
54
+
55
+
Agent traces are the spans that the agent itself produces, such as its model calls and tool use when it is instrumented with OpenTelemetry. Each sandbox supervisor relays them to the collector you choose. The relay is off by default. To send agent traces to MLflow, which needs OTLP/HTTP and an experiment header:
While the relay is on, every workload process gets `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at it, so an agent that uses a standard OpenTelemetry SDK exports without further configuration. Each relayed trace carries `openshell.telemetry.source=agent` and the sandbox's `openshell.sandbox.id`, so you can separate agent traces from OpenShell's own and join them with the sandbox's supervisor spans. For the workload environment, request limits, and delivery guarantees, refer to [Agent Trace Relay](/how-it-works/gateways/configuration#agent-trace-relay).
66
+
38
67
## Metrics
39
68
40
69
The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity, on a listener separate from its API. Use them to alert on saturation and to choose autoscaling signals.
Copy file name to clipboardExpand all lines: docs/observability/traces.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,8 @@ In the Helm chart, set the table under `gatewayConfig`. The gateway passes the e
30
30
| Compute drivers | Child spans for each compute-driver operation the gateway performs, from both the gateway's side and the driver's side. |
31
31
| Sandbox supervisors | Supervisor startup, such as loading policy. |
32
32
33
+
These are OpenShell's own spans. Spans that the agent produces travel separately, through the [agent trace relay](/observability/overview#agent-traces).
34
+
33
35
Spans for routine or high-volume operations use DEBUG level, and the default INFO level does not export them. These include store operations, health checks, the RPCs that supervisors call on a timer (`GetSandboxConfig` and `ReportProviderReadiness`), and how supervisors handle egress connections. Set the gateway or sandbox log level to `debug` to include them.
0 commit comments