Skip to content

Commit 98c3194

Browse files
committed
docs(observability): add observability overview and traces pages
Add an Observability overview that lists the traces, metrics, and OCSF events OpenShell produces, where each goes, and the minimal configuration to enable it, linking to the detailed pages instead of repeating them. Add a Traces page for what is traced, span names, resource attributes, gateway names, and trace context, and point the OTLP Export reference to it instead of describing spans there. Move metric naming conventions into Gateway Metrics, and replace the stale denied-connection example in OCSF JSON Export with a real record. List every common OTEL_* environment variable in the gateway OTLP tuning reference, split into the variables the gateway honors and the ones it ignores, with the reasons. Closes #4548 Signed-off-by: Kris Hicks <[email protected]>
1 parent 6dbe4ad commit 98c3194

6 files changed

Lines changed: 272 additions & 29 deletions

File tree

‎docs/how-it-works/gateways/configuration.mdx‎

Lines changed: 25 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -417,15 +417,9 @@ Use an `https://` endpoint for TLS. The gateway verifies the collector certifica
417417

418418
The OpenTelemetry SDK logs export failures after startup. Spans in a failed batch are dropped rather than retried.
419419

420-
`service_name` sets the gateway's `service.name` resource attribute and defaults to `openshell-gateway`. The gateway also reports `service.version`, `openshell.gateway.name` from the gateway's configured `name`, and `openshell.gateway.compute_driver`.
420+
`service_name` sets the gateway's `service.name` resource attribute and defaults to `openshell-gateway`.
421421

422-
Inbound gRPC and HTTP requests produce server spans named for the RPC or HTTP method. Compute-driver operations appear as child spans. Internal reconciliation and driver-watch loops create operation roots because no inbound request supplies a parent, and credential refresh creates one only when a credential is due or needs cleanup. The gateway continues valid W3C `traceparent` context and starts a new trace when none is supplied. Request spans carry `method`, `path`, and the `request_id` that also appears in gateway logs. Store operation spans, health endpoint spans, and spans for RPCs that sandbox supervisors call on a timer (`GetSandboxConfig` and `ReportProviderReadiness`, including its peer-forwarded form) use DEBUG level. The default INFO filter does not export them. Set the gateway log level to `debug` to include them.
423-
424-
The gateway forwards the trace endpoint, configured gateway name, configured compute driver, and W3C trace context to managed external drivers. Built-in drivers also export their spans to the trace collector through dedicated in-process providers. Driver spans retain the gateway trace context, use a distinct service name such as `openshell-driver-docker` or `openshell-driver-podman`, and carry the same `openshell.gateway.name` and `openshell.gateway.compute_driver` resource attributes as gateway spans. Compute-driver client and server spans use the same fully qualified protobuf operation name, such as `openshell.compute.v1.ComputeDriver/CreateSandbox`, in both the span name and `rpc.method`. The service name and span kind distinguish each side. Backend-prefixed child spans identify implementation work. A streaming watch records a terminal status when observed; consumer teardown without a terminal status leaves the span status unset. Operator-run external drivers own their own telemetry configuration.
425-
426-
Sandbox supervisors export spans to the trace collector, so its endpoint must also be reachable from sandboxes.
427-
428-
Spans for routine or high-volume operations use DEBUG level, and the default INFO level does not export them. Set the gateway or sandbox log level to `debug` to include them.
422+
The gateway passes the trace endpoint to compute drivers and sandbox supervisors, which export their own spans to it, so the trace collector must also be reachable from sandboxes. Spans for routine or high-volume operations use DEBUG level and are exported only when the gateway or sandbox log level is `debug`. For what is traced, span names, and resource attributes, refer to [Traces](/observability/traces).
429423

430424
For Helm deployments, set `server.otlp.endpoint` to render this table. The
431425
optional `server.otlp.serviceName` value overrides the gateway service name;
@@ -490,28 +484,45 @@ Metrics include `openshell_ocsf_otlp_queued_total`, `openshell_ocsf_otlp_exporte
490484

491485
### Tuning
492486

493-
This table decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys:
487+
`[openshell.gateway.otlp]` decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys. They apply to the gateway and its in-process compute drivers. The gateway does not pass them to sandbox supervisors, which receive only the endpoints, the OCSF schema version, and the [OCSF collector headers](#ocsf-log-export).
488+
489+
The gateway honors these variables:
494490

495491
| Variable | Effect |
496492
|---|---|
493+
| `OTEL_SERVICE_NAME` | The gateway's `service.name` when `service_name` is not set in `[openshell.gateway.otlp]`. Compute-driver and supervisor service names are fixed. |
494+
| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes, such as `deployment.environment=prod`. |
497495
| `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | Sampling strategy and ratio. Defaults to `parentbased_always_on`. |
498496
| `OTEL_BSP_SCHEDULE_DELAY`, `OTEL_BSP_MAX_QUEUE_SIZE`, `OTEL_BSP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BSP_EXPORT_TIMEOUT` | Batch span processor tuning. |
499-
| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes, such as `deployment.environment=prod`. |
500497
| `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT`, `OTEL_SPAN_EVENT_COUNT_LIMIT`, `OTEL_SPAN_LINK_COUNT_LIMIT` | Per-span limits. |
501498
| `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT` | Exporter transport tuning for every signal. |
502499
| `OTEL_EXPORTER_OTLP_COMPRESSION` | Set to `gzip` to compress exports of every signal. `zstd` is not supported. |
500+
| `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION`, `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` | Exporter transport tuning for spans only. These override the general variables. |
503501
| `OTEL_EXPORTER_OTLP_LOGS_HEADERS`, `OTEL_EXPORTER_OTLP_LOGS_COMPRESSION`, `OTEL_EXPORTER_OTLP_LOGS_TIMEOUT` | Exporter transport tuning for OCSF log records only. These override the general variables. |
504-
| `OTEL_EXPORTER_OTLP_PROTOCOL` | No effect. The gateway is built with the gRPC exporter only. |
505-
| `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | No effect. OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). |
502+
| `SSL_CERT_FILE`, `SSL_CERT_DIR` | Additional trust roots for `https://` endpoints, such as a private CA. |
503+
504+
These variables have no effect. Only `[openshell.gateway.otlp]` decides which signals are exported and where they go, for two reasons:
505+
506+
- OCSF events are security records, so where they go must be a deliberate choice. Platform tooling can add `OTEL_*` variables to a process, as the OpenTelemetry Operator does for annotated pods on Kubernetes. Such a variable must not start sending security records to a collector, redirect them, or turn them off.
507+
- The gateway passes its export settings to compute drivers and sandbox supervisors. Supervisors run in each sandbox and do not inherit the gateway's environment, so they could not follow a setting that exists only there.
508+
509+
510+
| Variable | Reason |
511+
|---|---|
512+
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Enablement has one source, `[openshell.gateway.otlp]`, so an environment variable cannot silently turn export on or redirect it. |
513+
| `OTEL_SDK_DISABLED`, `OTEL_TRACES_EXPORTER`, `OTEL_LOGS_EXPORTER`, `OTEL_METRICS_EXPORTER` | `[openshell.gateway.otlp]` decides which signals are exported. To disable export, omit it. |
514+
| `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | The gateway is built with the gRPC exporter only. |
515+
| `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_EXPORTER_OTLP_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_KEY`, and their per-signal forms | The endpoint scheme selects TLS, and collector certificates are verified against the system trust roots. Client certificates are not supported. |
516+
| `OTEL_PROPAGATORS` | The gateway always propagates W3C Trace Context. |
517+
| `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). |
518+
| `OTEL_EXPORTER_OTLP_METRICS_*`, `OTEL_METRIC_EXPORT_INTERVAL` | The gateway serves metrics for Prometheus scraping and does not export them over OTLP. |
506519

507520
To sample 10% of traces:
508521

509522
```shell
510523
OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1
511524
```
512525

513-
`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` are deliberately ignored. Enablement has one source, so an environment variable cannot silently turn export on or redirect it. `OTEL_RESOURCE_ATTRIBUTES` does apply and adds attributes, but a `service_name` set here wins over `OTEL_SERVICE_NAME`.
514-
515526
The gateway flushes buffered spans during shutdown, so spans from in-flight requests survive a `SIGTERM`.
516527

517528
## Supervisor Middleware Services

‎docs/observability/gateway-metrics.mdx‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ title: "Gateway Metrics"
55
sidebar-title: "Gateway Metrics"
66
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments."
77
keywords: "Generative AI, Cybersecurity, Observability, Metrics, Prometheus, Gateway, Kubernetes, High Availability"
8+
position: 3
89
---
910

1011
The gateway exports Prometheus metrics for API requests, database readiness,
@@ -136,6 +137,22 @@ Then read the metrics from another terminal:
136137
curl -s http://localhost:9090/metrics | grep '^openshell_server_supervisor_sessions'
137138
```
138139

140+
## Metric Names
141+
142+
Every metric name starts with `openshell_`, followed by the subsystem that
143+
records it:
144+
145+
| Prefix | Covers |
146+
|---|---|
147+
| `openshell_server_` | API requests, readiness, supervisor sessions, relays, and requests forwarded between replicas |
148+
| `openshell_gateway_interceptor_` | Gateway interceptor evaluations, failures, and latency |
149+
| `openshell_ocsf_log_` | The gateway's OCSF JSONL file |
150+
| `openshell_ocsf_otlp_` | OCSF log export over OTLP |
151+
152+
Names follow Prometheus conventions. Counters end in `_total`, and names with a
153+
unit end in that unit, such as `_seconds` or `_bytes`. Metrics carry no gateway
154+
name label, so use your scrape configuration to tell installations apart.
155+
139156
## Histograms and Summaries
140157

141158
The multi-replica latency metrics are Prometheus histograms with `_bucket`,
@@ -291,6 +308,7 @@ enabled record its queue and exporter:
291308

292309
## Next Steps
293310

311+
- See the [Observability Overview](/observability/overview) for traces and OCSF events.
294312
- To monitor capacity and choose autoscaling signals for multiple gateway
295313
replicas, refer to
296314
[High Availability](/kubernetes/high-availability#monitor-capacity).

‎docs/observability/logging.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,7 @@ Docker and Podman use tmpfs for this directory; Kubernetes uses an `emptyDir` in
274274

275275
## Next Steps
276276

277+
- See the [Observability Overview](/observability/overview) for every signal OpenShell produces and how to enable it.
277278
- [Access logs](/observability/accessing-logs) through the CLI, TUI, or supervisor resources.
278279
- [Enable OCSF JSON export](/observability/ocsf-json-export) for SIEM integration and compliance.
279280
- Learn about [network policies](/how-it-works/policies/overview) that generate these events.

‎docs/observability/ocsf-json-export.mdx‎

Lines changed: 51 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -129,8 +129,8 @@ Each JSONL file line, or the payload after the stderr marker, is a complete OCSF
129129
"metadata": {
130130
"product": {
131131
"name": "OpenShell Sandbox Supervisor",
132-
"vendor_name": "NVIDIA",
133-
"version": "0.3.0"
132+
"vendor_name": "OpenShell",
133+
"version": "0.1.4"
134134
},
135135
"version": "1.8.0"
136136
},
@@ -159,38 +159,73 @@ Each JSONL file line, or the payload after the stderr marker, is a complete OCSF
159159
}
160160
```
161161

162-
And a denied connection:
162+
And a denied connection, produced when `curl https://github.com` runs in a sandbox without a network policy that allows `github.com`:
163163

164164
```json
165165
{
166166
"class_uid": 4001,
167167
"class_name": "Network Activity",
168+
"category_uid": 4,
169+
"category_name": "Network Activity",
168170
"activity_id": 1,
169171
"activity_name": "Open",
172+
"type_uid": 400101,
173+
"type_name": "Network Activity: Open",
174+
"time": 1791996125789,
170175
"severity_id": 3,
171176
"severity": "Medium",
172177
"status_id": 2,
173178
"status": "Failure",
174-
"action_id": 2,
175-
"action": "Denied",
176-
"disposition_id": 2,
177-
"disposition": "Blocked",
178-
"status_detail": "no matching policy",
179-
"message": "CONNECT denied httpbin.org:443",
179+
"message": "Transparent TCP denied before relay: network connections not allowed by policy",
180+
"status_detail": "transparent_tcp_policy_denied",
181+
"metadata": {
182+
"version": "1.8.0",
183+
"product": {
184+
"name": "OpenShell Sandbox Supervisor",
185+
"vendor_name": "OpenShell",
186+
"version": "0.1.4"
187+
},
188+
"profiles": [
189+
"security_control",
190+
"network_proxy",
191+
"container",
192+
"host"
193+
],
194+
"uid": "b885957d-124a-4af7-9eef-75cc410ff070"
195+
},
196+
"device": {
197+
"hostname": "docker-desktop",
198+
"type_id": 99,
199+
"type": "Sandbox",
200+
"os": {
201+
"name": "Linux"
202+
}
203+
},
204+
"container": {
205+
"name": "docs-deny",
206+
"uid": "3949f6fc-2b84-45b2-a631-46debe41d84f"
207+
},
180208
"dst_endpoint": {
181-
"domain": "httpbin.org",
209+
"domain": "github.com",
210+
"ip": "198.18.0.2",
182211
"port": 443
183212
},
213+
"proxy_endpoint": {
214+
"ip": "127.0.0.1",
215+
"port": 3128
216+
},
184217
"actor": {
185218
"process": {
186219
"name": "/usr/bin/curl",
187-
"pid": 63
220+
"pid": 2187,
221+
"cmd_line": "curl https://github.com"
188222
}
189223
},
190-
"firewall_rule": {
191-
"name": "-",
192-
"type": "opa"
193-
}
224+
"action_id": 2,
225+
"action": "Denied",
226+
"disposition_id": 2,
227+
"disposition": "Blocked",
228+
"is_src_dst_assignment_known": true
194229
}
195230
```
196231

@@ -315,5 +350,6 @@ Shorthand stderr output follows the diagnostic log filter; its local file is ava
315350

316351
## Next Steps
317352

353+
- See the [Observability Overview](/observability/overview) to compare JSONL files with OTLP log export.
318354
- Learn how to [read the shorthand format](/observability/logging) for real-time monitoring.
319355
- Refer to the [OCSF specification](https://schema.ocsf.io/) for the full schema reference.

‎docs/observability/overview.mdx‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
title: "Observability Overview"
5+
sidebar-title: "Overview"
6+
description: "The traces, metrics, and OCSF events that OpenShell produces, where each one goes, and how to enable it."
7+
keywords: "Generative AI, Cybersecurity, Observability, OpenTelemetry, OTLP, Traces, Metrics, Prometheus, OCSF, SIEM"
8+
position: 1
9+
---
10+
11+
OpenShell produces three kinds of observability signals: traces, metrics, and OCSF security events. This page describes what each signal is for and how to turn it on, and links to the page that covers it in detail.
12+
13+
| Signal | Produced by | Destination |
14+
|---|---|---|
15+
| Traces | Gateway, compute drivers, sandbox supervisors | OTLP collector |
16+
| Metrics | Gateway | Prometheus scrape |
17+
| OCSF events, shorthand | Gateway, sandbox supervisors | Gateway console, `openshell logs`, supervisor log file |
18+
| OCSF events, JSONL | Gateway, sandbox supervisors | Local file |
19+
| OCSF events, OTLP | Gateway, sandbox supervisors | OTLP collector |
20+
21+
The examples on this page are `gateway.toml` tables. In the Helm chart, set each table under `gatewayConfig`, as described in [Configure Chart Values](/kubernetes/setup#configure-chart-values).
22+
23+
All OTLP export uses OTLP over gRPC. Configure it in `gateway.toml`, and the gateway passes the settings to the sandbox supervisors it starts. Supervisors connect to the collector directly, so the collector must be reachable from sandboxes as well as from the gateway. The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. Refer to [Tuning](/how-it-works/gateways/configuration#tuning) for the full list.
24+
25+
## Traces
26+
27+
Traces show how the gateway handles each request, including the compute-driver work it performs, and how each sandbox supervisor starts up. Use them to find where time goes in an operation such as sandbox creation.
28+
29+
To export traces, set an OTLP collector endpoint in `gateway.toml`:
30+
31+
```toml
32+
[openshell.gateway.otlp]
33+
endpoint = "https://backend:4317"
34+
```
35+
36+
For span names, resource attributes, and gateway names, refer to [Traces](/observability/traces).
37+
38+
## Metrics
39+
40+
The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity, on a listener separate from its API. Use them to alert on saturation and to choose autoscaling signals.
41+
42+
To enable the listener, set `metrics_bind_address` in `gateway.toml`:
43+
44+
```toml
45+
[openshell.gateway]
46+
metrics_bind_address = "0.0.0.0:9090"
47+
```
48+
49+
The port is an example. Until the listener is enabled, the gateway serves no metrics. The endpoint has no authentication, so restrict who can reach it. For metric names, deployment defaults, the catalog, and scrape configuration, refer to [Gateway Metrics](/observability/gateway-metrics).
50+
51+
## OCSF Events
52+
53+
OCSF events record security-relevant activity, such as network and HTTP policy decisions, process lifecycle, SSH sessions, and configuration changes. They use the [Open Cybersecurity Schema Framework](https://ocsf.io). Sandbox supervisors produce most of them, and the gateway produces events for its own decisions, such as approving or rejecting draft policy rules. The same events are available in three forms.
54+
55+
### Shorthand Logs
56+
57+
Shorthand logs are always on. Each event appears as a one-line summary in the gateway console, in the sandbox log stream that `openshell logs` and the TUI read, and in the supervisor's log file. For example, this policy denial is the event a sandbox produces when `curl https://github.com` runs without a network policy that allows `github.com`:
58+
59+
```text
60+
[1791996125.789] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(2187) -> github.com:443 [reason:transparent_tcp_policy_denied]
61+
```
62+
63+
To read the format, refer to [Sandbox Logging](/observability/logging). To find logs for each compute driver, refer to [Accessing Logs](/observability/accessing-logs).
64+
65+
### JSONL Files
66+
67+
JSONL files hold the full OCSF record, one JSON object per line, for a log shipper to collect. The gateway and supervisors are enabled separately:
68+
69+
```toml
70+
# Gateway events, in gateway.toml.
71+
[openshell.gateway.ocsf_log]
72+
path = "/var/log/openshell/gateway-ocsf.jsonl"
73+
```
74+
75+
```shell
76+
# Supervisor events, for every sandbox.
77+
openshell settings set --global --key ocsf_json_enabled --value true
78+
```
79+
80+
For the record structure, file locations, rotation, and schema version downgrades for older SIEMs, refer to [OCSF JSON Export](/observability/ocsf-json-export).
81+
82+
### OTLP Log Export
83+
84+
OTLP log export sends each OCSF event to a collector as an OTLP log record that contains the full JSON event. It is off by default, even when trace export is on. Set `enabled = true` in `[openshell.gateway.otlp.ocsf]` to turn it on for the gateway and for every sandbox supervisor the gateway starts:
85+
86+
```toml
87+
[openshell.gateway.otlp]
88+
endpoint = "https://backend:4317"
89+
90+
[openshell.gateway.otlp.ocsf]
91+
enabled = true
92+
```
93+
94+
The records go to the `[openshell.gateway.otlp]` endpoint unless the table sets its own `endpoint`, such as a collector that feeds a SIEM. For the record format, collector authentication, delivery guarantees, and loss reporting, refer to [OCSF Log Export](/how-it-works/gateways/configuration#ocsf-log-export).
95+
96+
## Next Steps
97+
98+
- Learn how spans are named and identified in [Traces](/observability/traces).
99+
- Learn how to [read the shorthand format](/observability/logging).
100+
- Learn what anonymous usage data OpenShell collects in [Telemetry](/observability/telemetry). It is separate from the signals on this page.

0 commit comments

Comments
 (0)