Skip to content

Commit 9705f65

Browse files
authored
docs(observability): add an observability overview page
Add an Observability overview that lists the traces, metrics, and OCSF events OpenShell produces, where each goes, and the minimal configuration to enable it, linking to the detailed pages instead of repeating them. Link the overview from the logging, OCSF JSON export, and gateway metrics pages. List every common OTEL_* environment variable in the gateway OTLP tuning reference, split into the variables the gateway honors and the ones it ignores, with the reason for each, and summarize that split in the overview. Closes #4548 Signed-off-by: Kris Hicks <[email protected]>
1 parent 6dbe4ad commit 9705f65

5 files changed

Lines changed: 263 additions & 6 deletions

File tree

‎docs/how-it-works/gateways/configuration.mdx‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -490,28 +490,41 @@ Metrics include `openshell_ocsf_otlp_queued_total`, `openshell_ocsf_otlp_exporte
490490

491491
### Tuning
492492

493-
This table decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys:
493+
This table decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys. They apply to the gateway and its in-process compute drivers. The gateway does not pass them to sandbox supervisors, which receive only the endpoints, the OCSF schema version, and the [OCSF collector headers](#ocsf-log-export).
494+
495+
The gateway honors these variables:
494496

495497
| Variable | Effect |
496498
|---|---|
499+
| `OTEL_SERVICE_NAME` | The gateway's `service.name` when `service_name` is not set in this table. Compute-driver and supervisor service names are fixed. |
500+
| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes, such as `deployment.environment=prod`. |
497501
| `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | Sampling strategy and ratio. Defaults to `parentbased_always_on`. |
498502
| `OTEL_BSP_SCHEDULE_DELAY`, `OTEL_BSP_MAX_QUEUE_SIZE`, `OTEL_BSP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BSP_EXPORT_TIMEOUT` | Batch span processor tuning. |
499-
| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes, such as `deployment.environment=prod`. |
500503
| `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT`, `OTEL_SPAN_EVENT_COUNT_LIMIT`, `OTEL_SPAN_LINK_COUNT_LIMIT` | Per-span limits. |
501504
| `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT` | Exporter transport tuning for every signal. |
502505
| `OTEL_EXPORTER_OTLP_COMPRESSION` | Set to `gzip` to compress exports of every signal. `zstd` is not supported. |
506+
| `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION`, `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` | Exporter transport tuning for spans only. These override the general variables. |
503507
| `OTEL_EXPORTER_OTLP_LOGS_HEADERS`, `OTEL_EXPORTER_OTLP_LOGS_COMPRESSION`, `OTEL_EXPORTER_OTLP_LOGS_TIMEOUT` | Exporter transport tuning for OCSF log records only. These override the general variables. |
504-
| `OTEL_EXPORTER_OTLP_PROTOCOL` | No effect. The gateway is built with the gRPC exporter only. |
505-
| `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | No effect. OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). |
508+
| `SSL_CERT_FILE`, `SSL_CERT_DIR` | Additional trust roots for `https://` endpoints, such as a private CA. |
509+
510+
These variables have no effect:
511+
512+
| Variable | Reason |
513+
|---|---|
514+
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Enablement has one source, this table, so an environment variable cannot silently turn export on or redirect it. |
515+
| `OTEL_SDK_DISABLED`, `OTEL_TRACES_EXPORTER`, `OTEL_LOGS_EXPORTER`, `OTEL_METRICS_EXPORTER` | This table decides which signals are exported. To disable export, omit it. |
516+
| `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | The gateway is built with the gRPC exporter only. |
517+
| `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_EXPORTER_OTLP_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_KEY`, and their per-signal forms | The endpoint scheme selects TLS, and collector certificates are verified against the system trust roots. Client certificates are not supported. |
518+
| `OTEL_PROPAGATORS` | The gateway always propagates W3C Trace Context. |
519+
| `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). |
520+
| `OTEL_EXPORTER_OTLP_METRICS_*`, `OTEL_METRIC_EXPORT_INTERVAL` | The gateway serves metrics for Prometheus scraping and does not export them over OTLP. |
506521

507522
To sample 10% of traces:
508523

509524
```shell
510525
OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1
511526
```
512527

513-
`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` are deliberately ignored. Enablement has one source, so an environment variable cannot silently turn export on or redirect it. `OTEL_RESOURCE_ATTRIBUTES` does apply and adds attributes, but a `service_name` set here wins over `OTEL_SERVICE_NAME`.
514-
515528
The gateway flushes buffered spans during shutdown, so spans from in-flight requests survive a `SIGTERM`.
516529

517530
## Supervisor Middleware Services

‎docs/observability/gateway-metrics.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,7 @@ enabled record its queue and exporter:
291291

292292
## Next Steps
293293

294+
- See the [Observability Overview](/observability/overview) for traces and OCSF events.
294295
- To monitor capacity and choose autoscaling signals for multiple gateway
295296
replicas, refer to
296297
[High Availability](/kubernetes/high-availability#monitor-capacity).

‎docs/observability/logging.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,7 @@ Docker and Podman use tmpfs for this directory; Kubernetes uses an `emptyDir` in
274274

275275
## Next Steps
276276

277+
- See the [Observability Overview](/observability/overview) for every signal OpenShell produces and how to enable it.
277278
- [Access logs](/observability/accessing-logs) through the CLI, TUI, or supervisor resources.
278279
- [Enable OCSF JSON export](/observability/ocsf-json-export) for SIEM integration and compliance.
279280
- Learn about [network policies](/how-it-works/policies/overview) that generate these events.

‎docs/observability/ocsf-json-export.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,5 +315,6 @@ Shorthand stderr output follows the diagnostic log filter; its local file is ava
315315

316316
## Next Steps
317317

318+
- See the [Observability Overview](/observability/overview) to compare JSONL files with OTLP log export.
318319
- Learn how to [read the shorthand format](/observability/logging) for real-time monitoring.
319320
- Refer to the [OCSF specification](https://schema.ocsf.io/) for the full schema reference.

‎docs/observability/overview.mdx‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
---
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
title: "Observability Overview"
5+
sidebar-title: "Overview"
6+
description: "The traces, metrics, and OCSF events that OpenShell produces, where each one goes, and how to enable it."
7+
keywords: "Generative AI, Cybersecurity, Observability, OpenTelemetry, OTLP, Traces, Metrics, Prometheus, OCSF, SIEM"
8+
position: 1
9+
---
10+
11+
OpenShell produces three kinds of observability signals: traces, metrics, and OCSF security events. This page describes what each signal covers and how to enable it, and links to the detailed page for each one.
12+
13+
| Signal | Produced by | Destination | Enabled by default |
14+
|---|---|---|---|
15+
| Traces | Gateway, compute drivers, sandbox supervisors | OTLP collector | No |
16+
| Metrics | Gateway | Prometheus scrape | Yes in the Helm chart. No for the standalone gateway. |
17+
| OCSF events, shorthand | Gateway, sandbox supervisors | Gateway console, `openshell logs`, supervisor log file | Yes |
18+
| OCSF events, JSONL | Gateway, sandbox supervisors | Local file | No |
19+
| OCSF events, OTLP | Gateway, sandbox supervisors | OTLP collector | No |
20+
21+
All OTLP export uses OTLP over gRPC. Configure it in the gateway, and the gateway passes the settings to the sandbox supervisors it starts. Supervisors connect to the collector directly, so the collector must be reachable from sandboxes as well as from the gateway.
22+
23+
## Traces
24+
25+
Traces show how the gateway handles each request, including the compute-driver work it performs, and how each sandbox supervisor starts up. At debug level, they also show how supervisors handle egress connections. Use them to find where time goes in an operation such as sandbox creation.
26+
27+
To export traces, set an OTLP collector endpoint in `gateway.toml`:
28+
29+
```toml
30+
[openshell.gateway.otlp]
31+
endpoint = "https://backend:4317"
32+
```
33+
34+
In the Helm chart, set `server.otlp.endpoint`. For log levels, TLS, sampling, and the `OTEL_*` tuning variables, refer to [OTLP Export](/how-it-works/gateways/configuration#otlp-export).
35+
36+
### Span Names
37+
38+
Each component names its spans after the operation it performs:
39+
40+
| Component | Span names | Examples |
41+
|---|---|---|
42+
| Gateway requests | The gRPC service and method, or the HTTP method for other requests | `openshell.v1.OpenShell/CreateSandbox`, `GET` |
43+
| Gateway background work | The task. These spans start their own traces. | `reconcile.sandboxes`, `refresh.provider_credentials` |
44+
| Compute drivers | The compute-driver RPC, with child spans prefixed by the driver name | `openshell.compute.v1.ComputeDriver/CreateSandbox`, `docker.provision` |
45+
| Sandbox supervisors | Prefixed with `supervisor.` | `supervisor.startup`, `supervisor.policy.load` |
46+
47+
The gateway continues a W3C `traceparent` sent by the caller, so a client's trace can include the gateway's spans for its requests.
48+
49+
### Resource Attributes
50+
51+
Resource attributes identify which process produced a span:
52+
53+
| Attribute | Value | Set on |
54+
|---|---|---|
55+
| `service.name` | `openshell-gateway`, `openshell-driver-<driver>`, or `openshell-supervisor` | Every span |
56+
| `service.version` | The OpenShell version | Every span |
57+
| `openshell.gateway.name` | The gateway name | Every span |
58+
| `openshell.gateway.compute_driver` | The configured compute driver, such as `kubernetes` | Gateway and compute-driver spans |
59+
| `openshell.sandbox.id` | The sandbox ID | Supervisor spans |
60+
61+
The gateway name comes from `name` under `[openshell.gateway]`, the `--name` flag, or `OPENSHELL_GATEWAY_NAME`, and defaults to `openshell`. The Helm chart sets it with `server.name`, which defaults to the chart's full name. Give each installation that reports to the same collector its own name, including each high-availability deployment, so that you can tell their spans apart. The replicas of one installation share its name.
62+
63+
To override the gateway's `service.name`, set `service_name` under `[openshell.gateway.otlp]`. To add your own attributes, such as `deployment.environment`, set `OTEL_RESOURCE_ATTRIBUTES` in the gateway's environment.
64+
65+
## Metrics
66+
67+
The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity. They are served on a separate listener.
68+
69+
To enable the listener, set `metrics_bind_address` in `gateway.toml`:
70+
71+
```toml
72+
[openshell.gateway]
73+
metrics_bind_address = "0.0.0.0:9090"
74+
```
75+
76+
The Helm chart serves metrics on port `9090` by default. The endpoint has no authentication, so restrict who can reach it. For the metric catalog and scrape configuration, refer to [Gateway Metrics](/observability/gateway-metrics).
77+
78+
### Metric Names
79+
80+
Every metric name starts with `openshell_`, followed by the subsystem that records it:
81+
82+
| Prefix | Covers |
83+
|---|---|
84+
| `openshell_server_` | API requests, readiness, supervisor sessions, relays, and requests forwarded between replicas |
85+
| `openshell_gateway_interceptor_` | Gateway interceptor evaluations, failures, and latency |
86+
| `openshell_ocsf_log_` | The gateway's OCSF JSONL file |
87+
| `openshell_ocsf_otlp_` | OCSF log export over OTLP |
88+
89+
Names follow Prometheus conventions. Counters end in `_total`, and names with a unit end in that unit, such as `_seconds` or `_bytes`. Metrics carry no gateway name or replica label. Your scrape configuration identifies where each series came from, so keep the `namespace` and `pod` labels, or their equivalents, when you scrape each replica.
90+
91+
## OCSF Events
92+
93+
OCSF events record security-relevant activity, such as network and HTTP policy decisions, process lifecycle, SSH sessions, and configuration changes. They use the [Open Cybersecurity Schema Framework](https://ocsf.io). Sandbox supervisors produce most of them, and the gateway produces events for its own decisions, such as approving or rejecting draft policy rules. The same events are available in three forms.
94+
95+
### Shorthand Logs
96+
97+
Shorthand logs are always on. Each event appears as a one-line summary in the gateway console, in the sandbox log stream that `openshell logs` and the TUI read, and in the supervisor's log file:
98+
99+
```shell
100+
openshell logs my-sandbox
101+
```
102+
103+
For example, this policy denial is the event a sandbox produces when `curl https://github.com` runs without a network policy that allows `github.com`:
104+
105+
```text
106+
[1791996125.789] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(2187) -> github.com:443 [reason:transparent_tcp_policy_denied]
107+
```
108+
109+
To read the format, refer to [Sandbox Logging](/observability/logging). To find logs for each compute driver, refer to [Accessing Logs](/observability/accessing-logs).
110+
111+
### JSONL Files
112+
113+
JSONL files hold the full OCSF record, one JSON object per line, for a log shipper to collect. The gateway and supervisors are enabled separately:
114+
115+
```toml
116+
# Gateway events, in gateway.toml.
117+
[openshell.gateway.ocsf_log]
118+
path = "/var/log/openshell/gateway-ocsf.jsonl"
119+
```
120+
121+
```shell
122+
# Supervisor events, for every sandbox.
123+
openshell settings set --global --key ocsf_json_enabled --value true
124+
```
125+
126+
The same policy denial appears in the JSONL file as this record, formatted here for readability:
127+
128+
```json
129+
{
130+
"class_uid": 4001,
131+
"class_name": "Network Activity",
132+
"category_uid": 4,
133+
"category_name": "Network Activity",
134+
"activity_id": 1,
135+
"activity_name": "Open",
136+
"type_uid": 400101,
137+
"type_name": "Network Activity: Open",
138+
"time": 1791996125789,
139+
"severity_id": 3,
140+
"severity": "Medium",
141+
"status_id": 2,
142+
"status": "Failure",
143+
"message": "Transparent TCP denied before relay: network connections not allowed by policy",
144+
"status_detail": "transparent_tcp_policy_denied",
145+
"metadata": {
146+
"version": "1.8.0",
147+
"product": {
148+
"name": "OpenShell Sandbox Supervisor",
149+
"vendor_name": "OpenShell",
150+
"version": "0.1.4"
151+
},
152+
"profiles": [
153+
"security_control",
154+
"network_proxy",
155+
"container",
156+
"host"
157+
],
158+
"uid": "b885957d-124a-4af7-9eef-75cc410ff070"
159+
},
160+
"device": {
161+
"hostname": "docker-desktop",
162+
"type_id": 99,
163+
"type": "Sandbox",
164+
"os": {
165+
"name": "Linux"
166+
}
167+
},
168+
"container": {
169+
"name": "docs-deny",
170+
"uid": "3949f6fc-2b84-45b2-a631-46debe41d84f"
171+
},
172+
"dst_endpoint": {
173+
"domain": "github.com",
174+
"ip": "198.18.0.2",
175+
"port": 443
176+
},
177+
"proxy_endpoint": {
178+
"ip": "127.0.0.1",
179+
"port": 3128
180+
},
181+
"actor": {
182+
"process": {
183+
"name": "/usr/bin/curl",
184+
"pid": 2187,
185+
"cmd_line": "curl https://github.com"
186+
}
187+
},
188+
"action_id": 2,
189+
"action": "Denied",
190+
"disposition_id": 2,
191+
"disposition": "Blocked",
192+
"is_src_dst_assignment_known": true
193+
}
194+
```
195+
196+
The gateway always writes to the configured `path`. When it rotates the file, daily by default, it renames the old file with the date and a unique suffix, such as `gateway-ocsf.jsonl.2026-10-13.<uuid>`. Supervisors write date-named files such as `openshell-ocsf.2026-10-14.log` instead. In the Helm chart, set `server.ocsfLog.enabled` for gateway events. For file locations, rotation, and schema version downgrades for older SIEMs, refer to [OCSF JSON Export](/observability/ocsf-json-export).
197+
198+
### OTLP Log Export
199+
200+
OTLP log export sends each OCSF event to a collector as an OTLP log record that contains the full JSON event. It is off by default, even when trace export is on. Set `enabled = true` in `[openshell.gateway.otlp.ocsf]` to turn it on for the gateway and for every sandbox supervisor the gateway starts. The records go to the `[openshell.gateway.otlp]` endpoint:
201+
202+
```toml
203+
[openshell.gateway.otlp]
204+
endpoint = "https://backend:4317"
205+
206+
[openshell.gateway.otlp.ocsf]
207+
enabled = true
208+
```
209+
210+
To send OCSF records to a different collector, such as one that feeds a SIEM, set `endpoint` in the table:
211+
212+
```toml
213+
[openshell.gateway.otlp.ocsf]
214+
enabled = true
215+
endpoint = "https://siem-backend:4317"
216+
```
217+
218+
In the Helm chart, set the table through `gatewayConfig`:
219+
220+
```yaml
221+
gatewayConfig:
222+
openshell.gateway.otlp.ocsf:
223+
enabled: true
224+
```
225+
226+
For the record format, collector authentication, delivery guarantees, and loss reporting, refer to [OCSF Log Export](/how-it-works/gateways/configuration#ocsf-log-export).
227+
228+
## OpenTelemetry Environment Variables
229+
230+
Only `gateway.toml` decides which signals are exported over OTLP and where they go. The gateway ignores the standard `OTEL_*` variables that would make those choices, such as `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_TRACES_EXPORTER`, and `OTEL_SDK_DISABLED`, for two reasons:
231+
232+
- OCSF events are security records, so where they go must be a deliberate choice. Platform tooling can add `OTEL_*` variables to a process, as the OpenTelemetry Operator does for annotated pods on Kubernetes. Such a variable must not start sending security records to a collector, redirect them, or turn them off.
233+
- The gateway passes its export settings to compute drivers and sandbox supervisors. Supervisors run in each sandbox and do not inherit the gateway's environment, so they could not follow a setting that exists only there.
234+
235+
The gateway honors the variables that tune export without changing where records go, such as `OTEL_TRACES_SAMPLER`, `OTEL_RESOURCE_ATTRIBUTES`, and the `OTEL_EXPORTER_OTLP_*_HEADERS`, `_COMPRESSION`, and `_TIMEOUT` variables. Supervisors receive only the OCSF collector headers from these. For the full list, refer to [Tuning](/how-it-works/gateways/configuration#tuning).
236+
237+
## Next Steps
238+
239+
- Configure the gateway's [OTLP export](/how-it-works/gateways/configuration#otlp-export) settings.
240+
- Learn how to [read the shorthand format](/observability/logging).
241+
- Learn what anonymous usage data OpenShell collects in [Telemetry](/observability/telemetry). It is separate from the signals on this page.

0 commit comments

Comments
 (0)