|
| 1 | +--- |
| 2 | +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. |
| 3 | +# SPDX-License-Identifier: Apache-2.0 |
| 4 | +title: "Observability Overview" |
| 5 | +sidebar-title: "Overview" |
| 6 | +description: "The traces, metrics, and OCSF events that OpenShell produces, where each one goes, and how to enable it." |
| 7 | +keywords: "Generative AI, Cybersecurity, Observability, OpenTelemetry, OTLP, Traces, Metrics, Prometheus, OCSF, SIEM" |
| 8 | +position: 1 |
| 9 | +--- |
| 10 | + |
| 11 | +OpenShell produces three kinds of observability signals: traces, metrics, and OCSF security events. This page describes what each signal covers and how to enable it, and links to the detailed page for each one. |
| 12 | + |
| 13 | +| Signal | Produced by | Destination | Enabled by default | |
| 14 | +|---|---|---|---| |
| 15 | +| Traces | Gateway, compute drivers, sandbox supervisors | OTLP collector | No | |
| 16 | +| Metrics | Gateway | Prometheus scrape | Yes in the Helm chart. No for the standalone gateway. | |
| 17 | +| OCSF events, shorthand | Gateway, sandbox supervisors | Gateway console, `openshell logs`, supervisor log file | Yes | |
| 18 | +| OCSF events, JSONL | Gateway, sandbox supervisors | Local file | No | |
| 19 | +| OCSF events, OTLP | Gateway, sandbox supervisors | OTLP collector | No | |
| 20 | + |
| 21 | +All OTLP export uses OTLP over gRPC. Configure it in the gateway, and the gateway passes the settings to the sandbox supervisors it starts. Supervisors connect to the collector directly, so the collector must be reachable from sandboxes as well as from the gateway. |
| 22 | + |
| 23 | +## Traces |
| 24 | + |
| 25 | +Traces show how the gateway handles each request, including the compute-driver work it performs, and how each sandbox supervisor starts up. At debug level, they also show how supervisors handle egress connections. Use them to find where time goes in an operation such as sandbox creation. |
| 26 | + |
| 27 | +To export traces, set an OTLP collector endpoint in `gateway.toml`: |
| 28 | + |
| 29 | +```toml |
| 30 | +[openshell.gateway.otlp] |
| 31 | +endpoint = "https://backend:4317" |
| 32 | +``` |
| 33 | + |
| 34 | +In the Helm chart, set `server.otlp.endpoint`. For log levels, TLS, sampling, and the `OTEL_*` tuning variables, refer to [OTLP Export](/how-it-works/gateways/configuration#otlp-export). |
| 35 | + |
| 36 | +### Span Names |
| 37 | + |
| 38 | +Each component names its spans after the operation it performs: |
| 39 | + |
| 40 | +| Component | Span names | Examples | |
| 41 | +|---|---|---| |
| 42 | +| Gateway requests | The gRPC service and method, or the HTTP method for other requests | `openshell.v1.OpenShell/CreateSandbox`, `GET` | |
| 43 | +| Gateway background work | The task. These spans start their own traces. | `reconcile.sandboxes`, `refresh.provider_credentials` | |
| 44 | +| Compute drivers | The compute-driver RPC, with child spans prefixed by the driver name | `openshell.compute.v1.ComputeDriver/CreateSandbox`, `docker.provision` | |
| 45 | +| Sandbox supervisors | Prefixed with `supervisor.` | `supervisor.startup`, `supervisor.policy.load` | |
| 46 | + |
| 47 | +The gateway continues a W3C `traceparent` sent by the caller, so a client's trace can include the gateway's spans for its requests. |
| 48 | + |
| 49 | +### Resource Attributes |
| 50 | + |
| 51 | +Resource attributes identify which process produced a span: |
| 52 | + |
| 53 | +| Attribute | Value | Set on | |
| 54 | +|---|---|---| |
| 55 | +| `service.name` | `openshell-gateway`, `openshell-driver-<driver>`, or `openshell-supervisor` | Every span | |
| 56 | +| `service.version` | The OpenShell version | Every span | |
| 57 | +| `openshell.gateway.name` | The gateway name | Every span | |
| 58 | +| `openshell.gateway.compute_driver` | The configured compute driver, such as `kubernetes` | Gateway and compute-driver spans | |
| 59 | +| `openshell.sandbox.id` | The sandbox ID | Supervisor spans | |
| 60 | + |
| 61 | +The gateway name comes from `name` under `[openshell.gateway]`, the `--name` flag, or `OPENSHELL_GATEWAY_NAME`, and defaults to `openshell`. The Helm chart sets it with `server.name`, which defaults to the chart's full name. Give each installation that reports to the same collector its own name, including each high-availability deployment, so that you can tell their spans apart. The replicas of one installation share its name. |
| 62 | + |
| 63 | +To override the gateway's `service.name`, set `service_name` under `[openshell.gateway.otlp]`. To add your own attributes, such as `deployment.environment`, set `OTEL_RESOURCE_ATTRIBUTES` in the gateway's environment. |
| 64 | + |
| 65 | +## Metrics |
| 66 | + |
| 67 | +The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity. They are served on a separate listener. |
| 68 | + |
| 69 | +To enable the listener, set `metrics_bind_address` in `gateway.toml`: |
| 70 | + |
| 71 | +```toml |
| 72 | +[openshell.gateway] |
| 73 | +metrics_bind_address = "0.0.0.0:9090" |
| 74 | +``` |
| 75 | + |
| 76 | +The Helm chart serves metrics on port `9090` by default. The endpoint has no authentication, so restrict who can reach it. For the metric catalog and scrape configuration, refer to [Gateway Metrics](/observability/gateway-metrics). |
| 77 | + |
| 78 | +### Metric Names |
| 79 | + |
| 80 | +Every metric name starts with `openshell_`, followed by the subsystem that records it: |
| 81 | + |
| 82 | +| Prefix | Covers | |
| 83 | +|---|---| |
| 84 | +| `openshell_server_` | API requests, readiness, supervisor sessions, relays, and requests forwarded between replicas | |
| 85 | +| `openshell_gateway_interceptor_` | Gateway interceptor evaluations, failures, and latency | |
| 86 | +| `openshell_ocsf_log_` | The gateway's OCSF JSONL file | |
| 87 | +| `openshell_ocsf_otlp_` | OCSF log export over OTLP | |
| 88 | + |
| 89 | +Names follow Prometheus conventions. Counters end in `_total`, and names with a unit end in that unit, such as `_seconds` or `_bytes`. Metrics carry no gateway name or replica label. Your scrape configuration identifies where each series came from, so keep the `namespace` and `pod` labels, or their equivalents, when you scrape each replica. |
| 90 | + |
| 91 | +## OCSF Events |
| 92 | + |
| 93 | +OCSF events record security-relevant activity, such as network and HTTP policy decisions, process lifecycle, SSH sessions, and configuration changes. They use the [Open Cybersecurity Schema Framework](https://ocsf.io). Sandbox supervisors produce most of them, and the gateway produces events for its own decisions, such as approving or rejecting draft policy rules. The same events are available in three forms. |
| 94 | + |
| 95 | +### Shorthand Logs |
| 96 | + |
| 97 | +Shorthand logs are always on. Each event appears as a one-line summary in the gateway console, in the sandbox log stream that `openshell logs` and the TUI read, and in the supervisor's log file: |
| 98 | + |
| 99 | +```shell |
| 100 | +openshell logs my-sandbox |
| 101 | +``` |
| 102 | + |
| 103 | +For example, this policy denial is the event a sandbox produces when `curl https://github.com` runs without a network policy that allows `github.com`: |
| 104 | + |
| 105 | +```text |
| 106 | +[1791996125.789] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(2187) -> github.com:443 [reason:transparent_tcp_policy_denied] |
| 107 | +``` |
| 108 | + |
| 109 | +To read the format, refer to [Sandbox Logging](/observability/logging). To find logs for each compute driver, refer to [Accessing Logs](/observability/accessing-logs). |
| 110 | + |
| 111 | +### JSONL Files |
| 112 | + |
| 113 | +JSONL files hold the full OCSF record, one JSON object per line, for a log shipper to collect. The gateway and supervisors are enabled separately: |
| 114 | + |
| 115 | +```toml |
| 116 | +# Gateway events, in gateway.toml. |
| 117 | +[openshell.gateway.ocsf_log] |
| 118 | +path = "/var/log/openshell/gateway-ocsf.jsonl" |
| 119 | +``` |
| 120 | + |
| 121 | +```shell |
| 122 | +# Supervisor events, for every sandbox. |
| 123 | +openshell settings set --global --key ocsf_json_enabled --value true |
| 124 | +``` |
| 125 | + |
| 126 | +The same policy denial appears in the JSONL file as this record, formatted here for readability: |
| 127 | + |
| 128 | +```json |
| 129 | +{ |
| 130 | + "class_uid": 4001, |
| 131 | + "class_name": "Network Activity", |
| 132 | + "category_uid": 4, |
| 133 | + "category_name": "Network Activity", |
| 134 | + "activity_id": 1, |
| 135 | + "activity_name": "Open", |
| 136 | + "type_uid": 400101, |
| 137 | + "type_name": "Network Activity: Open", |
| 138 | + "time": 1791996125789, |
| 139 | + "severity_id": 3, |
| 140 | + "severity": "Medium", |
| 141 | + "status_id": 2, |
| 142 | + "status": "Failure", |
| 143 | + "message": "Transparent TCP denied before relay: network connections not allowed by policy", |
| 144 | + "status_detail": "transparent_tcp_policy_denied", |
| 145 | + "metadata": { |
| 146 | + "version": "1.8.0", |
| 147 | + "product": { |
| 148 | + "name": "OpenShell Sandbox Supervisor", |
| 149 | + "vendor_name": "OpenShell", |
| 150 | + "version": "0.1.4" |
| 151 | + }, |
| 152 | + "profiles": [ |
| 153 | + "security_control", |
| 154 | + "network_proxy", |
| 155 | + "container", |
| 156 | + "host" |
| 157 | + ], |
| 158 | + "uid": "b885957d-124a-4af7-9eef-75cc410ff070" |
| 159 | + }, |
| 160 | + "device": { |
| 161 | + "hostname": "docker-desktop", |
| 162 | + "type_id": 99, |
| 163 | + "type": "Sandbox", |
| 164 | + "os": { |
| 165 | + "name": "Linux" |
| 166 | + } |
| 167 | + }, |
| 168 | + "container": { |
| 169 | + "name": "docs-deny", |
| 170 | + "uid": "3949f6fc-2b84-45b2-a631-46debe41d84f" |
| 171 | + }, |
| 172 | + "dst_endpoint": { |
| 173 | + "domain": "github.com", |
| 174 | + "ip": "198.18.0.2", |
| 175 | + "port": 443 |
| 176 | + }, |
| 177 | + "proxy_endpoint": { |
| 178 | + "ip": "127.0.0.1", |
| 179 | + "port": 3128 |
| 180 | + }, |
| 181 | + "actor": { |
| 182 | + "process": { |
| 183 | + "name": "/usr/bin/curl", |
| 184 | + "pid": 2187, |
| 185 | + "cmd_line": "curl https://github.com" |
| 186 | + } |
| 187 | + }, |
| 188 | + "action_id": 2, |
| 189 | + "action": "Denied", |
| 190 | + "disposition_id": 2, |
| 191 | + "disposition": "Blocked", |
| 192 | + "is_src_dst_assignment_known": true |
| 193 | +} |
| 194 | +``` |
| 195 | + |
| 196 | +The gateway always writes to the configured `path`. When it rotates the file, daily by default, it renames the old file with the date and a unique suffix, such as `gateway-ocsf.jsonl.2026-10-13.<uuid>`. Supervisors write date-named files such as `openshell-ocsf.2026-10-14.log` instead. In the Helm chart, set `server.ocsfLog.enabled` for gateway events. For file locations, rotation, and schema version downgrades for older SIEMs, refer to [OCSF JSON Export](/observability/ocsf-json-export). |
| 197 | + |
| 198 | +### OTLP Log Export |
| 199 | + |
| 200 | +OTLP log export sends each OCSF event to a collector as an OTLP log record that contains the full JSON event. It is off by default, even when trace export is on. Set `enabled = true` in `[openshell.gateway.otlp.ocsf]` to turn it on for the gateway and for every sandbox supervisor the gateway starts. The records go to the `[openshell.gateway.otlp]` endpoint: |
| 201 | + |
| 202 | +```toml |
| 203 | +[openshell.gateway.otlp] |
| 204 | +endpoint = "https://backend:4317" |
| 205 | + |
| 206 | +[openshell.gateway.otlp.ocsf] |
| 207 | +enabled = true |
| 208 | +``` |
| 209 | + |
| 210 | +To send OCSF records to a different collector, such as one that feeds a SIEM, set `endpoint` in the table: |
| 211 | + |
| 212 | +```toml |
| 213 | +[openshell.gateway.otlp.ocsf] |
| 214 | +enabled = true |
| 215 | +endpoint = "https://siem-backend:4317" |
| 216 | +``` |
| 217 | + |
| 218 | +In the Helm chart, set the table through `gatewayConfig`: |
| 219 | + |
| 220 | +```yaml |
| 221 | +gatewayConfig: |
| 222 | + openshell.gateway.otlp.ocsf: |
| 223 | + enabled: true |
| 224 | +``` |
| 225 | +
|
| 226 | +For the record format, collector authentication, delivery guarantees, and loss reporting, refer to [OCSF Log Export](/how-it-works/gateways/configuration#ocsf-log-export). |
| 227 | +
|
| 228 | +## OpenTelemetry Environment Variables |
| 229 | +
|
| 230 | +Only `gateway.toml` decides which signals are exported over OTLP and where they go. The gateway ignores the standard `OTEL_*` variables that would make those choices, such as `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_TRACES_EXPORTER`, and `OTEL_SDK_DISABLED`, for two reasons: |
| 231 | + |
| 232 | +- OCSF events are security records, so where they go must be a deliberate choice. Platform tooling can add `OTEL_*` variables to a process, as the OpenTelemetry Operator does for annotated pods on Kubernetes. Such a variable must not start sending security records to a collector, redirect them, or turn them off. |
| 233 | +- The gateway passes its export settings to compute drivers and sandbox supervisors. Supervisors run in each sandbox and do not inherit the gateway's environment, so they could not follow a setting that exists only there. |
| 234 | + |
| 235 | +The gateway honors the variables that tune export without changing where records go, such as `OTEL_TRACES_SAMPLER`, `OTEL_RESOURCE_ATTRIBUTES`, and the `OTEL_EXPORTER_OTLP_*_HEADERS`, `_COMPRESSION`, and `_TIMEOUT` variables. Supervisors receive only the OCSF collector headers from these. For the full list, refer to [Tuning](/how-it-works/gateways/configuration#tuning). |
| 236 | + |
| 237 | +## Next Steps |
| 238 | + |
| 239 | +- Configure the gateway's [OTLP export](/how-it-works/gateways/configuration#otlp-export) settings. |
| 240 | +- Learn how to [read the shorthand format](/observability/logging). |
| 241 | +- Learn what anonymous usage data OpenShell collects in [Telemetry](/observability/telemetry). It is separate from the signals on this page. |
0 commit comments