You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 98c3194
Browse filesBrowse the repository at this point in the historyBrowse files
docs(observability): add observability overview and traces pages
Add an Observability overview that lists the traces, metrics, and OCSF
events OpenShell produces, where each goes, and the minimal configuration
to enable it, linking to the detailed pages instead of repeating them.
Add a Traces page for what is traced, span names, resource attributes,
gateway names, and trace context, and point the OTLP Export reference
to it instead of describing spans there. Move metric naming conventions
into Gateway Metrics, and replace the stale denied-connection example in
OCSF JSON Export with a real record.
List every common OTEL_* environment variable in the gateway OTLP tuning
reference, split into the variables the gateway honors and the ones it
ignores, with the reasons.
Closes#4548
Signed-off-by: Kris Hicks <[email protected]>
Copy file name to clipboardExpand all lines: docs/how-it-works/gateways/configuration.mdx
+25-14Lines changed: 25 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -417,15 +417,9 @@ Use an `https://` endpoint for TLS. The gateway verifies the collector certifica
417
417
418
418
The OpenTelemetry SDK logs export failures after startup. Spans in a failed batch are dropped rather than retried.
419
419
420
-
`service_name` sets the gateway's `service.name` resource attribute and defaults to `openshell-gateway`. The gateway also reports `service.version`, `openshell.gateway.name` from the gateway's configured `name`, and `openshell.gateway.compute_driver`.
420
+
`service_name` sets the gateway's `service.name` resource attribute and defaults to `openshell-gateway`.
421
421
422
-
Inbound gRPC and HTTP requests produce server spans named for the RPC or HTTP method. Compute-driver operations appear as child spans. Internal reconciliation and driver-watch loops create operation roots because no inbound request supplies a parent, and credential refresh creates one only when a credential is due or needs cleanup. The gateway continues valid W3C `traceparent` context and starts a new trace when none is supplied. Request spans carry `method`, `path`, and the `request_id` that also appears in gateway logs. Store operation spans, health endpoint spans, and spans for RPCs that sandbox supervisors call on a timer (`GetSandboxConfig` and `ReportProviderReadiness`, including its peer-forwarded form) use DEBUG level. The default INFO filter does not export them. Set the gateway log level to `debug` to include them.
423
-
424
-
The gateway forwards the trace endpoint, configured gateway name, configured compute driver, and W3C trace context to managed external drivers. Built-in drivers also export their spans to the trace collector through dedicated in-process providers. Driver spans retain the gateway trace context, use a distinct service name such as `openshell-driver-docker` or `openshell-driver-podman`, and carry the same `openshell.gateway.name` and `openshell.gateway.compute_driver` resource attributes as gateway spans. Compute-driver client and server spans use the same fully qualified protobuf operation name, such as `openshell.compute.v1.ComputeDriver/CreateSandbox`, in both the span name and `rpc.method`. The service name and span kind distinguish each side. Backend-prefixed child spans identify implementation work. A streaming watch records a terminal status when observed; consumer teardown without a terminal status leaves the span status unset. Operator-run external drivers own their own telemetry configuration.
425
-
426
-
Sandbox supervisors export spans to the trace collector, so its endpoint must also be reachable from sandboxes.
427
-
428
-
Spans for routine or high-volume operations use DEBUG level, and the default INFO level does not export them. Set the gateway or sandbox log level to `debug` to include them.
422
+
The gateway passes the trace endpoint to compute drivers and sandbox supervisors, which export their own spans to it, so the trace collector must also be reachable from sandboxes. Spans for routine or high-volume operations use DEBUG level and are exported only when the gateway or sandbox log level is `debug`. For what is traced, span names, and resource attributes, refer to [Traces](/observability/traces).
429
423
430
424
For Helm deployments, set `server.otlp.endpoint` to render this table. The
431
425
optional `server.otlp.serviceName` value overrides the gateway service name;
@@ -490,28 +484,45 @@ Metrics include `openshell_ocsf_otlp_queued_total`, `openshell_ocsf_otlp_exporte
490
484
491
485
### Tuning
492
486
493
-
This table decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys:
487
+
`[openshell.gateway.otlp]`decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys. They apply to the gateway and its in-process compute drivers. The gateway does not pass them to sandbox supervisors, which receive only the endpoints, the OCSF schema version, and the [OCSF collector headers](#ocsf-log-export).
488
+
489
+
The gateway honors these variables:
494
490
495
491
| Variable | Effect |
496
492
|---|---|
493
+
| `OTEL_SERVICE_NAME` | The gateway's `service.name` when `service_name` is not set in `[openshell.gateway.otlp]`. Compute-driver and supervisor service names are fixed. |
494
+
| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes, such as `deployment.environment=prod`. |
497
495
| `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | Sampling strategy and ratio. Defaults to `parentbased_always_on`. |
| `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT` | Exporter transport tuning for every signal. |
502
499
| `OTEL_EXPORTER_OTLP_COMPRESSION` | Set to `gzip` to compress exports of every signal. `zstd` is not supported. |
500
+
| `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION`, `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` | Exporter transport tuning for spans only. These override the general variables. |
503
501
| `OTEL_EXPORTER_OTLP_LOGS_HEADERS`, `OTEL_EXPORTER_OTLP_LOGS_COMPRESSION`, `OTEL_EXPORTER_OTLP_LOGS_TIMEOUT` | Exporter transport tuning for OCSF log records only. These override the general variables. |
504
-
| `OTEL_EXPORTER_OTLP_PROTOCOL` | No effect. The gateway is built with the gRPC exporter only. |
505
-
| `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | No effect. OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). |
502
+
| `SSL_CERT_FILE`, `SSL_CERT_DIR` | Additional trust roots for `https://` endpoints, such as a private CA. |
503
+
504
+
These variables have no effect. Only `[openshell.gateway.otlp]` decides which signals are exported and where they go, for two reasons:
505
+
506
+
- OCSF events are security records, so where they go must be a deliberate choice. Platform tooling can add `OTEL_*` variables to a process, as the OpenTelemetry Operator does for annotated pods on Kubernetes. Such a variable must not start sending security records to a collector, redirect them, or turn them off.
507
+
- The gateway passes its export settings to compute drivers and sandbox supervisors. Supervisors run in each sandbox and do not inherit the gateway's environment, so they could not follow a setting that exists only there.
508
+
509
+
510
+
| Variable | Reason |
511
+
|---|---|
512
+
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Enablement has one source, `[openshell.gateway.otlp]`, so an environment variable cannot silently turn export on or redirect it. |
513
+
| `OTEL_SDK_DISABLED`, `OTEL_TRACES_EXPORTER`, `OTEL_LOGS_EXPORTER`, `OTEL_METRICS_EXPORTER` | `[openshell.gateway.otlp]` decides which signals are exported. To disable export, omit it. |
514
+
| `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | The gateway is built with the gRPC exporter only. |
515
+
| `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_EXPORTER_OTLP_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_KEY`, and their per-signal forms | The endpoint scheme selects TLS, and collector certificates are verified against the system trust roots. Client certificates are not supported. |
| `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). |
518
+
| `OTEL_EXPORTER_OTLP_METRICS_*`, `OTEL_METRIC_EXPORT_INTERVAL` | The gateway serves metrics for Prometheus scraping and does not export them over OTLP. |
`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` are deliberately ignored. Enablement has one source, so an environment variable cannot silently turn export on or redirect it. `OTEL_RESOURCE_ATTRIBUTES` does apply and adds attributes, but a `service_name` set here wins over `OTEL_SERVICE_NAME`.
514
-
515
526
The gateway flushes buffered spans during shutdown, so spans from in-flight requests survive a `SIGTERM`.
Copy file name to clipboardExpand all lines: docs/observability/gateway-metrics.mdx
+18Lines changed: 18 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,7 @@ title: "Gateway Metrics"
5
5
sidebar-title: "Gateway Metrics"
6
6
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments."
Copy file name to clipboardExpand all lines: docs/observability/ocsf-json-export.mdx
+51-15Lines changed: 51 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -129,8 +129,8 @@ Each JSONL file line, or the payload after the stderr marker, is a complete OCSF
129
129
"metadata": {
130
130
"product": {
131
131
"name": "OpenShell Sandbox Supervisor",
132
-
"vendor_name": "NVIDIA",
133
-
"version": "0.3.0"
132
+
"vendor_name": "OpenShell",
133
+
"version": "0.1.4"
134
134
},
135
135
"version": "1.8.0"
136
136
},
@@ -159,38 +159,73 @@ Each JSONL file line, or the payload after the stderr marker, is a complete OCSF
159
159
}
160
160
```
161
161
162
-
And a denied connection:
162
+
And a denied connection, produced when `curl https://github.com` runs in a sandbox without a network policy that allows `github.com`:
163
163
164
164
```json
165
165
{
166
166
"class_uid": 4001,
167
167
"class_name": "Network Activity",
168
+
"category_uid": 4,
169
+
"category_name": "Network Activity",
168
170
"activity_id": 1,
169
171
"activity_name": "Open",
172
+
"type_uid": 400101,
173
+
"type_name": "Network Activity: Open",
174
+
"time": 1791996125789,
170
175
"severity_id": 3,
171
176
"severity": "Medium",
172
177
"status_id": 2,
173
178
"status": "Failure",
174
-
"action_id": 2,
175
-
"action": "Denied",
176
-
"disposition_id": 2,
177
-
"disposition": "Blocked",
178
-
"status_detail": "no matching policy",
179
-
"message": "CONNECT denied httpbin.org:443",
179
+
"message": "Transparent TCP denied before relay: network connections not allowed by policy",
@@ -315,5 +350,6 @@ Shorthand stderr output follows the diagnostic log filter; its local file is ava
315
350
316
351
## Next Steps
317
352
353
+
- See the [Observability Overview](/observability/overview) to compare JSONL files with OTLP log export.
318
354
- Learn how to [read the shorthand format](/observability/logging) for real-time monitoring.
319
355
- Refer to the [OCSF specification](https://schema.ocsf.io/) for the full schema reference.
OpenShell produces three kinds of observability signals: traces, metrics, and OCSF security events. This page describes what each signal is for and how to turn it on, and links to the page that covers it in detail.
The examples on this page are `gateway.toml` tables. In the Helm chart, set each table under `gatewayConfig`, as described in [Configure Chart Values](/kubernetes/setup#configure-chart-values).
22
+
23
+
All OTLP export uses OTLP over gRPC. Configure it in `gateway.toml`, and the gateway passes the settings to the sandbox supervisors it starts. Supervisors connect to the collector directly, so the collector must be reachable from sandboxes as well as from the gateway. The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. Refer to [Tuning](/how-it-works/gateways/configuration#tuning) for the full list.
24
+
25
+
## Traces
26
+
27
+
Traces show how the gateway handles each request, including the compute-driver work it performs, and how each sandbox supervisor starts up. Use them to find where time goes in an operation such as sandbox creation.
28
+
29
+
To export traces, set an OTLP collector endpoint in `gateway.toml`:
For span names, resource attributes, and gateway names, refer to [Traces](/observability/traces).
37
+
38
+
## Metrics
39
+
40
+
The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity, on a listener separate from its API. Use them to alert on saturation and to choose autoscaling signals.
41
+
42
+
To enable the listener, set `metrics_bind_address` in `gateway.toml`:
43
+
44
+
```toml
45
+
[openshell.gateway]
46
+
metrics_bind_address = "0.0.0.0:9090"
47
+
```
48
+
49
+
The port is an example. Until the listener is enabled, the gateway serves no metrics. The endpoint has no authentication, so restrict who can reach it. For metric names, deployment defaults, the catalog, and scrape configuration, refer to [Gateway Metrics](/observability/gateway-metrics).
50
+
51
+
## OCSF Events
52
+
53
+
OCSF events record security-relevant activity, such as network and HTTP policy decisions, process lifecycle, SSH sessions, and configuration changes. They use the [Open Cybersecurity Schema Framework](https://ocsf.io). Sandbox supervisors produce most of them, and the gateway produces events for its own decisions, such as approving or rejecting draft policy rules. The same events are available in three forms.
54
+
55
+
### Shorthand Logs
56
+
57
+
Shorthand logs are always on. Each event appears as a one-line summary in the gateway console, in the sandbox log stream that `openshell logs` and the TUI read, and in the supervisor's log file. For example, this policy denial is the event a sandbox produces when `curl https://github.com` runs without a network policy that allows `github.com`:
To read the format, refer to [Sandbox Logging](/observability/logging). To find logs for each compute driver, refer to [Accessing Logs](/observability/accessing-logs).
64
+
65
+
### JSONL Files
66
+
67
+
JSONL files hold the full OCSF record, one JSON object per line, for a log shipper to collect. The gateway and supervisors are enabled separately:
68
+
69
+
```toml
70
+
# Gateway events, in gateway.toml.
71
+
[openshell.gateway.ocsf_log]
72
+
path = "/var/log/openshell/gateway-ocsf.jsonl"
73
+
```
74
+
75
+
```shell
76
+
# Supervisor events, for every sandbox.
77
+
openshell settings set --global --key ocsf_json_enabled --value true
78
+
```
79
+
80
+
For the record structure, file locations, rotation, and schema version downgrades for older SIEMs, refer to [OCSF JSON Export](/observability/ocsf-json-export).
81
+
82
+
### OTLP Log Export
83
+
84
+
OTLP log export sends each OCSF event to a collector as an OTLP log record that contains the full JSON event. It is off by default, even when trace export is on. Set `enabled = true` in `[openshell.gateway.otlp.ocsf]` to turn it on for the gateway and for every sandbox supervisor the gateway starts:
The records go to the `[openshell.gateway.otlp]` endpoint unless the table sets its own `endpoint`, such as a collector that feeds a SIEM. For the record format, collector authentication, delivery guarantees, and loss reporting, refer to [OCSF Log Export](/how-it-works/gateways/configuration#ocsf-log-export).
95
+
96
+
## Next Steps
97
+
98
+
- Learn how spans are named and identified in [Traces](/observability/traces).
99
+
- Learn how to [read the shorthand format](/observability/logging).
100
+
- Learn what anonymous usage data OpenShell collects in [Telemetry](/observability/telemetry). It is separate from the signals on this page.
0 commit comments