Skip to content

Commit a0f8e42

Browse files
committed
feat(observability): export gateway traces over OTLP
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled by the presence of an `[openshell.gateway.otlp]` table with an endpoint; there is no separate toggle. Instrumented: - Request spans, one root per entrypoint, named for the RPC (`$service/$method`) or `{method} {path}` for plain HTTP. gRPC requests also carry `rpc.system`, `rpc.service`, and `rpc.method`, which RPC-aware backends use to build service maps. - Compute driver calls (create, delete, list, get, validate, watch) as client spans anchored on the `ComputeDriver` contract. - Store reads and writes. - Work with no inbound request: the sandbox reconcile sweep, the provider credential refresh tick, and driver watch events. Each roots its own trace, so the store calls they make are attributable instead of arriving as anonymous single-span traces. This is deliberately not exhaustive. Auth, policy evaluation, and middleware remain uninstrumented, as do store lifecycle calls (`ping`, `close`) that a readiness poll would turn into a span per tick. The aim is a working design at a reviewable size; coverage can grow against real traces. Design notes: - The TOML table owns whether and where to export. The SDK's `OTEL_*` variables own how: sampling, batching, and limits are not mirrored into gateway config. - Telemetry never blocks the gateway. A malformed endpoint degrades to export-off rather than failing startup. - Failed spans carry error status but no separate `error.type` attribute. Where a span wraps a `Result`-returning function, `#[instrument(err)]` sets the status and carries the message as a span event; the three spans built by hand, whose names are only known at runtime, use an `otlp::record_error` helper. The cause stays filterable through attributes those spans already carry (`grpc.code`, `http.response.status_code`). - Runtime export failures are silent: the SDK's `internal-logs` feature emitted nothing when enabled for testing, and its retry sits behind an experimental feature, so a dropped batch stays dropped. - The compute driver is reachable only through `TracedDriver::call`, so a call cannot skip its span. This is the client half of a client/server pair; when drivers move out of process it is also the single place trace context needs injecting. - Tests share one process-wide subscriber and in-memory exporter. `tracing` caches callsite interest globally, so a per-test subscriber leaves callsites permanently dark once an untraced test touches them. Trace context is not propagated yet: no W3C `traceparent` is injected into outbound driver calls, so spans from an out-of-process driver would not join the gateway's trace. The driver seam is the one place that injection needs to happen when it lands. The Helm chart is intentionally unchanged, so this cannot yet be enabled on a chart-deployed gateway. Adds a `mise run observability:*` task group that stands up Jaeger in k3d for local verification. Refs #2507 Signed-off-by: Kris Hicks <[email protected]>
1 parent 2b7f04f commit a0f8e42

20 files changed

Lines changed: 2336 additions & 201 deletions

File tree

‎Cargo.lock‎

Lines changed: 93 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,13 @@ tracing = "0.1"
6060
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
6161
tracing-appender = "0.2"
6262

63+
# OpenTelemetry — OTLP/gRPC export. Kept in lockstep with the workspace's
64+
# tonic 0.14 / prost 0.14 via opentelemetry-proto's `grpc-tonic` feature.
65+
opentelemetry = "0.32"
66+
opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] }
67+
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["grpc-tonic", "trace"] }
68+
tracing-opentelemetry = { version = "0.33", default-features = false, features = ["tracing-log"] }
69+
6370
# Metrics
6471
metrics = "0.24"
6572
metrics-exporter-prometheus = { version = "0.18", default-features = false, features = ["http-listener"] }

‎architecture/gateway.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -596,6 +596,27 @@ Driver-specific values that are not part of the inheritance allowlist
596596
(e.g. Podman `socket_path`, VM `vcpus`) only come from the driver's own
597597
table.
598598

599+
### OTLP export
600+
601+
`[openshell.gateway.otlp]` is the only enablement path for OpenTelemetry
602+
export: the table's presence is the on-switch, and `OTEL_EXPORTER_OTLP_ENDPOINT`
603+
is ignored so enablement has a single source. TOML decides whether and where
604+
to export; the SDK's `OTEL_*` variables tune how. Transport is OTLP over gRPC
605+
only.
606+
607+
Span emission requires no per-handler instrumentation. The `tower_http`
608+
`TraceLayer` in `multiplex.rs` opens a span per inbound request, and that span
609+
is the exported trace root, named for the RPC and carrying the request ID that
610+
also appears in the gateway's logs — the identifier that lets an operator pivot
611+
between a trace and its log lines.
612+
613+
Two invariants shape the failure behavior. Telemetry is diagnostic, so no OTLP
614+
failure stops the gateway from serving: a malformed endpoint is logged at
615+
startup and disables export. And export is best-effort — runtime failures are
616+
silent and a failed batch is dropped rather than retried, so missing traces are
617+
diagnosed against the collector, not from gateway logs. Buffered spans flush
618+
after the server loop exits so `SIGTERM` does not drop in-flight traces.
619+
599620
### Package-managed gateway registry
600621

601622
The CLI reads its active-gateway and per-gateway metadata from

‎crates/openshell-server/Cargo.toml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,12 @@ anyhow = { workspace = true }
6969
tracing = { workspace = true }
7070
tracing-subscriber = { workspace = true }
7171

72+
# OpenTelemetry (OTLP trace export, opt-in via [openshell.gateway.otlp])
73+
opentelemetry = { workspace = true }
74+
opentelemetry_sdk = { workspace = true }
75+
opentelemetry-otlp = { workspace = true }
76+
tracing-opentelemetry = { workspace = true }
77+
7278
# Metrics
7379
metrics = { workspace = true }
7480
metrics-exporter-prometheus = { workspace = true }
@@ -118,6 +124,12 @@ rcgen = { version = "0.13", features = ["crypto", "pem"] }
118124
tokio-tungstenite = { workspace = true }
119125
futures-util = "0.3"
120126
wiremock = "0.6"
127+
# `testing` provides InMemorySpanExporter, so span assertions do not need a
128+
# collector, a network hop, or a flush barrier.
129+
opentelemetry_sdk = { workspace = true, features = ["testing"] }
130+
# In-process OTLP collector, used by the one test that asserts the gateway
131+
# really puts spans on the wire rather than only into an exporter.
132+
opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace"] }
121133

122134
[lints]
123135
workspace = true

‎crates/openshell-server/src/cli.rs‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ use openshell_core::ComputeDriverKind;
1010
use openshell_core::config::DEFAULT_SERVER_PORT;
1111
use std::net::{IpAddr, SocketAddr};
1212
use std::path::PathBuf;
13-
use tracing::{info, warn};
13+
use tracing::{error, info, warn};
1414
use tracing_subscriber::EnvFilter;
1515

1616
use crate::certgen;
@@ -439,10 +439,20 @@ fn prepare_server_config(args: &mut RunArgs, matches: &ArgMatches) -> Result<Ser
439439
async fn run_from_args(mut args: RunArgs, matches: ArgMatches) -> Result<()> {
440440
let prepared = prepare_server_config(&mut args, &matches)?;
441441

442+
// A broken exporter degrades to no traces, never to a failed start. The
443+
// error is reported below, once the subscriber exists to carry it.
444+
let (otlp_provider, otlp_error) = crate::otlp::provider_for(
445+
prepared
446+
.config_file
447+
.as_ref()
448+
.and_then(|f| f.openshell.gateway.otlp.as_ref()),
449+
);
450+
442451
let tracing_log_bus = TracingLogBus::new();
443452
tracing_log_bus.install_subscriber(
444453
EnvFilter::try_from_default_env()
445454
.unwrap_or_else(|_| EnvFilter::new(&prepared.config.log_level)),
455+
otlp_provider.as_ref(),
446456
);
447457

448458
let has_client_ca = prepared
@@ -468,6 +478,18 @@ async fn run_from_args(mut args: RunArgs, matches: ArgMatches) -> Result<()> {
468478
if has_oidc {
469479
info!("OIDC authentication enabled");
470480
}
481+
if let Some(err) = &otlp_error {
482+
error!(
483+
error = %err,
484+
"OTLP exporting is configured but could not be started; continuing without it"
485+
);
486+
} else if let Some(otlp) = prepared
487+
.config_file
488+
.as_ref()
489+
.and_then(|f| f.openshell.gateway.otlp.as_ref())
490+
{
491+
info!(endpoint = %otlp.endpoint, "OTLP exporting enabled");
492+
}
471493
if prepared.config.auth.allow_unauthenticated_users {
472494
warn!(
473495
"Unauthenticated user access enabled — only use this for trusted local development or a fully trusted fronting proxy"
@@ -487,9 +509,17 @@ async fn run_from_args(mut args: RunArgs, matches: ArgMatches) -> Result<()> {
487509

488510
info!(bind = %prepared.config.bind_address, "Starting OpenShell server");
489511

490-
Box::pin(run_server(prepared, tracing_log_bus))
491-
.await
492-
.into_diagnostic()
512+
let result = Box::pin(run_server(prepared, tracing_log_bus)).await;
513+
514+
// Drain the batch span processor before the process exits, so spans from
515+
// the final requests are not lost on SIGTERM.
516+
if let Some(provider) = &otlp_provider
517+
&& let Err(err) = provider.shutdown()
518+
{
519+
warn!(error = %err, "OTLP tracer provider shutdown failed");
520+
}
521+
522+
result.into_diagnostic()
493523
}
494524

495525
fn parse_compute_driver(value: &str) -> std::result::Result<String, String> {

0 commit comments

Comments
 (0)