Repository navigation
Commit a0f8e42
committed
feat(observability): export gateway traces over OTLP
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled
by the presence of an `[openshell.gateway.otlp]` table with an endpoint;
there is no separate toggle.
Instrumented:
- Request spans, one root per entrypoint, named for the RPC
(`$service/$method`) or `{method} {path}` for plain HTTP. gRPC requests
also carry `rpc.system`, `rpc.service`, and `rpc.method`, which RPC-aware
backends use to build service maps.
- Compute driver calls (create, delete, list, get, validate, watch) as
client spans anchored on the `ComputeDriver` contract.
- Store reads and writes.
- Work with no inbound request: the sandbox reconcile sweep, the provider
credential refresh tick, and driver watch events. Each roots its own
trace, so the store calls they make are attributable instead of
arriving as anonymous single-span traces.
This is deliberately not exhaustive. Auth, policy evaluation, and
middleware remain uninstrumented, as do store lifecycle calls (`ping`,
`close`) that a readiness poll would turn into a span per tick. The aim
is a working design at a reviewable size; coverage can grow against real
traces.
Design notes:
- The TOML table owns whether and where to export. The SDK's `OTEL_*`
variables own how: sampling, batching, and limits are not mirrored into
gateway config.
- Telemetry never blocks the gateway. A malformed endpoint degrades to
export-off rather than failing startup.
- Failed spans carry error status but no separate `error.type` attribute.
Where a span wraps a `Result`-returning function, `#[instrument(err)]`
sets the status and carries the message as a span event; the three
spans built by hand, whose names are only known at runtime, use an
`otlp::record_error` helper. The cause stays filterable through
attributes those spans already carry (`grpc.code`,
`http.response.status_code`).
- Runtime export failures are silent: the SDK's `internal-logs` feature
emitted nothing when enabled for testing, and its retry sits behind an
experimental feature, so a dropped batch stays dropped.
- The compute driver is reachable only through `TracedDriver::call`, so a
call cannot skip its span. This is the client half of a client/server
pair; when drivers move out of process it is also the single place
trace context needs injecting.
- Tests share one process-wide subscriber and in-memory exporter.
`tracing` caches callsite interest globally, so a per-test subscriber
leaves callsites permanently dark once an untraced test touches them.
Trace context is not propagated yet: no W3C `traceparent` is injected into
outbound driver calls, so spans from an out-of-process driver would not join
the gateway's trace. The driver seam is the one place that injection needs
to happen when it lands.
The Helm chart is intentionally unchanged, so this cannot yet be enabled
on a chart-deployed gateway. Adds a `mise run observability:*` task group
that stands up Jaeger in k3d for local verification.
Refs #2507
Signed-off-by: Kris Hicks <[email protected]>1 parent 2b7f04f commit a0f8e42
20 files changed
Lines changed: 2336 additions & 201 deletions
File tree
- architecture
- crates/openshell-server
- src
- compute
- grpc
- persistence
- docs/reference
- tasks
- scripts
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
63 | 70 | | |
64 | 71 | | |
65 | 72 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
596 | 596 | | |
597 | 597 | | |
598 | 598 | | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
599 | 620 | | |
600 | 621 | | |
601 | 622 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
72 | 78 | | |
73 | 79 | | |
74 | 80 | | |
| |||
118 | 124 | | |
119 | 125 | | |
120 | 126 | | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
121 | 133 | | |
122 | 134 | | |
123 | 135 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
439 | 439 | | |
440 | 440 | | |
441 | 441 | | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
442 | 451 | | |
443 | 452 | | |
444 | 453 | | |
445 | 454 | | |
| 455 | + | |
446 | 456 | | |
447 | 457 | | |
448 | 458 | | |
| |||
468 | 478 | | |
469 | 479 | | |
470 | 480 | | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
471 | 493 | | |
472 | 494 | | |
473 | 495 | | |
| |||
487 | 509 | | |
488 | 510 | | |
489 | 511 | | |
490 | | - | |
491 | | - | |
492 | | - | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
493 | 523 | | |
494 | 524 | | |
495 | 525 | | |
| |||
0 commit comments