Repository navigation
Phase 3: per-engine OTLP tracing integrations #3766
Description
Summary
Phase 3 of distributed tracing (epic #3761, design cozystack/community#38): wire each supported managed engine to emit OTLP spans to the per-tenant collector (otel-traces.<tenant-ns>.svc:4317, from phase 2 #3763). One PR per engine, all linked here.
Per the design's §4 per-engine breakdown, depth and mechanism differ per engine:
- ClickHouse — native
opentelemetry_span_logsystem table (currently disabled viaremove="1"); re-enable + add an OTLP export path. - NATS — client-side instrumentation (server tracing emits NATS-format, not OTLP).
- Kafka — JVM/agent or client instrumentation.
- RabbitMQ — plugin/agent instrumentation.
- MariaDB — agent/proxy instrumentation.
- Redis — OTLP exporter/agent sidecar.
- Postgres —
pg_tracingextension (or client-side); a sidecar cannot produce statement-level spans.
Cross-cutting decision (from the design's open question)
Enablement follows the metrics model (platform-configured), not a per-app tracing.enabled toggle; the OTLP endpoint is platform-decided (the in-namespace collector Service), never carried in app values. The concrete Application→HelmRelease wiring is established by the first engine PR (ClickHouse) and reused by the rest.
Tracked under epic #3761.
Reacted by dosubot
Activity
scooby87 commentedon Aug 12, 2026
Scope update after verification
Deeper verification of each engine changed the picture: the managed servers do not emit OTLP spans themselves, so the original "one server-side integration PR per engine" premise does not hold.
system.opentelemetry_span_logbut has no native OTLP push; neither Vector nor Fluent Bit reads ClickHouse tables, so central export needs a bespoke shipper agent. Deferred as an explicit follow-up (not a clean chart PR).pg_tracingneedsshared_preload_libraries, which is preload-restricted in the chart (decision: not lifting it).This matches the design's own per-engine table (community#38 §4): the platform delivers transport/storage/tenancy/correlation, while span depth is an engine/client concern. Auto-instrumenting tenant workloads is a design non-goal.
Delivered (client-side): #3772 —
docs/observability/distributed-tracing.md: the collector endpoint, the standardOTEL_*env convention for instrumented workloads, tenant attributes, the honest per-engine table, the ClickHouse span_log + shipper caveat, and sampling/redaction/tenancy.Remaining (optional, bespoke): a ClickHouse span_log → OTLP shipper agent, if we decide the internal query spans are worth a custom forwarder.
docs(observability): add a distributed tracing guide (#3772)