Skip to content

Commit bd75a12

Browse files
committed
Refine tracing design: staged A→B ingest grounded in current architecture
Direct-to-vtinsert MVP (mirrors logs) then an opt-in OpenTelemetry Collector gateway; ExternalName redirect via cozystack-basics; poller readiness gate on VTCluster; clarify WorkloadMonitor is operational-only so tracing opt-in lives in app values. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent f05d4bd commit bd75a12

1 file changed

Lines changed: 26 additions & 16 deletions

File tree

  • design-proposals/distributed-tracing

‎design-proposals/distributed-tracing/README.md‎

Lines changed: 26 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ Metrics storage is declared as a list of tiers in values and rendered into Victo
2929

3030
Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`.
3131

32-
Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy.
32+
Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status for the dashboard and billing surfaces — it does not emit scrape configs, and tracing opt-in therefore belongs in each app's `values.yaml`, not in `WorkloadMonitor`.
3333

3434
Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim.
3535

@@ -103,17 +103,21 @@ spec:
103103
{{- end }}
104104
```
105105

106-
The monitoring HelmRelease readiness gate keys on the new `VTCluster` the same way it keys on `VLCluster` today.
106+
The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as it does for `VLCluster` today: `waitStrategy: poller` plus a `healthCheckExprs` entry that waits for the CR's `status.updateStatus == 'operational'` (see `packages/extra/monitoring/templates/helmrelease.yaml`). Without the poller gate the release flips Ready as soon as Helm applies the CR — the exact silent-black-hole failure mode that motivated `cozystack/cozystack#3181` for logs.
107107

108-
### 2. OTLP ingest via an OpenTelemetry Collector
108+
### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway)
109109

110-
VictoriaTraces ingests OTLP natively, but a collector in front gives the platform a stable tenant-facing endpoint, sampling/rate-limiting, and a place to attach the `tenant:` resource attribute — the same role vmagent and fluent-bit play for metrics and logs. Deploy an **OpenTelemetry Collector** in `cozy-monitoring`, following the `packages/system/monitoring-agents` deployment/service pattern:
110+
`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert-generic`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, with **no app-visible endpoint change between stages**.
111+
112+
**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery.
113+
114+
**Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in.
111115

112116
- Receivers: `otlp` on gRPC `4317` and HTTP `4318`.
113117
- Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`.
114118
- Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend.
115119

116-
Tenants target an in-cluster OTLP endpoint (e.g. `otel-collector.cozy-monitoring.svc`); per-tenant namespaces get an ExternalName redirect to the root collector, mirroring the existing `vlinsert-generic` logs redirect, so the central-vs-isolated choice works identically for traces.
120+
Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317`); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role.
117121

118122
### 3. Grafana datasource and correlation
119123

@@ -149,8 +153,10 @@ How an app emits spans depends on the engine:
149153

150154
```mermaid
151155
flowchart LR
152-
app["Managed app<br/>(tracing.enabled)"] -- OTLP 4317/4318 --> col["OpenTelemetry Collector<br/>cozy-monitoring"]
153-
col -- OTLP --> vt["VictoriaTraces<br/>VTCluster (vtinsert→vtstorage)"]
156+
app["Managed app<br/>(tracing.enabled)"] -- "OTLP 4317/4318<br/>(stable svc endpoint)" --> ext["ExternalName redirect<br/>cozy-monitoring → tenant-root"]
157+
ext -- "Stage A: direct" --> vt["VictoriaTraces<br/>VTCluster (vtinsert→vtstorage)"]
158+
ext -. "Stage B: via gateway" .-> col["OpenTelemetry Collector<br/>(sampling / rate-limit / tenant)"]
159+
col -- OTLP --> vt
154160
gr["Grafana"] -- Jaeger query --> vt
155161
gr -. trace_id .-> vl["VictoriaLogs"]
156162
gr -. span metrics .-> vm["VictoriaMetrics"]
@@ -169,18 +175,18 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c
169175

170176
## Security
171177

172-
- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. The collector is the trust boundary — it enforces per-tenant `tenant` resource attribution, rate-limits, and sampling to prevent a noisy or hostile tenant from exhausting the shared backend.
178+
- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant `tenant` resource attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend.
173179
- **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant.
174180
- **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs.
175181
- **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any.
176182

177183
## Failure and edge cases
178184

179185
- Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole.
180-
- Collector unreachable from an app → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected.
181-
- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the collector, not the app.
186+
- OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected.
187+
- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the receiver (`vtinsert` in Stage A, the collector in Stage B), not the app.
182188
- `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead.
183-
- App emits OTLP but no backend deployed → collector accepts and drops (or the toggle is guarded to require a backend); documented, not surprising.
189+
- App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend.
184190

185191
## Testing
186192

@@ -190,22 +196,26 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c
190196

191197
## Rollout
192198

193-
1. **Backend + ingest**: `tracingStorages`/`VTCluster` and the OpenTelemetry Collector in `packages/system/monitoring` and `packages/extra/monitoring`. No app changes yet.
199+
1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet.
194200
2. **Grafana**: traces datasource + correlation links.
195201
3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app.
196-
4. **Docs**: enablement guide under `docs/observability/`.
202+
4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the ExternalName from `vtinsert` to the collector (transparent to apps). Ships sampling/rate-limiting/tenant-attribution.
203+
5. **Docs**: enablement guide under `docs/observability/`.
197204

198205
## Open questions
199206

200207
- **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change.
201208
- **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters?
202-
- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default?
203-
- **Collector deployment**: Deployment (gateway) vs DaemonSet (agent) — gateway matches the central-backend model; DaemonSet matches fluent-bit.
209+
- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? (Only relevant once Stage B lands.)
204210
- **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path?
211+
- **Stage-B trigger**: what concrete signal (backend load, abuse, a tenant-attribution requirement) promotes the collector gateway from optional to default?
212+
213+
Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only).
205214

206215
## Alternatives considered
207216

208217
- **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap.
209218
- **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have.
210-
- **No collector, app → backend directly** (ingest): simpler, but loses the shared trust boundary, per-tenant attribution, and central sampling/rate-limiting; rejected for the same reasons metrics go through vmagent and logs through fluent-bit rather than writing to storage directly.
219+
- **No collector, app → backend directly** (ingest): adopted as **Stage A**, not rejected — `vtinsert` is OTLP-native, so direct ingest is the fastest correct MVP and an exact mirror of the logs path. Its limitations (no shared trust boundary, per-tenant attribution, or central sampling/rate-limiting) are exactly what **Stage B**'s collector gateway adds later, without changing the app-facing endpoint.
220+
- **Collector as a DaemonSet agent** (ingest topology): rejected — that node-local shape fits fluent-bit tailing log files, but OTLP traces are pushed over the network by the apps themselves, so a centralized gateway Deployment (the `vtinsert`/vmagent role) is the right shape.
211221
- **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise.

0 commit comments

Comments
 (0)