You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bd75a12
Browse filesBrowse the repository at this point in the historyBrowse files
Refine tracing design: staged A→B ingest grounded in current architecture
Direct-to-vtinsert MVP (mirrors logs) then an opt-in OpenTelemetry Collector
gateway; ExternalName redirect via cozystack-basics; poller readiness gate on
VTCluster; clarify WorkloadMonitor is operational-only so tracing opt-in lives
in app values.
Signed-off-by: Alexey Artamonov <[email protected]>
Copy file name to clipboardExpand all lines: design-proposals/distributed-tracing/README.md
+26-16Lines changed: 26 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,7 +29,7 @@ Metrics storage is declared as a list of tiers in values and rendered into Victo
29
29
30
30
Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`.
31
31
32
-
Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy.
32
+
Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status for the dashboard and billing surfaces — it does not emit scrape configs, and tracing opt-in therefore belongs in each app's `values.yaml`, not in `WorkloadMonitor`.
33
33
34
34
Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim.
35
35
@@ -103,17 +103,21 @@ spec:
103
103
{{- end }}
104
104
```
105
105
106
-
The monitoring HelmRelease readiness gate keys on the new `VTCluster` the same way it keys on `VLCluster` today.
106
+
The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as it does for `VLCluster` today: `waitStrategy: poller`plus a `healthCheckExprs` entry that waits for the CR's `status.updateStatus == 'operational'` (see `packages/extra/monitoring/templates/helmrelease.yaml`). Without the poller gate the release flips Ready as soon as Helm applies the CR — the exact silent-black-hole failure mode that motivated `cozystack/cozystack#3181` for logs.
107
107
108
-
### 2. OTLP ingest via an OpenTelemetry Collector
108
+
### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway)
109
109
110
-
VictoriaTraces ingests OTLP natively, but a collector in front gives the platform a stable tenant-facing endpoint, sampling/rate-limiting, and a place to attach the `tenant:` resource attribute — the same role vmagent and fluent-bit play for metrics and logs. Deploy an **OpenTelemetry Collector** in `cozy-monitoring`, following the `packages/system/monitoring-agents` deployment/service pattern:
110
+
`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert-generic`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, with **no app-visible endpoint change between stages**.
111
+
112
+
**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery.
113
+
114
+
**Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in.
111
115
112
116
- Receivers: `otlp`on gRPC `4317` and HTTP `4318`.
113
117
- Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`.
114
118
- Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend.
115
119
116
-
Tenants target an in-cluster OTLP endpoint (e.g. `otel-collector.cozy-monitoring.svc`); per-tenant namespaces get an ExternalName redirect to the root collector, mirroring the existing `vlinsert-generic` logs redirect, so the central-vs-isolated choice works identically for traces.
120
+
Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317`); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role.
117
121
118
122
### 3. Grafana datasource and correlation
119
123
@@ -149,8 +153,10 @@ How an app emits spans depends on the engine:
@@ -169,18 +175,18 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c
169
175
170
176
## Security
171
177
172
-
- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. The collector is the trust boundary — it enforces per-tenant `tenant` resource attribution, rate-limits, and sampling to prevent a noisy or hostile tenant from exhausting the shared backend.
178
+
- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant `tenant` resource attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend.
173
179
- **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant.
174
180
- **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs.
175
181
- **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any.
176
182
177
183
## Failure and edge cases
178
184
179
185
- Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole.
180
-
- Collector unreachable from an app → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected.
181
-
- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the collector, not the app.
186
+
- OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected.
187
+
- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the receiver (`vtinsert` in Stage A, the collector in Stage B), not the app.
182
188
- `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead.
183
-
- App emits OTLP but no backend deployed → collector accepts and drops (or the toggle is guarded to require a backend); documented, not surprising.
189
+
- App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend.
184
190
185
191
## Testing
186
192
@@ -190,22 +196,26 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c
190
196
191
197
## Rollout
192
198
193
-
1. **Backend + ingest**: `tracingStorages`/`VTCluster` and the OpenTelemetry Collector in `packages/system/monitoring` and `packages/extra/monitoring`. No app changes yet.
199
+
1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet.
3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app.
196
-
4. **Docs**: enablement guide under `docs/observability/`.
202
+
4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the ExternalName from `vtinsert` to the collector (transparent to apps). Ships sampling/rate-limiting/tenant-attribution.
203
+
5. **Docs**: enablement guide under `docs/observability/`.
197
204
198
205
## Open questions
199
206
200
207
- **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change.
201
208
- **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters?
202
-
- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default?
203
-
- **Collector deployment**: Deployment (gateway) vs DaemonSet (agent) — gateway matches the central-backend model; DaemonSet matches fluent-bit.
209
+
- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? (Only relevant once Stage B lands.)
204
210
- **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path?
211
+
- **Stage-B trigger**: what concrete signal (backend load, abuse, a tenant-attribution requirement) promotes the collector gateway from optional to default?
212
+
213
+
Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only).
205
214
206
215
## Alternatives considered
207
216
208
217
- **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap.
209
218
- **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have.
210
-
- **No collector, app → backend directly** (ingest): simpler, but loses the shared trust boundary, per-tenant attribution, and central sampling/rate-limiting; rejected for the same reasons metrics go through vmagent and logs through fluent-bit rather than writing to storage directly.
219
+
- **No collector, app → backend directly** (ingest): adopted as **Stage A**, not rejected — `vtinsert` is OTLP-native, so direct ingest is the fastest correct MVP and an exact mirror of the logs path. Its limitations (no shared trust boundary, per-tenant attribution, or central sampling/rate-limiting) are exactly what **Stage B**'s collector gateway adds later, without changing the app-facing endpoint.
220
+
- **Collector as a DaemonSet agent** (ingest topology): rejected — that node-local shape fits fluent-bit tailing log files, but OTLP traces are pushed over the network by the apps themselves, so a centralized gateway Deployment (the `vtinsert`/vmagent role) is the right shape.
211
221
- **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise.
0 commit comments