Skip to content

Commit cffe509

Browse files
scooby87scooby87
authored
feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does Phase 2 of distributed tracing (design cozystack/community#38, epic #3761), built on the phase-1 backend (#3762, merged): a per-tenant **OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It lives in the tenant namespace (like a per-tenant vmagent), so app→collector traffic is intra-namespace and never crosses the tenant boundary. No OTel operator exists in the platform, so this is a plain Deployment + ConfigMap + Service (the fluent-bit pattern). ## Changes - **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`, HTTP `:4318`) → `memory_limiter` + head sampling (`probabilistic_sampler`, default 10%) + optional `attributes/redact` (PII/secret control point) + `batch` → `otlphttp` exporter to the `tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428` single, endpoints per upstream docs). PSS-restricted securityContext, digest-pinned image via `cozy-lib.image`, config-checksum roll. - **app-facing Service `otel-traces`** (4317/4318). - **`tracingCollector` values** — `enabled` / `replicas` / `samplingPercentage` (schema-bounded 0-100) / `redactAttributes` / `resources`. A **fail-guard** rejects an enabled collector fronting more than one `tracingStorages` backend (it exports to a single backend by design). - **`otel-traces` WorkloadMonitor** and dashboard resource-map entry, rendered under the collector's own gate. The same resource-map Role also gets the pre-existing `alermanager` typo corrected to `alertmanager`, so the alertmanager WorkloadMonitor is readable from the dashboard. - helm-unittest coverage for the collector and its WorkloadMonitor; regenerated `values.schema.json` / `README.md` / `monitoring-rd` ApplicationDefinition. ## Verification - `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all pass. - `helm template` renders cleanly; `make generate` idempotent. - Live check on a test cluster (single-mode backend): the collector starts with `redactAttributes` containing a numeric, a colon-bearing and an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted and is queryable from the backend through the Jaeger API, with the redacted attributes removed and the others kept. - Local `/branch-review` — **LGTM** (one blocker + four recommendations fixed, each with a test: multi-backend guard, memory_limiter cgroup limit, configurable replicas, sampling bounds, trust-boundary wording). ## Scope / deferred (epic #3761) `AccountID`/`ProjectID` header injection for the shared-central topology (phase 4) and engine integrations (phase 3) are follow-up PRs. ### Screenshots N/A, no UI change. ### Downstream repositories The diff touches only `packages/extra/monitoring`, `packages/system/monitoring` and `packages/system/monitoring-rd`. The website's monitoring reference page is regenerated from the package README by the release bot, and the Terraform provider models only the tenant's `monitoring` flag, not these values, so no follow-up is needed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable trace storage in cluster or single-instance modes, with retention settings and Grafana datasources. * Added an enabled-by-default OpenTelemetry collector with configurable replicas, sampling, attribute redaction, and resource limits. * Added workload monitoring and automatic resource recommendations for configured trace components. * Added readiness health checks for both tracing backend modes. * **Bug Fixes** * Corrected dashboard permissions for alert monitoring and trace collector resources. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 368d00b + 9d6fea7 commit cffe509

11 files changed

Lines changed: 801 additions & 10 deletions

File tree

‎packages/extra/monitoring/README.md‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@
5757

5858
| Name | Description | Type | Value |
5959
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------ |
60-
| `tracingStorages` | Configuration of traces storage instances. Empty by default (tracing off): nothing writes spans until the trace collector ships, so no backend is provisioned. Add an entry to enable a VictoriaTraces backend. | `[]object` | `[]` |
60+
| `tracingStorages` | Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. | `[]object` | `[]` |
6161
| `tracingStorages[i].name` | Name of the storage instance: a lowercase RFC 1123 label of at most 42 characters. Treat as immutable once a backend exists: the VTCluster/VTSingle, its Services, PVCs and Grafana datasource are all named after it, so renaming an entry deploys a new, empty backend and removes the old one, the same way changing mode does (single mode loses the old PVC, cluster mode leaves the old vtstorage PVCs Bound until the whole Monitoring release is deleted). | `string` | `""` |
6262
| `tracingStorages[i].mode` | Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters. Treat as immutable once a backend exists: mode selects which Kind is rendered under the same name, so changing it on a running backend re-identifies the object rather than reconfiguring it. Switching from "single" to "cluster" prunes the VTSingle and the operator garbage-collects its PVC (stored traces are lost); switching from "cluster" to "single" (or removing the entry) leaves the vtstorage PVCs Bound and billed until the whole Monitoring release is deleted. The API cannot guard this (transition rules are not available on an array-item field), so this note is the only guard: migrate deliberately, accepting the loss. | `string` | `cluster` |
6363
| `tracingStorages[i].retentionPeriod` | Retention period for traces. | `string` | `14d` |
@@ -66,6 +66,24 @@
6666
| `tracingStorages[i].retentionDiskUsageBytes` | Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. | `string` | `""` |
6767

6868

69+
### Traces collector configuration
70+
71+
| Name | Description | Type | Value |
72+
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------- |
73+
| `tracingCollector` | OpenTelemetry Collector for trace ingestion. | `object` | `{}` |
74+
| `tracingCollector.enabled` | Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction). The collector fronts a single backend: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled. | `bool` | `true` |
75+
| `tracingCollector.replicas` | Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. | `int` | `1` |
76+
| `tracingCollector.samplingPercentage` | Head-sampling percentage applied at the collector before export. | `int` | `10` |
77+
| `tracingCollector.redactAttributes` | Span attribute keys to drop before export (the PII/secret control point, e.g. db.statement). | `[]string` | `[]` |
78+
| `tracingCollector.resources` | Resource configuration for the collector. | `object` | `{}` |
79+
| `tracingCollector.resources.requests` | Resource requests. | `object` | `{}` |
80+
| `tracingCollector.resources.requests.cpu` | CPU request. | `quantity` | `100m` |
81+
| `tracingCollector.resources.requests.memory` | Memory request. | `quantity` | `256Mi` |
82+
| `tracingCollector.resources.limits` | Resource limits. | `object` | `{}` |
83+
| `tracingCollector.resources.limits.cpu` | CPU limit. | `quantity` | `1` |
84+
| `tracingCollector.resources.limits.memory` | Memory limit. | `quantity` | `1Gi` |
85+
86+
6987
### Alerta configuration
7088

7189
| Name | Description | Type | Value |

‎packages/extra/monitoring/templates/dashboard-resourcemap.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ rules:
3535
- grafana
3636
- grafana-db
3737
- alerta-db
38-
- alermanager
38+
- alertmanager
3939
{{- range .Values.metricsStorages }}
4040
- {{ .name }}-vmstorage
4141
- {{ .name }}-vmselect
@@ -50,6 +50,9 @@ rules:
5050
- vmalert-{{ .name }}
5151
{{- break }}
5252
{{- end }}
53+
{{- if and .Values.tracingCollector.enabled (not (empty .Values.tracingStorages)) }}
54+
- otel-traces
55+
{{- end }}
5356
verbs: ["get", "list", "watch"]
5457
---
5558
kind: RoleBinding

‎packages/extra/monitoring/templates/workloadmonitors.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,3 +178,22 @@ spec:
178178
app.kubernetes.io/instance: alertmanager
179179
app.kubernetes.io/name: vmalertmanager
180180
version: {{ $.Chart.Version }}
181+
{{- /* The OTLP trace collector renders under the same gate as the collector
182+
itself (tracingCollector.enabled AND a configured tracingStorages backend),
183+
so tenant workload accounting covers it like every other monitoring
184+
component. expectedReplicas tracks the configured replica count. */}}
185+
{{- if and .Values.tracingCollector.enabled (not (empty .Values.tracingStorages)) }}
186+
---
187+
apiVersion: cozystack.io/v1alpha1
188+
kind: WorkloadMonitor
189+
metadata:
190+
name: otel-traces
191+
spec:
192+
replicas: {{ .Values.tracingCollector.replicas }}
193+
minReplicas: 1
194+
kind: monitoring
195+
type: otel-collector
196+
selector:
197+
app.kubernetes.io/name: otel-traces
198+
version: {{ $.Chart.Version }}
199+
{{- end }}

‎packages/extra/monitoring/tests/workloadmonitors_traces_test.yaml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
suite: OTLP trace collector is accounted for in tenant workload monitoring
2+
# Every workload the monitoring chart ships is enumerated in workloadmonitors.yaml
3+
# (so tenant workload health is tracked) and whitelisted in the dashboard-resources
4+
# Role (so the tenant dashboard may read its WorkloadMonitor). The OTLP collector
5+
# must be covered the same way — under the same gate it renders on
6+
# (tracingCollector.enabled AND a configured tracingStorages backend) and not when
7+
# tracing is off.
8+
templates:
9+
- templates/workloadmonitors.yaml
10+
- templates/dashboard-resourcemap.yaml
11+
12+
release:
13+
name: monitoring
14+
namespace: tenant-test
15+
16+
tests:
17+
- it: renders a WorkloadMonitor for the collector when enabled with a backend
18+
template: templates/workloadmonitors.yaml
19+
set:
20+
tracingCollector:
21+
enabled: true
22+
replicas: 2
23+
samplingPercentage: 10
24+
tracingStorages:
25+
- name: generic
26+
mode: cluster
27+
retentionPeriod: "14d"
28+
storage: 10Gi
29+
storageClassName: replicated
30+
documentSelector:
31+
path: metadata.name
32+
value: otel-traces
33+
asserts:
34+
- equal:
35+
path: kind
36+
value: WorkloadMonitor
37+
- equal:
38+
path: spec.type
39+
value: otel-collector
40+
# expectedReplicas tracks the configured replica count, minReplicas 1.
41+
- equal:
42+
path: spec.replicas
43+
value: 2
44+
- equal:
45+
path: spec.minReplicas
46+
value: 1
47+
- equal:
48+
path: spec.selector["app.kubernetes.io/name"]
49+
value: otel-traces
50+
51+
- it: whitelists the collector WorkloadMonitor in the dashboard-resources Role
52+
template: templates/dashboard-resourcemap.yaml
53+
set:
54+
tracingCollector:
55+
enabled: true
56+
replicas: 1
57+
samplingPercentage: 10
58+
tracingStorages:
59+
- name: generic
60+
mode: cluster
61+
retentionPeriod: "14d"
62+
storage: 10Gi
63+
storageClassName: replicated
64+
documentSelector:
65+
path: kind
66+
value: Role
67+
asserts:
68+
- contains:
69+
path: rules[3].resourceNames
70+
content: otel-traces
71+
# The Role must whitelist each WorkloadMonitor by its real name. Guard the
72+
# alertmanager entry, which shipped misspelled ("alermanager") so the
73+
# alertmanager WorkloadMonitor was silently not RBAC-covered for the
74+
# dashboard — a defect in the exact resourceNames list this PR edits.
75+
- contains:
76+
path: rules[3].resourceNames
77+
content: alertmanager
78+
- notContains:
79+
path: rules[3].resourceNames
80+
content: alermanager
81+
82+
# Absence is asserted on the resourcemap Role (a single deterministic document)
83+
# via notContains rather than on workloadmonitors.yaml: that template emits a
84+
# leading `---`, which makes containsDocument+not a no-op (an empty lead doc).
85+
# Both templates gate the collector on the identical condition, so the Role's
86+
# resourceNames list is a faithful proxy for whether the WorkloadMonitor renders.
87+
- it: does not whitelist the collector when tracing is off (empty tracingStorages)
88+
template: templates/dashboard-resourcemap.yaml
89+
documentSelector:
90+
path: kind
91+
value: Role
92+
asserts:
93+
- notContains:
94+
path: rules[3].resourceNames
95+
content: otel-traces
96+
97+
- it: does not whitelist the collector when the collector is disabled
98+
template: templates/dashboard-resourcemap.yaml
99+
set:
100+
tracingCollector:
101+
enabled: false
102+
replicas: 1
103+
samplingPercentage: 10
104+
tracingStorages:
105+
- name: generic
106+
mode: cluster
107+
retentionPeriod: "14d"
108+
storage: 10Gi
109+
storageClassName: replicated
110+
documentSelector:
111+
path: kind
112+
value: Role
113+
asserts:
114+
- notContains:
115+
path: rules[3].resourceNames
116+
content: otel-traces

0 commit comments

Comments
 (0)