Skip to content

Commit c2fce3d

Browse files
scooby87scooby87
committed
feat(monitoring): add VictoriaTraces backend + Grafana datasource for tracing
Phase 1 of the distributed-tracing design proposal (cozystack/community#38): a per-tenant traces backend in the monitoring stack, mirroring the logs (VLCluster) implementation. No new operator: VTCluster/VTSingle already ship in victoria-metrics-operator. - tracingStorages values (extra + system), per-entry mode cluster|single - templates/vtraces/vtraces.yaml: VTCluster (default) / VTSingle (edge/dev), #3181-style empty-list render guard, retentionPeriod + optional disk cap (retentionMaxDiskSpaceUsageBytes), release-scoped PVC cleanup label - templates/vtraces/grafana-datasource.yaml: Jaeger-typed traces datasource - extra/monitoring readiness gate: VTCluster/VTSingle healthCheckExprs on status.updateStatus, alongside VLCluster - helm-unittest suites (required/empty-guard, cluster-vs-single, datasource, PVC label) + regenerated values.schema.json / README / ApplicationDefinition The per-tenant OTLP Collector, engine integrations, shared-central topology and docs are out of scope and tracked in the tracing epic. Signed-off-by: Alexey Artamonov <[email protected]> Assisted-by: LLM
1 parent 5b91c6b commit c2fce3d

10 files changed

Lines changed: 337 additions & 2 deletions

File tree

‎packages/extra/monitoring/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,19 @@
5353
| `logsStorages[i].storageClassName` | StorageClass used to store the data. | `string` | `replicated` |
5454

5555

56+
### Traces storage configuration
57+
58+
| Name | Description | Type | Value |
59+
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------- | ------------ |
60+
| `tracingStorages` | Configuration of traces storage instances. | `[]object` | `[...]` |
61+
| `tracingStorages[i].name` | Name of the storage instance. | `string` | `""` |
62+
| `tracingStorages[i].mode` | Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters. | `string` | `cluster` |
63+
| `tracingStorages[i].retentionPeriod` | Retention period for traces. | `string` | `14d` |
64+
| `tracingStorages[i].storage` | Persistent volume size. | `string` | `10Gi` |
65+
| `tracingStorages[i].storageClassName` | StorageClass used to store the data. | `string` | `replicated` |
66+
| `tracingStorages[i].retentionDiskUsageBytes` | Optional disk-usage cap as a byte quantity (e.g. "8GB"); bounds disk in addition to the age-based retentionPeriod. | `string` | `""` |
67+
68+
5669
### Alerta configuration
5770

5871
| Name | Description | Type | Value |

‎packages/extra/monitoring/templates/helmrelease.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,12 +48,27 @@ spec:
4848
# nothing to match. That configuration is rejected at render time by the
4949
# system/monitoring chart (templates/vlogs/vlogs.yaml fails on an empty
5050
# logsStorages), so the "Ready with no log store" hole cannot be reached.
51+
#
52+
# The traces backend (VTCluster / VTSingle) uses the same operator and the
53+
# same status.updateStatus contract as VLCluster, so it is gated identically.
54+
# tracingStorages is likewise render-rejected when empty (templates/vtraces/
55+
# vtraces.yaml), so the "Ready with no trace store" hole cannot be reached.
5156
healthCheckExprs:
5257
- apiVersion: operator.victoriametrics.com/v1
5358
kind: VLCluster
5459
inProgress: has(status.updateStatus) && status.updateStatus == 'expanding'
5560
failed: has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'
5661
current: has(status.updateStatus) && status.updateStatus == 'operational'
62+
- apiVersion: operator.victoriametrics.com/v1
63+
kind: VTCluster
64+
inProgress: has(status.updateStatus) && status.updateStatus == 'expanding'
65+
failed: has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'
66+
current: has(status.updateStatus) && status.updateStatus == 'operational'
67+
- apiVersion: operator.victoriametrics.com/v1
68+
kind: VTSingle
69+
inProgress: has(status.updateStatus) && status.updateStatus == 'expanding'
70+
failed: has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'
71+
current: has(status.updateStatus) && status.updateStatus == 'operational'
5772
valuesFrom:
5873
- kind: Secret
5974
name: cozystack-values

‎packages/extra/monitoring/values.schema.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -323,6 +323,59 @@
323323
}
324324
}
325325
},
326+
"tracingStorages": {
327+
"description": "Configuration of traces storage instances.",
328+
"type": "array",
329+
"default": [
330+
{
331+
"mode": "cluster",
332+
"name": "generic",
333+
"retentionPeriod": "14d",
334+
"storage": "10Gi",
335+
"storageClassName": "replicated"
336+
}
337+
],
338+
"items": {
339+
"type": "object",
340+
"required": [
341+
"mode",
342+
"name",
343+
"retentionPeriod",
344+
"storage",
345+
"storageClassName"
346+
],
347+
"properties": {
348+
"mode": {
349+
"description": "Deployment mode: \"cluster\" renders a VTCluster; \"single\" renders a single-binary VTSingle for edge/dev/small clusters.",
350+
"type": "string",
351+
"default": "cluster"
352+
},
353+
"name": {
354+
"description": "Name of the storage instance.",
355+
"type": "string"
356+
},
357+
"retentionDiskUsageBytes": {
358+
"description": "Optional disk-usage cap as a byte quantity (e.g. \"8GB\"); bounds disk in addition to the age-based retentionPeriod.",
359+
"type": "string"
360+
},
361+
"retentionPeriod": {
362+
"description": "Retention period for traces.",
363+
"type": "string",
364+
"default": "14d"
365+
},
366+
"storage": {
367+
"description": "Persistent volume size.",
368+
"type": "string",
369+
"default": "10Gi"
370+
},
371+
"storageClassName": {
372+
"description": "StorageClass used to store the data.",
373+
"type": "string",
374+
"default": "replicated"
375+
}
376+
}
377+
}
378+
},
326379
"alerta": {
327380
"description": "Configuration for the Alerta service.",
328381
"type": "object",

‎packages/extra/monitoring/values.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,26 @@ logsStorages:
9292
storage: 10Gi
9393
storageClassName: replicated
9494

95+
##
96+
## @section Traces storage configuration
97+
##
98+
99+
## @typedef {struct} TracingStorage - Configuration of traces storage instance.
100+
## @field {string} name - Name of the storage instance.
101+
## @field {string} mode="cluster" - Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters.
102+
## @field {string} retentionPeriod="14d" - Retention period for traces.
103+
## @field {string} storage="10Gi" - Persistent volume size.
104+
## @field {string} storageClassName="replicated" - StorageClass used to store the data.
105+
## @field {string} [retentionDiskUsageBytes] - Optional disk-usage cap as a byte quantity (e.g. "8GB"); bounds disk in addition to the age-based retentionPeriod.
106+
107+
## @param {[]TracingStorage} tracingStorages - Configuration of traces storage instances.
108+
tracingStorages:
109+
- name: generic
110+
mode: cluster
111+
retentionPeriod: "14d"
112+
storage: 10Gi
113+
storageClassName: replicated
114+
95115
##
96116
## @section Alerta configuration
97117
##

‎packages/system/monitoring-rd/cozyrds/monitoring.yaml‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/system/monitoring/templates/vtraces/grafana-datasource.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{{- range .Values.tracingStorages }}
2+
---
3+
apiVersion: grafana.integreatly.org/v1beta1
4+
kind: GrafanaDatasource
5+
metadata:
6+
name: vtraces-{{ .name }}
7+
spec:
8+
datasource:
9+
access: proxy
10+
# VictoriaTraces exposes a Jaeger-compatible query API, and `jaeger` is a
11+
# built-in Grafana datasource (no plugin install needed, unlike the logs
12+
# datasource). The exact select service name, port and query path below are
13+
# confirm-before-implementation against the pinned VictoriaTraces version
14+
# (see design-proposals/distributed-tracing in cozystack/community#38); the
15+
# upstream query API is also converging on a Tempo-compatible surface, so
16+
# `type` may switch to `tempo` once confirmed on a live cluster.
17+
type: jaeger
18+
name: vtraces-{{ .name }}
19+
{{- if eq (.mode | default "cluster") "single" }}
20+
url: http://vtsingle-{{ .name }}.{{ $.Release.Namespace }}.svc:10428/select/jaeger
21+
{{- else }}
22+
url: http://vtselect-{{ .name }}.{{ $.Release.Namespace }}.svc:10471/select/jaeger
23+
{{- end }}
24+
instanceSelector:
25+
matchLabels:
26+
dashboards: grafana
27+
{{- end }}

‎packages/system/monitoring/templates/vtraces/vtraces.yaml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
{{- /*
2+
Reject an empty tracingStorages outright, mirroring the logsStorages guard
3+
(issue #3181). With no entry no VTCluster/VTSingle is rendered, the
4+
monitoring-system HelmRelease readiness gate (which keys on the traces CR) has
5+
nothing to match, and OTLP producers ship spans to a backend that never
6+
exists — silently dropping traces. Fail the render instead so the
7+
misconfiguration is loud, not silent.
8+
*/ -}}
9+
{{- if not .Values.tracingStorages }}
10+
{{- fail "monitoring: tracingStorages must define at least one entry; an empty list renders no VictoriaTraces backend and OTLP producers silently drop spans" }}
11+
{{- end }}
12+
{{- range .Values.tracingStorages }}
13+
---
14+
{{- if eq (.mode | default "cluster") "single" }}
15+
apiVersion: operator.victoriametrics.com/v1
16+
kind: VTSingle
17+
metadata:
18+
name: {{ .name }}
19+
spec:
20+
managedMetadata:
21+
labels:
22+
apps.cozystack.io/application.group: apps.cozystack.io
23+
apps.cozystack.io/application.kind: Monitoring
24+
apps.cozystack.io/application.name: {{ $.Release.Name }}
25+
retentionPeriod: {{ .retentionPeriod | quote }}
26+
{{- with .retentionDiskUsageBytes }}
27+
retentionMaxDiskSpaceUsageBytes: {{ . | quote }}
28+
{{- end }}
29+
# Stamped on the storage PVC so the monitoring post-delete cleanup hook can
30+
# select it by this label (same release-scoped cleanup as logs/metrics).
31+
storageMetadata:
32+
labels:
33+
apps.cozystack.io/application.name: {{ $.Release.Name }}
34+
storage:
35+
{{- with .storageClassName }}
36+
storageClassName: {{ . }}
37+
{{- end }}
38+
accessModes:
39+
- ReadWriteOnce
40+
resources:
41+
requests:
42+
storage: {{ .storage }}
43+
{{- else }}
44+
apiVersion: operator.victoriametrics.com/v1
45+
kind: VTCluster
46+
metadata:
47+
name: {{ .name }}
48+
spec:
49+
managedMetadata:
50+
labels:
51+
apps.cozystack.io/application.group: apps.cozystack.io
52+
apps.cozystack.io/application.kind: Monitoring
53+
apps.cozystack.io/application.name: {{ $.Release.Name }}
54+
insert:
55+
replicaCount: 2
56+
resources: {}
57+
select:
58+
replicaCount: 2
59+
resources: {}
60+
storage:
61+
retentionPeriod: {{ .retentionPeriod | quote }}
62+
{{- with .retentionDiskUsageBytes }}
63+
retentionMaxDiskSpaceUsageBytes: {{ . | quote }}
64+
{{- end }}
65+
replicaCount: 2
66+
resources: {}
67+
storage:
68+
volumeClaimTemplate:
69+
metadata:
70+
# Stamped on the claim template so vm-operator copies it onto the
71+
# storage PVCs; managedMetadata is not applied to PVCs. The monitoring
72+
# post-delete cleanup hook selects PVCs by this label to stay
73+
# release-scoped (mirrors vlogs/vmcluster).
74+
labels:
75+
apps.cozystack.io/application.name: {{ $.Release.Name }}
76+
spec:
77+
{{- with .storageClassName }}
78+
storageClassName: {{ . }}
79+
{{- end }}
80+
resources:
81+
requests:
82+
storage: {{ .storage }}
83+
{{- end }}
84+
{{- end }}

‎packages/system/monitoring/tests/storage_pvc_label_test.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,14 @@ tests:
4747
- equal:
4848
path: spec.vlstorage.storage.volumeClaimTemplate.metadata.labels["apps.cozystack.io/application.name"]
4949
value: monitoring-system
50+
51+
- it: stamps the cleanup label on the VTCluster storage claim template
52+
templates:
53+
- templates/vtraces/vtraces.yaml
54+
documentSelector:
55+
path: metadata.name
56+
value: generic
57+
asserts:
58+
- equal:
59+
path: spec.storage.storage.volumeClaimTemplate.metadata.labels["apps.cozystack.io/application.name"]
60+
value: monitoring-system

‎packages/system/monitoring/tests/vtraces_tracingstorages_required_test.yaml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
suite: VictoriaTraces rendering requires a non-empty tracingStorages
2+
templates:
3+
- templates/vtraces/vtraces.yaml
4+
- templates/vtraces/grafana-datasource.yaml
5+
6+
# tracingStorages drives the VTCluster/VTSingle backend(s) for distributed
7+
# tracing. An empty list renders no backend, so the monitoring-system
8+
# HelmRelease readiness gate has nothing to match and OTLP producers silently
9+
# drop spans to a backend that never exists (same failure class as logs
10+
# issue #3181). The template rejects that outright rather than documenting it,
11+
# so the misconfiguration is loud. These cases pin the valid defaults, the
12+
# cluster/single split, the datasource, and the rejected empty configuration.
13+
14+
release:
15+
name: monitoring
16+
namespace: tenant-root
17+
18+
tests:
19+
- it: renders a VTCluster for the default (cluster-mode) tracingStorages
20+
templates:
21+
- templates/vtraces/vtraces.yaml
22+
asserts:
23+
- containsDocument:
24+
apiVersion: operator.victoriametrics.com/v1
25+
kind: VTCluster
26+
name: generic
27+
28+
- it: renders a VTSingle when mode is single
29+
templates:
30+
- templates/vtraces/vtraces.yaml
31+
set:
32+
tracingStorages:
33+
- name: generic
34+
mode: single
35+
retentionPeriod: "14d"
36+
storage: 10Gi
37+
asserts:
38+
- containsDocument:
39+
apiVersion: operator.victoriametrics.com/v1
40+
kind: VTSingle
41+
name: generic
42+
43+
- it: renders one traces backend per tracingStorages entry
44+
templates:
45+
- templates/vtraces/vtraces.yaml
46+
set:
47+
tracingStorages:
48+
- name: generic
49+
mode: cluster
50+
retentionPeriod: "14d"
51+
storage: 10Gi
52+
- name: audit
53+
mode: single
54+
retentionPeriod: "7d"
55+
storage: 10Gi
56+
asserts:
57+
- hasDocuments:
58+
count: 2
59+
60+
- it: sets the retention period on the VTCluster storage component
61+
templates:
62+
- templates/vtraces/vtraces.yaml
63+
documentSelector:
64+
path: metadata.name
65+
value: generic
66+
asserts:
67+
- equal:
68+
path: spec.storage.retentionPeriod
69+
value: "14d"
70+
71+
- it: renders an optional disk-usage cap only when set
72+
templates:
73+
- templates/vtraces/vtraces.yaml
74+
set:
75+
tracingStorages:
76+
- name: generic
77+
mode: cluster
78+
retentionPeriod: "14d"
79+
storage: 10Gi
80+
retentionDiskUsageBytes: "8GB"
81+
asserts:
82+
- equal:
83+
path: spec.storage.retentionMaxDiskSpaceUsageBytes
84+
value: "8GB"
85+
86+
- it: renders a Jaeger-typed Grafana datasource per tracingStorages entry
87+
templates:
88+
- templates/vtraces/grafana-datasource.yaml
89+
asserts:
90+
- containsDocument:
91+
apiVersion: grafana.integreatly.org/v1beta1
92+
kind: GrafanaDatasource
93+
name: vtraces-generic
94+
- equal:
95+
path: spec.datasource.type
96+
value: jaeger
97+
98+
- it: fails the render when tracingStorages is empty
99+
templates:
100+
- templates/vtraces/vtraces.yaml
101+
set:
102+
tracingStorages: []
103+
asserts:
104+
- failedTemplate:
105+
errorMessage: "monitoring: tracingStorages must define at least one entry; an empty list renders no VictoriaTraces backend and OTLP producers silently drop spans"

‎packages/system/monitoring/values.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,13 @@ logsStorages:
5050
retentionPeriod: "1"
5151
storage: 10Gi
5252
storageClassName: ""
53+
54+
tracingStorages:
55+
- name: generic
56+
mode: cluster
57+
retentionPeriod: "14d"
58+
storage: 10Gi
59+
storageClassName: ""
5360
alerta:
5461
storage: 10Gi
5562
storageClassName: ""

0 commit comments

Comments
 (0)