Skip to content

Commit 7e2c035

Browse files
authored
feat(monitoring): migrate VictoriaLogs from VLogs to VLCluster (#2153)
## What this PR does Migrates VictoriaLogs from the deprecated single-node `VLogs` CR to `VLCluster` (cluster mode) with vlinsert/vlselect/vlstorage components for reliability and horizontal scalability. **Operator upgrade:** - Upgrades victoria-metrics-operator from v0.55.0 to v0.68.1 to add VLCluster CRD support **VLCluster deployment:** - Replaces `VLogs` (v1beta1) with `VLCluster` (v1) — 2 replicas per component, consistent with VMCluster - Adds VPA for all VLCluster components (vlinsert, vlselect, vlstorage) - Updates WorkloadMonitors for the three-component architecture **Endpoint updates:** - Fluent-bit outputs: `vlogs-generic:9428` → `vlinsert-generic:9481` - Grafana datasource: `vlogs-{name}:9428` → `vlselect-{name}:9471` - ExternalName service: `vlogs-generic` → `vlinsert-generic` **Migration (35 → 36):** - Adds `helm.sh/resource-policy: keep` annotation to existing VLogs resources so they are preserved during upgrade - Users need to verify the new VLCluster is working, then optionally migrate historical data and manually delete old VLogs resources ### Release note ```release-note [monitoring] Migrate VictoriaLogs from single-node VLogs to VLCluster (cluster mode). Old VLogs resources are preserved with `helm.sh/resource-policy: keep` annotation. After upgrade, verify the new cluster is working, then optionally migrate historical data and delete old VLogs resources manually. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * CRD upgrade workflow with configurable hooks, Job, and ServiceAccount support * Monitoring storage split into vlinsert / vlselect / vlstorage with corresponding VPAs * Service traffic distribution option and optional shareProcessNamespace toggle * **Updates** * VictoriaMetrics Operator bumped to v0.68.1 * Fluent Bit and Grafana endpoints/ports updated to new monitoring targets * Global extra-labels support for resources * Migration target advanced to version 36 <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents bf31b7c + 9e16630 commit 7e2c035

44 files changed

Lines changed: 36342 additions & 58227 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎hack/e2e-install-cozystack.bats‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -175,8 +175,8 @@ EOF
175175

176176
# VictoriaMetrics components
177177
kubectl wait vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m
178-
kubectl wait vlogs/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=5m
179-
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.clusterStatus}'=operational --timeout=5m
178+
kubectl wait vlclusters/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=5m
179+
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=5m
180180

181181
# Grafana
182182
kubectl wait clusters.postgresql.cnpg.io/grafana-db -n tenant-root --for=condition=ready --timeout=5m

‎packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@ spec:
7373
[OUTPUT]
7474
Name http
7575
Match kube.*
76-
Host vlogs-generic.{{ $targetTenant }}.svc.{{ $clusterDomain }}
77-
port 9428
76+
Host vlinsert-generic.{{ $targetTenant }}.svc.{{ $clusterDomain }}
77+
port 9481
7878
compress gzip
7979
uri /insert/jsonline?_stream_fields=stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
8080
format json_lines
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/sh
2+
# Migration 35 --> 36
3+
# Add helm.sh/resource-policy=keep annotation to existing VLogs resources
4+
# so they are preserved when the monitoring helm release upgrades to VLCluster.
5+
# Users will need to manually verify the new cluster is working, then optionally
6+
# migrate historical data and delete old VLogs resources.
7+
8+
set -euo pipefail
9+
10+
VLOGS=$(kubectl get vlogs.operator.victoriametrics.com --all-namespaces --output jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' 2>/dev/null || true)
11+
for resource in $VLOGS; do
12+
NS="${resource%%/*}"
13+
NAME="${resource##*/}"
14+
echo "Adding keep annotation to VLogs/$NAME in $NS"
15+
kubectl annotate vlogs.operator.victoriametrics.com --namespace "$NS" "$NAME" \
16+
helm.sh/resource-policy=keep \
17+
--overwrite
18+
done
19+
20+
kubectl create configmap --namespace cozy-system cozystack-version \
21+
--from-literal=version=36 --dry-run=client --output yaml | kubectl apply --filename -

‎packages/core/platform/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sourceRef:
66
migrations:
77
enabled: false
88
image: ghcr.io/cozystack/cozystack/platform-migrations:v1.1.0@sha256:d7e8955c1ad8c8fbd4ce42b014c0f849d73d0c3faf0cedaac8e15d647fb2f663
9-
targetVersion: 35
9+
targetVersion: 36
1010
# Bundle deployment configuration
1111
bundles:
1212
system:

‎packages/extra/monitoring/templates/dashboard-resourcemap.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ rules:
4242
- {{ .name }}-vminsert
4343
{{- end }}
4444
{{- range .Values.logsStorages }}
45-
- {{ $.Release.Name }}-vlogs-{{ .name }}
45+
- {{ .name }}-vlstorage
46+
- {{ .name }}-vlselect
47+
- {{ .name }}-vlinsert
4648
{{- end }}
4749
{{- range .Values.metricsStorages }}
4850
- vmalert-{{ .name }}

‎packages/extra/monitoring/templates/workloadmonitors.yaml‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,16 +67,46 @@ spec:
6767
apiVersion: cozystack.io/v1alpha1
6868
kind: WorkloadMonitor
6969
metadata:
70-
name: vlogs-{{ .name }}
70+
name: {{ .name }}-vlstorage
7171
spec:
72-
replicas: 1
72+
replicas: 2
73+
minReplicas: 1
74+
kind: monitoring
75+
type: vlstorage
76+
selector:
77+
app.kubernetes.io/component: monitoring
78+
app.kubernetes.io/instance: {{ .name }}
79+
app.kubernetes.io/name: vlstorage
80+
version: {{ $.Chart.Version }}
81+
---
82+
apiVersion: cozystack.io/v1alpha1
83+
kind: WorkloadMonitor
84+
metadata:
85+
name: {{ .name }}-vlselect
86+
spec:
87+
replicas: 2
88+
minReplicas: 1
89+
kind: monitoring
90+
type: vlselect
91+
selector:
92+
app.kubernetes.io/component: monitoring
93+
app.kubernetes.io/instance: {{ .name }}
94+
app.kubernetes.io/name: vlselect
95+
version: {{ $.Chart.Version }}
96+
---
97+
apiVersion: cozystack.io/v1alpha1
98+
kind: WorkloadMonitor
99+
metadata:
100+
name: {{ .name }}-vlinsert
101+
spec:
102+
replicas: 2
73103
minReplicas: 1
74104
kind: monitoring
75-
type: vlogs
105+
type: vlinsert
76106
selector:
77107
app.kubernetes.io/component: monitoring
78108
app.kubernetes.io/instance: {{ .name }}
79-
app.kubernetes.io/name: vlogs
109+
app.kubernetes.io/name: vlinsert
80110
version: {{ $.Chart.Version }}
81111
{{- end }}
82112
---

‎packages/system/cozystack-basics/templates/monitoring-external-services.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22
apiVersion: v1
33
kind: Service
44
metadata:
5-
name: vlogs-generic
5+
name: vlinsert-generic
66
namespace: cozy-monitoring
77
spec:
88
type: ExternalName
9-
externalName: vlogs-generic.tenant-root.svc.cluster.local
9+
externalName: vlinsert-generic.tenant-root.svc.cluster.local
1010
---
1111
apiVersion: v1
1212
kind: Service

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -344,8 +344,8 @@ fluent-bit:
344344
[OUTPUT]
345345
Name http
346346
Match kube.*
347-
Host vlogs-generic.{{ .Values.global.target }}.svc
348-
port 9428
347+
Host vlinsert-generic.{{ .Values.global.target }}.svc
348+
port 9481
349349
compress gzip
350350
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
351351
format json_lines
@@ -355,8 +355,8 @@ fluent-bit:
355355
[OUTPUT]
356356
Name http
357357
Match events.*
358-
Host vlogs-generic.{{ .Values.global.target }}.svc
359-
port 9428
358+
Host vlinsert-generic.{{ .Values.global.target }}.svc
359+
port 9481
360360
compress gzip
361361
uri /insert/jsonline?_stream_fields=log_source,reason,meatdata_namespace,metadata_name&_msg_field=message&_time_field=date
362362
format json_lines
@@ -366,8 +366,8 @@ fluent-bit:
366366
[OUTPUT]
367367
Name http
368368
Match audit.*
369-
Host vlogs-generic.{{ .Values.global.target }}.svc
370-
port 9428
369+
Host vlinsert-generic.{{ .Values.global.target }}.svc
370+
port 9481
371371
compress gzip
372372
uri /insert/jsonline?_stream_fields=log_source,stage,user_username,verb,requestUri&_msg_field=requestURI&_time_field=date
373373
format json_lines

‎packages/system/monitoring/templates/vlogs/grafana-datasource.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ spec:
88
access: proxy
99
type: victoriametrics-logs-datasource
1010
name: vlogs-{{ .name }}
11-
url: http://vlogs-{{ .name }}.{{ $.Release.Namespace }}.svc:9428
11+
url: http://vlselect-{{ .name }}.{{ $.Release.Namespace }}.svc:9471
1212
instanceSelector:
1313
matchLabels:
1414
dashboards: grafana
Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
{{- range .Values.logsStorages }}
2-
apiVersion: operator.victoriametrics.com/v1beta1
3-
kind: VLogs
2+
---
3+
apiVersion: operator.victoriametrics.com/v1
4+
kind: VLCluster
45
metadata:
56
name: {{ .name }}
67
spec:
@@ -9,14 +10,23 @@ spec:
910
apps.cozystack.io/application.group: apps.cozystack.io
1011
apps.cozystack.io/application.kind: Monitoring
1112
apps.cozystack.io/application.name: {{ $.Release.Name }}
12-
image:
13-
tag: v1.17.0-victorialogs
14-
storage:
15-
resources:
16-
requests:
17-
storage: {{ .storage }}
18-
storageClassName: {{ .storageClassName }}
19-
accessModes: [ReadWriteOnce]
20-
retentionPeriod: "{{ .retentionPeriod }}"
21-
removePvcAfterDelete: true
13+
vlinsert:
14+
replicaCount: 2
15+
resources: {}
16+
vlselect:
17+
replicaCount: 2
18+
resources: {}
19+
vlstorage:
20+
retentionPeriod: {{ .retentionPeriod | quote }}
21+
replicaCount: 2
22+
resources: {}
23+
storage:
24+
volumeClaimTemplate:
25+
spec:
26+
{{- with .storageClassName }}
27+
storageClassName: {{ . }}
28+
{{- end }}
29+
resources:
30+
requests:
31+
storage: {{ .storage }}
2232
{{- end }}

0 commit comments

Comments
 (0)