Skip to content

Commit d301a57

Browse files
scooby87claude
andcommitted
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring. The template reads the live Tenant/root spec.monitoring via lookup (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value monitoring.rootEnabled (via _cluster.monitoring-enabled) when lookup is empty (dry-run / first render). Default preserves the historical always-create behaviour. Covered by helm-unittest across all four charts (readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent fallback contract, and the platform wiring into the cozystack-values Secret). Adds the missing `test:` target to packages/extra/monitoring so its suite runs in CI. Known limitation (follow-up): the legacy platform monitoring release installed directly into cozy-monitoring via PackageSource cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. Signed-off-by: Алексей Артамонов <[email protected]> Assisted-By: Claude <[email protected]> Co-Authored-By: Claude Opus 4.8 <[email protected]>
1 parent f101426 commit d301a57

10 files changed

Lines changed: 285 additions & 0 deletions

File tree

‎packages/core/platform/templates/apps.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,16 @@ stringData:
126126
proxy-protocol: {{ .Values.publishing.proxyProtocol | default false | quote }}
127127
gateway-enabled: {{ .Values.gateway.enabled | default false | quote }}
128128
gateway-attached-namespaces: {{ .Values.gateway.attachedNamespaces | default (list) | join "," | quote }}
129+
{{- /* Whether the root tenant hosts the platform monitoring stack
130+
(Tenant/root spec.monitoring). cozystack-basics gates the
131+
cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName
132+
redirects on this: when the root tenant has no monitoring, those
133+
redirects dead-end and fluent-bit silently drops logs (#3181).
134+
No `default` here: Go's `default` treats a boolean false as empty
135+
and would flip an explicit rootEnabled=false back to true. The
136+
chart values.yaml supplies the true default, so quote the value
137+
as-is. */}}
138+
monitoring-enabled: {{ .Values.monitoring.rootEnabled | quote }}
129139
cluster-domain: {{ .Values.networking.clusterDomain | quote }}
130140
api-server-endpoint: {{ .Values.publishing.apiServerEndpoint | quote }}
131141
{{- with .Values.branding }}
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
suite: apps.yaml monitoring.rootEnabled → _cluster wiring
2+
templates:
3+
- templates/apps.yaml
4+
5+
release:
6+
name: cozy-platform
7+
namespace: cozy-system
8+
9+
# monitoring.rootEnabled records whether the root tenant hosts the platform
10+
# monitoring stack (Tenant/root spec.monitoring). It reaches cozystack-basics
11+
# through _cluster.monitoring-enabled in the cozystack-values Secret, where it
12+
# gates the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName
13+
# redirects. Without the flag in the Secret those redirects would always be
14+
# created and dead-end when the root tenant has no monitoring, silently
15+
# dropping logs (issue #3181). These assertions pin that the flag lands in the
16+
# Secret both ways.
17+
18+
tests:
19+
- it: writes monitoring-enabled true by default
20+
asserts:
21+
- matchRegex:
22+
path: stringData["values.yaml"]
23+
pattern: '(?m)^\s*monitoring-enabled:\s*"true"'
24+
25+
- it: writes monitoring-enabled false when monitoring.rootEnabled is disabled
26+
set:
27+
monitoring.rootEnabled: false
28+
asserts:
29+
- matchRegex:
30+
path: stringData["values.yaml"]
31+
pattern: '(?m)^\s*monitoring-enabled:\s*"false"'

‎packages/core/platform/values.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,6 +363,19 @@ gateway:
363363
- cozy-monitoring
364364
- cozy-linstor-gui
365365
- default
366+
# Platform monitoring configuration.
367+
monitoring:
368+
# Whether the root tenant (tenant-root) hosts the platform
369+
# VictoriaMetrics/VictoriaLogs stack, i.e. Tenant/root spec.monitoring=true.
370+
# The platform centralises metrics and logs in tenant-root, and
371+
# cozystack-basics publishes vlinsert-generic / vminsert-* ExternalName
372+
# Services in cozy-monitoring that redirect there. When the root tenant has
373+
# monitoring disabled those redirects dead-end and fluent-bit silently drops
374+
# logs (issue #3181), so the ExternalNames are only created when this is
375+
# true. Keep this in sync with Tenant/root spec.monitoring — the default
376+
# tracks the standard deployment, which enables monitoring on the root
377+
# tenant.
378+
rootEnabled: true
366379
# Pod scheduling configuration
367380
scheduling:
368381
globalAppTopologySpreadConstraints: ""

‎packages/extra/monitoring/Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ generate:
99
cozyvalues-gen -v values.yaml -s values.schema.json -r README.md
1010
../../../hack/update-crd.sh
1111

12+
test:
13+
helm unittest .
14+
1215
image:
1316
docker buildx build images/grafana \
1417
$(call image-tags,grafana,$(GRAFANA_TAG)) \

‎packages/extra/monitoring/templates/helmrelease.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,34 @@ spec:
1717
upgrade:
1818
remediation:
1919
retries: -1
20+
# `poller` waitStrategy is required for healthCheckExprs to be evaluated.
21+
# Without it the HR flips Ready as soon as helm applies the VLCluster CR,
22+
# before the victoria-metrics-operator has stood up the log store — which is
23+
# exactly how a missing/broken VictoriaLogs stayed invisible (issue #3181:
24+
# fluent-bit drops logs at a dead-end vlinsert-generic while the release is
25+
# Ready). Gating on the VLCluster's own status makes that failure loud.
26+
waitStrategy:
27+
name: poller
28+
# VLCluster has no status.conditions; it exposes status.updateStatus, whose
29+
# operator enum is exactly: operational, expanding, failed, paused. Map
30+
# `operational` -> current and `expanding` -> in-progress. Everything else
31+
# (the operator's `failed`, and `paused`) is treated as failed on purpose: a
32+
# paused or failed log store is not ingesting, so the release must not report
33+
# Ready. Cozystack never sets spec.paused on these VLClusters, so `paused`
34+
# is not reachable in normal operation and cannot cause a spurious failure.
35+
# The HR `timeout:` is the real backstop. Matching by kind covers every
36+
# logsStorages entry.
37+
#
38+
# The gate keys on a VLCluster existing, so an empty logsStorages would leave
39+
# nothing to match. That configuration is rejected at render time by the
40+
# system/monitoring chart (templates/vlogs/vlogs.yaml fails on an empty
41+
# logsStorages), so the "Ready with no log store" hole cannot be reached.
42+
healthCheckExprs:
43+
- apiVersion: operator.victoriametrics.com/v1
44+
kind: VLCluster
45+
inProgress: has(status.updateStatus) && status.updateStatus == 'expanding'
46+
failed: has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'
47+
current: has(status.updateStatus) && status.updateStatus == 'operational'
2048
valuesFrom:
2149
- kind: Secret
2250
name: cozystack-values
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
suite: monitoring-system HelmRelease readiness gate
2+
templates:
3+
- templates/helmrelease.yaml
4+
5+
release:
6+
name: monitoring
7+
namespace: tenant-root
8+
9+
# The monitoring-system HelmRelease owns the VLCluster(s) rendered from
10+
# logsStorages. Without a health gate it flips Ready as soon as helm applies the
11+
# CR, so a missing or broken VictoriaLogs stays invisible while fluent-bit drops
12+
# logs at a dead-end vlinsert-generic (issue #3181). A `poller` waitStrategy plus
13+
# a VLCluster healthCheckExpr makes that failure surface as not-ready.
14+
tests:
15+
- it: names the child release <release>-system
16+
asserts:
17+
- containsDocument:
18+
kind: HelmRelease
19+
apiVersion: helm.toolkit.fluxcd.io/v2
20+
name: monitoring-system
21+
22+
- it: uses the poller waitStrategy so healthCheckExprs are evaluated
23+
asserts:
24+
- equal:
25+
path: spec.waitStrategy.name
26+
value: poller
27+
28+
- it: gates readiness on the VLCluster status.updateStatus
29+
asserts:
30+
- equal:
31+
path: spec.healthCheckExprs[0].apiVersion
32+
value: operator.victoriametrics.com/v1
33+
- equal:
34+
path: spec.healthCheckExprs[0].kind
35+
value: VLCluster
36+
- equal:
37+
path: spec.healthCheckExprs[0].current
38+
value: "has(status.updateStatus) && status.updateStatus == 'operational'"
39+
- equal:
40+
path: spec.healthCheckExprs[0].inProgress
41+
value: "has(status.updateStatus) && status.updateStatus == 'expanding'"
42+
- equal:
43+
path: spec.healthCheckExprs[0].failed
44+
value: "has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'"

‎packages/system/cozystack-basics/templates/monitoring-external-services.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,36 @@
1+
{{- /*
2+
These ExternalName Services live in cozy-monitoring and redirect
3+
vlinsert-generic / vminsert-* to the platform monitoring stack the root
4+
tenant hosts in tenant-root. They only make sense when tenant-root actually
5+
runs that stack (Tenant/root spec.monitoring=true). When it does not, the
6+
redirects resolve to nothing: fluent-bit ships to vlinsert-generic and its
7+
chunks dead-end at a non-existent tenant-root VictoriaLogs, dropping logs
8+
silently (issue #3181).
9+
10+
Whether the root tenant runs monitoring is a runtime choice on Tenant/root
11+
(set out-of-band, not by this chart), so there is no single render-time value
12+
that always reflects it. We resolve it in two steps, real state first:
13+
14+
1. lookup the live Tenant/root and read spec.monitoring — the source of
15+
truth. On a Flux reconcile against the cluster this is authoritative, so
16+
the ExternalNames and the tenant's actual monitoring can never drift.
17+
2. When lookup returns nothing — `helm template`, `--dry-run`, unit tests,
18+
or the first render before Tenant/root exists — fall back to
19+
_cluster.monitoring-enabled (fed from the platform monitoring.rootEnabled
20+
value), defaulting to "true" to preserve the historical always-create
21+
behaviour for cozystack-values Secrets written before that key existed.
22+
23+
Unit tests exercise the fallback branch (lookup is always empty without a
24+
cluster); the lookup branch is validated live.
25+
*/ -}}
26+
{{- $rootTenant := lookup "apps.cozystack.io/v1alpha1" "Tenant" "tenant-root" "root" -}}
27+
{{- $enabled := "" -}}
28+
{{- if $rootTenant -}}
29+
{{- $enabled = toString (dig "spec" "monitoring" false $rootTenant) -}}
30+
{{- else -}}
31+
{{- $enabled = toString (index .Values._cluster "monitoring-enabled" | default "true") -}}
32+
{{- end -}}
33+
{{- if eq $enabled "true" }}
134
---
235
apiVersion: v1
336
kind: Service
@@ -25,3 +58,4 @@ metadata:
2558
spec:
2659
type: ExternalName
2760
externalName: vminsert-longterm.tenant-root.svc.cluster.local
61+
{{- end }}
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
suite: monitoring ExternalName redirects in cozy-monitoring
2+
templates:
3+
- templates/monitoring-external-services.yaml
4+
5+
release:
6+
name: cozystack-basics
7+
namespace: cozy-system
8+
9+
# The vlinsert-generic / vminsert-shortterm / vminsert-longterm ExternalName
10+
# Services redirect cozy-monitoring at the platform monitoring stack the root
11+
# tenant hosts in tenant-root. They only resolve when the root tenant actually
12+
# runs that stack (Tenant/root spec.monitoring=true). When it is off the
13+
# redirects dead-end and fluent-bit silently drops logs (issue #3181), so they
14+
# must NOT be rendered.
15+
#
16+
# The template resolves that in two steps: a lookup of the live Tenant/root
17+
# (source of truth) with a fallback to _cluster.monitoring-enabled when lookup
18+
# returns nothing. helm-unittest renders without a cluster, so lookup is always
19+
# empty here and these cases exercise the FALLBACK branch — the flag plus its
20+
# "true" default. The lookup branch is validated live, not in unit tests. The
21+
# cases below pin both halves of the fallback contract.
22+
tests:
23+
- it: renders all three ExternalName redirects when monitoring-enabled is "true"
24+
set:
25+
_cluster:
26+
monitoring-enabled: "true"
27+
asserts:
28+
- hasDocuments:
29+
count: 3
30+
- equal:
31+
path: metadata.name
32+
value: vlinsert-generic
33+
documentIndex: 0
34+
- equal:
35+
path: metadata.namespace
36+
value: cozy-monitoring
37+
documentIndex: 0
38+
- equal:
39+
path: metadata.name
40+
value: vminsert-shortterm
41+
documentIndex: 1
42+
- equal:
43+
path: metadata.name
44+
value: vminsert-longterm
45+
documentIndex: 2
46+
47+
- it: keeps the redirects (default) when monitoring-enabled is absent
48+
asserts:
49+
- hasDocuments:
50+
count: 3
51+
52+
- it: omits all redirects when monitoring-enabled is "false"
53+
set:
54+
_cluster:
55+
monitoring-enabled: "false"
56+
asserts:
57+
- hasDocuments:
58+
count: 0
59+
60+
- it: vlinsert-generic still points at tenant-root when rendered
61+
set:
62+
_cluster:
63+
monitoring-enabled: "true"
64+
documentIndex: 0
65+
asserts:
66+
- equal:
67+
path: spec.type
68+
value: ExternalName
69+
- equal:
70+
path: spec.externalName
71+
value: vlinsert-generic.tenant-root.svc.cluster.local

‎packages/system/monitoring/templates/vlogs/vlogs.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,13 @@
1+
{{- /*
2+
Reject an empty logsStorages outright. With no entry no VLCluster is
3+
rendered, the monitoring-system HelmRelease readiness gate (which keys on the
4+
VLCluster) has nothing to match, and fluent-bit ends up shipping to a
5+
vlinsert-generic that never exists — silently dropping logs (issue #3181).
6+
Fail the render instead so the misconfiguration is loud, not silent.
7+
*/ -}}
8+
{{- if not .Values.logsStorages }}
9+
{{- fail "monitoring: logsStorages must define at least one entry; an empty list renders no VLCluster and fluent-bit silently drops logs to a non-existent vlinsert-generic (issue #3181)" }}
10+
{{- end }}
111
{{- range .Values.logsStorages }}
212
---
313
apiVersion: operator.victoriametrics.com/v1
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
suite: VLCluster rendering requires a non-empty logsStorages
2+
templates:
3+
- templates/vlogs/vlogs.yaml
4+
5+
release:
6+
name: monitoring
7+
namespace: tenant-root
8+
9+
# logsStorages drives the VLCluster(s) that back platform log ingestion. An
10+
# empty list renders no VLCluster, so the monitoring-system HelmRelease
11+
# readiness gate has nothing to match and fluent-bit silently drops logs to a
12+
# non-existent vlinsert-generic (issue #3181). The template rejects that
13+
# outright rather than documenting it, so the misconfiguration is loud. These
14+
# cases pin both the valid default and the rejected empty configuration.
15+
tests:
16+
- it: renders a VLCluster for the default (non-empty) logsStorages
17+
asserts:
18+
- containsDocument:
19+
apiVersion: operator.victoriametrics.com/v1
20+
kind: VLCluster
21+
name: generic
22+
23+
- it: renders one VLCluster per logsStorages entry
24+
set:
25+
logsStorages:
26+
- name: generic
27+
retentionPeriod: "1"
28+
storage: 10Gi
29+
- name: audit
30+
retentionPeriod: "7"
31+
storage: 10Gi
32+
asserts:
33+
- hasDocuments:
34+
count: 2
35+
36+
- it: fails the render when logsStorages is empty
37+
set:
38+
logsStorages: []
39+
asserts:
40+
- failedTemplate:
41+
errorMessage: "monitoring: logsStorages must define at least one entry; an empty list renders no VLCluster and fluent-bit silently drops logs to a non-existent vlinsert-generic (issue #3181)"

0 commit comments

Comments
 (0)