Skip to content

Commit 052a73c

Browse files
committed
feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that a noisy or hostile tenant cannot exhaust a shared backend. The disk cap alone does not do that: VictoriaTraces always keeps the last two days, so a tenant writing faster than the cap allows over two days fills the disk for every tenant on the store. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and it reaches every tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each monitoring HelmRelease reads, so Flux upgrades them all; a rate read with lookup at render time would never reach a tenant whose release is already in sync. The platform writes the key only when it is set, since every release in every tenant reads that Secret. Without it the collector holds the tenant to the rate of a default 10Gi store. Each central tenant's collector splits the rate across its replicas and refuses data above it before the batch processor, so the sender sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which the OTLP specification makes final, so central tenants should export over HTTP. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 8f421a9 commit 052a73c

10 files changed

Lines changed: 237 additions & 18 deletions

File tree

‎packages/core/platform/templates/apps.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,12 @@ stringData:
253253
null/omitted in custom values. Missing key -> the true default,
254254
matching the chart values.yaml. */}}
255255
monitoring-enabled: {{ dig "rootEnabled" true (.Values.monitoring | default dict) | quote }}
256+
{{- /* The rate each shared-central tracing tenant's collector holds it
257+
to. Only written when set: every release in every tenant reads
258+
this Secret, and a key appearing would upgrade all of them. */}}
259+
{{- with dig "tracingCentralTenantBytesPerSecond" 0 (.Values.monitoring | default dict) | int64 }}
260+
tracing-central-tenant-bytes-per-second: {{ . | quote }}
261+
{{- end }}
256262
cluster-domain: {{ .Values.networking.clusterDomain | quote }}
257263
api-server-endpoint: {{ .Values.publishing.apiServerEndpoint | quote }}
258264
{{- /* Display-only DNS suffix for the dashboard's Services tab
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
suite: apps.yaml monitoring.tracingCentralTenantBytesPerSecond → _cluster wiring
2+
templates:
3+
- templates/apps.yaml
4+
5+
release:
6+
name: cozy-platform
7+
namespace: cozy-system
8+
9+
# The rate each shared-central tracing tenant may write reaches every tenant's
10+
# collector through _cluster in the cozystack-values Secret. A change there
11+
# changes the values every monitoring HelmRelease reads, so Flux upgrades each
12+
# of them; a rate read with lookup at render time never would.
13+
14+
tests:
15+
- it: writes the rate when monitoring.tracingCentralTenantBytesPerSecond is set
16+
set:
17+
monitoring.tracingCentralTenantBytesPerSecond: 500000
18+
asserts:
19+
- matchRegex:
20+
path: stringData["values.yaml"]
21+
pattern: '(?m)^\s*tracing-central-tenant-bytes-per-second:\s*"500000"$'
22+
23+
# Every release in every tenant reads this Secret, so a key written while
24+
# nobody set a rate would upgrade all of them on the platform upgrade.
25+
- it: writes no rate by default
26+
asserts:
27+
- notMatchRegex:
28+
path: stringData["values.yaml"]
29+
pattern: "tracing-central-tenant-bytes-per-second"
30+
31+
- it: writes no rate for an explicit zero
32+
set:
33+
monitoring.tracingCentralTenantBytesPerSecond: 0
34+
asserts:
35+
- notMatchRegex:
36+
path: stringData["values.yaml"]
37+
pattern: "tracing-central-tenant-bytes-per-second"

‎packages/core/platform/values.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -662,6 +662,14 @@ monitoring:
662662
# here is true only to preserve the historical always-create behaviour;
663663
# operators MUST set it to match their actual Tenant/root spec.monitoring.
664664
rootEnabled: true
665+
# The rate, in bytes per second counted uncompressed, at which each tenant
666+
# using shared-central tracing may write into the traces store tenant-root
667+
# hosts; each tenant's collector refuses data above it. 0 keeps the default
668+
# of about 97 KiB/s, at which one tenant alone cannot fill the disk cap of a
669+
# default 10Gi store within the two days VictoriaTraces always keeps. Raise
670+
# it with the store's size. A change upgrades every release in every tenant
671+
# once, since it travels in the cozystack-values Secret they all read.
672+
tracingCentralTenantBytesPerSecond: 0
665673
# Backup storage overrides for the platform-managed cozy-default BackupClass.
666674
# Everything set here is forwarded verbatim into the emitted
667675
# cozystack.backupstrategy-controller Package CR as

‎packages/extra/monitoring/README.md‎

Lines changed: 11 additions & 11 deletions
Large diffs are not rendered by default.

‎packages/extra/monitoring/values.schema.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -382,7 +382,7 @@
382382
}
383383
},
384384
"tracingCentralHost": {
385-
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
385+
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets no single tenant fill the cap of a default 10Gi store within the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
386386
"type": "boolean",
387387
"default": false
388388
},
@@ -410,7 +410,7 @@
410410
}
411411
},
412412
"replicas": {
413-
"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads.",
413+
"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well.",
414414
"type": "integer",
415415
"default": 1,
416416
"minimum": 1

‎packages/extra/monitoring/values.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,16 +111,17 @@ logsStorages:
111111
## @param {[]TracingStorage} tracingStorages - Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector, while tracingCollector.enabled is true, without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed.
112112
tracingStorages: []
113113

114-
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
114+
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets no single tenant fill the cap of a default 10Gi store within the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
115115
tracingCentralHost: false
116116

117+
117118
##
118119
## @section Traces collector configuration
119120
##
120121

121122
## @typedef {struct} TracingCollector - Per-tenant OpenTelemetry Collector that fronts the traces backend, the tenant's own or, with shared-central tracing, the shared store in tenant-root. It is the app-facing OTLP ingest gateway and the point where sampling/redaction are enforced. Tenant isolation does not come from this workload: for a local backend it comes from the tenant NetworkPolicies, and with shared-central tracing from the vmauth in tenant-root, which pins the tenant's account to its credentials whatever the collector sends.
122123
## @field {bool} enabled=true - Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction); with shared-central tracing they have no path to it, so no new spans arrive, while the traces already stored stay readable. The collector fronts a single backend, and the limit holds with shared-central tracing too, since turning that off makes it export to the local entry again: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled.
123-
## @field {int} replicas=1 - Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads.
124+
## @field {int} replicas=1 - Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well.
124125
## @minimum 1
125126
## @field {int} samplingPercentage=10 - Head-sampling percentage applied at the collector before export.
126127
## @minimum 0

‎packages/system/monitoring-rd/cozyrds/monitoring.yaml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/system/monitoring/templates/vtraces/collector.yaml‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,26 @@
4343
{{- end -}}
4444
{{- $central := eq (include "monitoring.tracingCentral" .) "true" -}}
4545
{{- $endpoint := "" -}}
46+
{{- $rate := 0 -}}
47+
{{- $burst := 0 -}}
4648
{{- if $central -}}
4749
{{- $endpoint = "http://vmauth-traces-central.tenant-root.svc:8427/insert/opentelemetry/v1/traces" -}}
50+
{{- /* The platform sets the tenant's rate in _cluster, which reaches this
51+
release through the cozystack-values Secret, so a change upgrades it.
52+
Without one, or on a value that is not a positive number, the default
53+
holds: floor(2 vtstorage replicas * 80% of a 10Gi volume / 172800 s),
54+
at which one tenant alone cannot fill a default store within the two
55+
days VictoriaTraces always keeps. The rate is split across the replicas
56+
the Service spreads its senders over. burst is at least the largest
57+
request vtinsert accepts (-opentelemetry.traces.maxRequestSize, 64 MiB),
58+
since a request larger than burst never passes the limiter, and at
59+
least one second of rate. */ -}}
60+
{{- $tenantRate := int64 99420 -}}
61+
{{- $configured := toString (index (.Values._cluster | default dict) "tracing-central-tenant-bytes-per-second" | default "") -}}
62+
{{- if regexMatch "^[1-9][0-9]*$" $configured }}{{ $tenantRate = int64 $configured }}{{ end -}}
63+
{{- $replicas := int64 $c.replicas -}}
64+
{{- $rate = div (add $tenantRate (sub $replicas 1)) $replicas -}}
65+
{{- $burst = max 67108864 $rate -}}
4866
{{- else -}}
4967
{{- $backend := index .Values.tracingStorages 0 -}}
5068
{{- $mode := $backend.mode | default "cluster" -}}
@@ -75,6 +93,15 @@ processors:
7593
- key: cozystack.tenant
7694
value: {{ .Release.Namespace }}
7795
action: upsert
96+
# Holds the tenant to the rate the platform sets, so no tenant fills the
97+
# shared store's disk cap faster than its retention frees it. A refused
98+
# request gets a 429 over OTLP/HTTP, which senders retry; over gRPC it gets
99+
# RESOURCE_EXHAUSTED without RetryInfo, which senders treat as final.
100+
ratelimit/tenant:
101+
strategy: bytes
102+
rate: {{ $rate }}
103+
burst: {{ $burst }}
104+
throttle_behavior: error
78105
{{- end }}
79106
{{- if $redact }}
80107
attributes/redact:
@@ -116,7 +143,7 @@ service:
116143
pipelines:
117144
traces:
118145
receivers: [otlp]
119-
processors: [memory_limiter, {{ if $central }}resource/tenant, {{ end }}probabilistic_sampler, {{ if $redact }}attributes/redact, {{ end }}batch]
146+
processors: [memory_limiter, {{ if $central }}resource/tenant, {{ end }}probabilistic_sampler, {{ if $redact }}attributes/redact, {{ end }}{{ if $central }}ratelimit/tenant, {{ end }}batch]
120147
exporters: [otlphttp/victoriatraces]
121148
{{- end -}}
122149
{{- if and .Values.tracingCollector.enabled (or (include "monitoring.tracingCentral" .) (not (empty .Values.tracingStorages))) }}

0 commit comments

Comments
 (0)