Commit 178edc6

"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to tracingCentralTenantBytesPerSecond, which by default lets no single tenant fill the cap within the two days VictoriaTraces always keeps; several tenants writing at their limit together still can. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",

"tracingCentralTenantBytesPerSecond": {

"description": "tenant-root only, with tracingCentralHost: the rate, in bytes per second, each central tenant may write into the shared store, counted uncompressed. 0 derives it from the `generic` store so that one tenant alone cannot fill the store's disk cap within the two days VictoriaTraces always keeps: twice the per-replica cap (retentionDiskUsageBytes, or 80% of the volume) over 172800 seconds, which for a 10Gi volume is about 97 KiB/s. Spans are stored compressed, so the default errs low. Each tenant's collector enforces it, split across its replicas, and refuses data above it: an OTLP/HTTP sender gets a 429 and retries, while an OTLP/gRPC sender gets RESOURCE_EXHAUSTED without the RetryInfo a retry needs and drops those spans, so applications of central tenants should export over HTTP.",

"type": "integer",

"default": 0,

"minimum": 0

},

"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (tenant-root's tracingCentralTenantBytesPerSecond) is split across them as well.",

## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to tracingCentralTenantBytesPerSecond, which by default lets no single tenant fill the cap within the two days VictoriaTraces always keeps; several tenants writing at their limit together still can. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.

tracingCentralTenantBytesPerSecond: 0

{{- /*

*/ -}}

{{- define "monitoring.quantityBytes" -}}

{{- $units := dict "" 1.0 "k" 1e3 "M" 1e6 "G" 1e9 "T" 1e12 "P" 1e15 "Ki" 1024.0 "Mi" 1048576.0 "Gi" 1073741824.0 "Ti" 1099511627776.0 "Pi" 1125899906842624.0 -}}

{{- $q := toString . -}}

{{- $m := regexFind "^[0-9]+(\\.[0-9]+)?" $q -}}

{{- $unit := trimPrefix $m $q -}}

{{- if or (not $m) (not (hasKey $units $unit)) -}}

{{- fail (printf "monitoring: cannot read %q as a volume size to derive the shared-central tenant rate from; write it as a number with an optional k/M/G/T/P or Ki/Mi/Gi/Ti/Pi suffix, or set tracingCentralTenantBytesPerSecond." $q) -}}

{{- end -}}

{{- mulf (float64 $m) (get $units $unit) -}}

{{- end -}}

{{- /* Bytes in a VictoriaMetrics BytesString (8GB, 8GiB), as a float. The

format is validated with the tracingStorages entry. */ -}}

{{- define "monitoring.bytesStringBytes" -}}

{{- $units := dict "" 1.0 "kb" 1e3 "mb" 1e6 "gb" 1e9 "tb" 1e12 "KB" 1e3 "MB" 1e6 "GB" 1e9 "TB" 1e12 "KiB" 1024.0 "MiB" 1048576.0 "GiB" 1073741824.0 "TiB" 1099511627776.0 -}}

{{- $s := toString . -}}

{{- $m := regexFind "^[0-9]+" $s -}}

{{- mulf (float64 $m) (get $units (trimPrefix $m $s)) -}}

{{- end -}}

{{- /*

*/ -}}

{{- define "monitoring.tracingCentralRateFromCap" -}}

{{- printf "%.0f" (floor (divf (mulf 2 .) 172800)) -}}

{{- end -}}

{{- /* The rate a central tenant's collector falls back to when tenant-root

publishes none: that of a default 10Gi store without a byte cap. */ -}}

{{- define "monitoring.tracingCentralDefaultTenantRate" -}}

{{- include "monitoring.tracingCentralRateFromCap" (mulf 0.8 (include "monitoring.quantityBytes" "10Gi")) -}}

{{- end -}}

{{- /*

*/ -}}

{{- define "monitoring.tracingCentralTenantRate" -}}

{{- $explicit := int64 (.Values.tracingCentralTenantBytesPerSecond | default 0) -}}

{{- if gt $explicit 0 -}}

{{- $explicit -}}

{{- else -}}

{{- range .Values.tracingStorages -}}

{{- if eq .name "generic" -}}

{{- $cap := 0.0 -}}

{{- if .retentionDiskUsageBytes -}}

{{- $cap = include "monitoring.bytesStringBytes" .retentionDiskUsageBytes | float64 -}}

{{- else -}}

{{- $cap = mulf 0.8 (include "monitoring.quantityBytes" .storage) -}}

{{- end -}}

{{- include "monitoring.tracingCentralRateFromCap" $cap -}}

{{- end -}}

{{- end -}}

{{- end -}}

{{- end -}}