Skip to content

Commit a34a687

Browse files
committed
docs(monitoring,tenant): tell central tenants to export over HTTP
The collector refuses spans above the tenant's write rate with a 429 over OTLP/HTTP, which senders retry, and with RESOURCE_EXHAUSTED without RetryInfo over gRPC, which the OTLP specification makes final. The warning lived in the description of the Monitoring knob that moved to the platform and went with it, so nothing a tenant reads said that a gRPC sender loses every span above the rate. It now sits in the Tenant's tracingCentral, the Monitoring tracingCollector and the platform key, and the image comment names the platform's rate. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 8ee9435 commit a34a687

11 files changed

Lines changed: 14 additions & 12 deletions

File tree

‎api/apps/v1alpha1/tenant/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/tenant/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ tenant-u1
7979
| `host` | The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). | `string` | `""` |
8080
| `etcd` | Deploy own Etcd cluster. | `bool` | `false` |
8181
| `monitoring` | Deploy own Monitoring Stack. | `bool` | `false` |
82-
| `tracingCentral` | Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. | `bool` | `false` |
82+
| `tracingCentral` | Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC. | `bool` | `false` |
8383
| `ingress` | Deploy own Ingress Controller. | `bool` | `false` |
8484
| `gateway` | Deploy own Gateway API Gateway (backed by Cilium Gateway API controller). When unset (the default) or set to `false`, the tenant does not own a Gateway and instead publishes through the nearest ancestor tenant that owns one, regardless of whether `host` is empty or custom (`false` currently behaves the same as unset and does not opt out of that inheritance). Set to `true` to give the tenant its own Gateway with its own LoadBalancer IP and Certificate. A tenant with a custom (non-derived) `host` that needs its apex publicly reachable must set `true` explicitly, because an ancestor's TLS certificate does not cover a custom apex. Note: leave the key absent (do not write `gateway: null`): the chart distinguishes "unset" via missing-key, not via null value, to satisfy the JSON schema generated from this comment. | `bool` | `false` |
8585
| `gatewayClass` | GatewayClass for this tenant's own Gateway. Empty uses the platform default (`gateway.className`). Only the classes the operator lists in the platform's `gateway.tenantSelectableClasses`, plus that default, may be named here — anything else fails this tenant's own gateway release and reaches no other tenant. Read only when this tenant owns a Gateway; an inheriting tenant uses its ancestor's. Whether a class terminates TLS itself or at its provider's edge is declared once per class on the platform, not here. | `string` | `""` |

‎packages/apps/tenant/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
"default": false
1919
},
2020
"tracingCentral": {
21-
"description": "Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary.",
21+
"description": "Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC.",
2222
"type": "boolean",
2323
"default": false
2424
},

‎packages/apps/tenant/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ etcd: false
1111
## @param {bool} monitoring - Deploy own Monitoring Stack.
1212
monitoring: false
1313

14-
## @param {bool} tracingCentral - Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary.
14+
## @param {bool} tracingCentral - Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC.
1515
tracingCentral: false
1616

1717
## @param {bool} ingress - Deploy own Ingress Controller.

‎packages/core/platform/values.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -664,7 +664,9 @@ monitoring:
664664
rootEnabled: true
665665
# The rate, in bytes per second counted uncompressed, at which each tenant
666666
# using shared-central tracing may write into the traces store tenant-root
667-
# hosts; each tenant's collector refuses data above it. 0 keeps the default
667+
# hosts; each tenant's collector refuses data above it, with a 429 an
668+
# OTLP/HTTP sender retries and a RESOURCE_EXHAUSTED an OTLP/gRPC sender
669+
# drops, so central tenants should export over HTTP. 0 keeps the default
668670
# of about 97 KiB/s, at which one tenant alone cannot fill the disk cap of a
669671
# default 10Gi store within the two days VictoriaTraces always keeps. Raise
670672
# it with the store's size. A change upgrades every release in every tenant

‎packages/extra/monitoring/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@
7171

7272
| Name | Description | Type | Value |
7373
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------- |
74-
| `tracingCollector` | OpenTelemetry Collector for trace ingestion. | `object` | `{}` |
74+
| `tracingCollector` | OpenTelemetry Collector for trace ingestion. With shared-central tracing it refuses spans above the platform's per-tenant write rate: OTLP/HTTP senders retry them, OTLP/gRPC senders drop them, so export over HTTP. | `object` | `{}` |
7575
| `tracingCollector.enabled` | Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction); with shared-central tracing they have no path to it, so no new spans arrive, while the traces already stored stay readable. The collector fronts a single backend, and the limit holds with shared-central tracing too, since turning that off makes it export to the local entry again: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled. | `bool` | `true` |
7676
| `tracingCollector.replicas` | Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well. | `int` | `1` |
7777
| `tracingCollector.samplingPercentage` | Head-sampling percentage applied at the collector before export. | `int` | `10` |

‎packages/extra/monitoring/templates/helmrelease.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ spec:
9292
- kind: Secret
9393
name: cozystack-values
9494
{{- /* The apps API does not prune keys the Monitoring schema leaves out, and
95-
a central tenant's collector enforces tenant-root's rate on it, so its
95+
a central tenant's collector enforces the platform's rate on it, so its
9696
image stays the platform's. */}}
9797
{{- $values := deepCopy .Values }}
9898
{{- if kindIs "map" $values.tracingCollector }}

‎packages/extra/monitoring/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -387,7 +387,7 @@
387387
"default": false
388388
},
389389
"tracingCollector": {
390-
"description": "OpenTelemetry Collector for trace ingestion.",
390+
"description": "OpenTelemetry Collector for trace ingestion. With shared-central tracing it refuses spans above the platform's per-tenant write rate: OTLP/HTTP senders retry them, OTLP/gRPC senders drop them, so export over HTTP.",
391391
"type": "object",
392392
"default": {},
393393
"required": [

0 commit comments

Comments
 (0)