Skip to content

Commit 954fdbd

Browse files
committed
docs(observability): name the platform key that sets the write rate
The per-tenant write rate moved from tenant-root's Monitoring to the platform's monitoring.tracingCentralTenantBytesPerSecond in #4648, so that a change reaches every tenant's collector through the cozystack-values Secret instead of a render-time lookup that an in-sync release never repeats. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 842eccd commit 954fdbd

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎docs/observability/distributed-tracing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ Instead of a VictoriaTraces backend per tenant, tenants can send their spans to
7777

7878
tenant-root opts in with `tracingCentralHost: true` on its Monitoring application, next to a `tracingStorages` entry named `generic` in `cluster` mode; that store is then shared with every tenant that opts in. A vmauth in tenant-root fronts it, and every central tenant's collector and Grafana go through that vmauth.
7979

80-
All central tenants share the store's disk and retention with tenant-root, and nothing reserves space per tenant. Every traces store is bounded by disk usage: without `retentionDiskUsageBytes` it drops its oldest day once its volume is 80% full. VictoriaTraces always keeps the last two days, though, so the disk cap alone does not stop a fast writer from filling the volume. Each tenant is therefore also held to a write rate, `tracingCentralTenantBytesPerSecond` on tenant-root's Monitoring. By default the rate is derived so that one tenant alone cannot fill the disk cap within two days, about 97 KiB/s for a 10Gi volume. Each tenant's collector enforces it and refuses data above it. An OTLP/HTTP exporter gets a 429 and retries; an OTLP/gRPC exporter gets `RESOURCE_EXHAUSTED` without the `RetryInfo` the OTLP specification requires for a retry, so it drops those spans. Applications of central tenants should therefore export over HTTP. Several tenants writing at their limits together can still exceed the cap, so size the volume for the tenants you admit.
80+
All central tenants share the store's disk and retention with tenant-root, and nothing reserves space per tenant. Every traces store is bounded by disk usage: without `retentionDiskUsageBytes` it drops its oldest day once its volume is 80% full. VictoriaTraces always keeps the last two days, though, so the disk cap alone does not stop a fast writer from filling the volume. Each tenant is therefore also held to a write rate, which the platform sets with `monitoring.tracingCentralTenantBytesPerSecond` in the platform package. By default it is about 97 KiB/s, at which one tenant alone cannot fill the disk cap of a default 10Gi store within two days, so raise it with the store. A change reaches every tenant's collector through the `cozystack-values` Secret, and upgrades every release in every tenant once. Each tenant's collector enforces it and refuses data above it. An OTLP/HTTP exporter gets a 429 and retries; an OTLP/gRPC exporter gets `RESOURCE_EXHAUSTED` without the `RetryInfo` the OTLP specification requires for a retry, so it drops those spans. Applications of central tenants should therefore export over HTTP. Several tenants writing at their limits together can still exceed the cap, so size the volume for the tenants you admit.
8181

8282
There is no per-tenant allow-list: with hosting on, every tenant that sets `tracingCentral` is admitted, nested tenants included. A tenant-root that needs to limit who shares its store keeps `tracingCentralHost` off.
8383

0 commit comments

Comments
 (0)