Skip to content

Commit a731fb2

Browse files
committed
fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and burst was the receiver's 4 MiB cap. Processors ahead of it grow a request: the sampler adds a tracestate to every span it keeps, and the limiter counts a request as the collector re-encodes it, a few bytes more per span, 13% for a span carrying only its IDs, the smallest valid one. A request the receiver admitted near its cap was therefore refused on every retry. The limiter now runs right after memory_limiter, so it counts the request before sampling, and burst is a quarter above the cap. The rate is counted before sampling, so the store keeps only the sampled share of it. The texts also say that a request above the cap gets a 400, which senders do not retry. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 2813a29 commit a731fb2

13 files changed

Lines changed: 99 additions & 70 deletions

File tree

‎api/apps/v1alpha1/tenant/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/tenant/README.md‎

Lines changed: 13 additions & 13 deletions
Large diffs are not rendered by default.

‎packages/apps/tenant/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
"default": false
1919
},
2020
"tracingCentral": {
21-
"description": "Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC.",
21+
"description": "Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP, in requests below 4 MiB: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC, and a larger request is refused and not retried.",
2222
"type": "boolean",
2323
"default": false
2424
},

‎packages/apps/tenant/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ etcd: false
1111
## @param {bool} monitoring - Deploy own Monitoring Stack.
1212
monitoring: false
1313

14-
## @param {bool} tracingCentral - Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC.
14+
## @param {bool} tracingCentral - Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. Applications should export over OTLP/HTTP, in requests below 4 MiB: spans above the platform's per-tenant write rate are retried over HTTP but dropped over gRPC, and a larger request is refused and not retried.
1515
tracingCentral: false
1616

1717
## @param {bool} ingress - Deploy own Ingress Controller.

‎packages/core/platform/values.yaml‎

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -662,19 +662,21 @@ monitoring:
662662
# here is true only to preserve the historical always-create behaviour;
663663
# operators MUST set it to match their actual Tenant/root spec.monitoring.
664664
rootEnabled: true
665-
# The rate, in bytes per second counted uncompressed, at which each tenant
666-
# using shared-central tracing may write into the traces store tenant-root
667-
# hosts; each tenant's collector refuses data above it, with a 429 an
668-
# OTLP/HTTP sender retries and a RESOURCE_EXHAUSTED an OTLP/gRPC sender
669-
# drops, so central tenants should export over HTTP. 0 keeps the default
670-
# of about 97 KiB/s: over the two days VictoriaTraces always keeps, one
671-
# tenant writes at most the disk cap of a default 10Gi store, counted
672-
# uncompressed, and the store keeps it compressed. Each collector pod
673-
# starts with a full bucket, so a pod start lets up to 4 MiB through above
674-
# the rate. Raise it with the store's size, as a whole number up to
675-
# 1099511627776 (1 TiB/s); anything else fails the render. A change
676-
# upgrades every release in every tenant once, since it travels in the
677-
# cozystack-values Secret they all read.
665+
# The rate, in bytes per second counted uncompressed and before the
666+
# collector samples, at which each tenant using shared-central tracing may
667+
# write into the traces store tenant-root hosts; each tenant's collector
668+
# refuses data above it, with a 429 an OTLP/HTTP sender retries and a
669+
# RESOURCE_EXHAUSTED an OTLP/gRPC sender drops, so central tenants should
670+
# export over HTTP; a request above 4 MiB gets a 400, not retried. 0 keeps
671+
# the default of about 97 KiB/s: over the two days VictoriaTraces always
672+
# keeps, one tenant writes at most the disk cap of a default 10Gi store,
673+
# counted uncompressed, and the store keeps it compressed and only the
674+
# sampled share of it (samplingPercentage). Each collector pod starts with a
675+
# full bucket, so a pod start lets up to 5 MiB through above the rate. Raise
676+
# it with the store's size, as a whole number up to 1099511627776 (1 TiB/s);
677+
# anything else fails the render. A change upgrades every release in every
678+
# tenant once, since it travels in the cozystack-values Secret they all
679+
# read.
678680
tracingCentralTenantBytesPerSecond: 0
679681
# Backup storage overrides for the platform-managed cozy-default BackupClass.
680682
# Everything set here is forwarded verbatim into the emitted

0 commit comments

Comments
 (0)