Skip to content

Commit 518a224

Browse files
committed
fix(monitoring): count a central request before the tenant attribute
The limiter refuses any request larger than its burst, and burst is sized from the request cap the receiver enforces. The tenant attribute was added to every resource before the limiter counted the request, so a request the receiver admitted near its cap reached the limiter above burst and was refused on every retry. The limiter now counts the request before the attribute is added; nothing ahead of it makes a request larger. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 2079ce0 commit 518a224

3 files changed

Lines changed: 8 additions & 5 deletions

File tree

‎packages/system/monitoring/templates/vtraces/collector.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,8 @@ processors:
9595
# The account already separates tenants; the attribute keeps the source
9696
# tenant on each span once it is copied out of the store, where the account
9797
# does not travel. Prefixed so it does not overwrite an application's own
98-
# `tenant`.
98+
# `tenant`. It runs after the limiter, which would otherwise count what it
99+
# adds and refuse for good a request the receiver admitted at its cap.
99100
resource/tenant:
100101
attributes:
101102
- key: cozystack.tenant
@@ -151,7 +152,7 @@ service:
151152
pipelines:
152153
traces:
153154
receivers: [otlp]
154-
processors: [memory_limiter, {{ if $central }}resource/tenant, {{ end }}probabilistic_sampler, {{ if $redact }}attributes/redact, {{ end }}{{ if $central }}ratelimit/tenant, {{ end }}batch]
155+
processors: [memory_limiter, probabilistic_sampler, {{ if $redact }}attributes/redact, {{ end }}{{ if $central }}ratelimit/tenant, resource/tenant, {{ end }}batch]
155156
exporters: [otlphttp/victoriatraces]
156157
{{- end -}}
157158
{{- if and .Values.tracingCollector.enabled (or (include "monitoring.tracingCentral" .) (not (empty .Values.tracingStorages))) }}

‎packages/system/monitoring/tests/vtraces_central_ratelimit_test.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,12 @@ tests:
3030
pattern: "(?m)^ ratelimit/tenant:\\n strategy: bytes\\n rate: 100000\\n burst: 4194304\\n throttle_behavior: error$"
3131
# Before batch: the batch processor sends asynchronously, so a refusal
3232
# behind it would never reach the application and its spans would be
33-
# dropped without the sender knowing.
33+
# dropped without the sender knowing. Before resource/tenant: the
34+
# attribute it adds to every resource would push a request the receiver
35+
# admitted at its cap past burst, and such a request never passes.
3436
- matchRegex:
3537
path: data["config.yaml"]
36-
pattern: "processors: \\[memory_limiter, resource/tenant, probabilistic_sampler, ratelimit/tenant, batch\\]"
38+
pattern: "processors: \\[memory_limiter, probabilistic_sampler, ratelimit/tenant, resource/tenant, batch\\]"
3739

3840
- it: splits the tenant rate across collector replicas
3941
template: templates/vtraces/collector.yaml

‎packages/system/monitoring/tests/vtraces_collector_test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ tests:
147147
pattern: "key: cozystack.tenant\\n value: tenant-foo\\n"
148148
- matchRegex:
149149
path: data["config.yaml"]
150-
pattern: "processors: \\[memory_limiter, resource/tenant, probabilistic_sampler, ratelimit/tenant, batch\\]"
150+
pattern: "processors: \\[memory_limiter, probabilistic_sampler, ratelimit/tenant, resource/tenant, batch\\]"
151151

152152
- it: stamps each tenant with its own namespace
153153
release:

0 commit comments

Comments
 (0)