Repository navigation
Commit 11499c3
authored
feat(monitoring): hold each shared-central tenant to a write rate (#4648)
## What this PR does
Closes the part of the distributed tracing design that #3775 left open:
the per-tenant collector rate-limits its tenant, so that a noisy or
hostile tenant cannot exhaust a shared backend (cozystack/community
`design-proposals/distributed-tracing`, Security and §2).
**Why the disk cap is not enough.** #3775 bounds every traces store by
disk usage, but VictoriaTraces always keeps the last two days, so a
tenant writing faster than the cap allows over two days fills the disk
every tenant on the shared store writes to. Only a per-tenant write rate
closes that.
**The collector is built here.** No upstream OpenTelemetry distribution
or processor limits throughput, and neither the open-source vmauth nor
VictoriaTraces limits a tenant's rate. Elastic's EDOT Collector
registers Elastic's `ratelimitprocessor` among its default components,
but it ships under the Elastic License 2.0 and carries far more
components than the chart uses. The collector is therefore built with
the OpenTelemetry Collector Builder from the upstream components the
chart already uses, at the same v0.136.0, plus that `ratelimitprocessor`
(Apache-2.0). Its v0.8.2 is the release built against collector
v0.135.0, the last one before v0.137.0, and it builds against v0.136.0
unchanged. The image is versioned by the collector release with a
cozystack respin counter (`v0.136.0-cozystack.0`), runs as the upstream
image's user, and is built with the other package images.
**How the rate is set and enforced.** The platform sets the rate with
`monitoring.tracingCentralTenantBytesPerSecond`, and it reaches every
tenant in `_cluster` through the `cozystack-values` Secret. A change
there is a change in the values every monitoring HelmRelease reads, so
Flux upgrades each of them; a rate read with `lookup` at render time
would never reach a tenant whose release is already in sync. The
platform writes the key only for a positive rate, because every release
in every tenant reads that Secret, and refuses to render on a value that
is not a whole number of bytes per second in plain decimal digits, up to
1 TiB/s ("1Mi", 1.5, "0100"), rather than reading it as the default.
Without it, or on a value that is not a positive number, the collector
holds the tenant to the default of about 97 KiB/s, at which one tenant
writes at most the disk cap of a default 10Gi store over two days,
counted uncompressed; the store keeps spans compressed, so they take
less than that. A larger store needs the rate raised with it. Each
central tenant's collector splits the rate across its replicas and
refuses data above it before the batch processor, so the application
sees the refusal, and right after the memory limiter, ahead of the
sampler, which adds a tracestate to every span it keeps, and of the
tenant attribute added to every resource, so it counts a request as the
receiver admitted it. The rate is therefore counted before sampling, and
the store keeps only the sampled share of it. Over OTLP/HTTP it gets a
429, which OTLP exporters retry. Over OTLP/gRPC the limiter answers
RESOURCE_EXHAUSTED without RetryInfo, which the OTLP specification makes
final, so a gRPC exporter drops those spans; the Tenant's
`tracingCentral`, the Monitoring `tracingCollector` and the platform key
tell central tenants to export over HTTP. The limiter sees the requests
the receiver admits, so in central mode the receiver caps a request at 3
MiB over both OTLP/HTTP and gRPC, decompressed for HTTP. The limiter
counts a request as the collector re-encodes it, which can be larger
than it arrived (13% for a span carrying only its IDs, the smallest
valid one, and up to 1.4x for spans with empty-valued attributes), so
burst is at least 5 MiB, above 1.4 times the cap, since a request larger
than burst never passes. Each collector pod starts with a full bucket,
so a pod start lets through above the rate up to 5 MiB or one second of
the pod's share of the rate, whichever is larger, and the chart refuses
to render a central tenant's collector with more than 10 replicas.
**Only the collector can write.** The rate holds only if the collector's
login is the only way the tenant writes. Each central tenant now has two
VMUsers on its one account: the collector's, which may only write, and
its Grafana datasource's, which may only read, each with its own Secret.
With one login for both, any user the tenant admits to its Grafana, a
Viewer included, could POST through the datasource proxy to `../insert`,
which vmauth cleans before it picks a route, and write into the shared
store past the limiter. The tenant's Grafana pod may reach the vmauth
too, so its image is the platform's, like the collector's, and only the
`auth.ini` key is mounted from the Secret a tenant names in
`oidc.customConfig.secretRef`: an image of the tenant's choosing with
the collector's Secret mounted would otherwise write past the limiter as
well.
**The collector image is the platform's.** The apps API defaults a
Monitoring spec from its schema but does not prune keys the schema
leaves out, and the Monitoring chart hands its values to the inner chart
as they are, so a tenant could set `tracingCollector.image` and run any
collector it liked, one without the limiter included. That key is no
longer passed through.
**Trade-offs.** The limiter is a vendor component, not an OpenTelemetry
one, and Elastic marks it as in development; it moves with Elastic's
releases, so bumping the collector means picking the matching release of
it (v0.74.0 builds against collector v1.67.0), and collector v0.136.0 is
a year old. It logs each refused request at error level, so a tenant
held at its limit produces steady log lines in its own collector.
Several tenants writing at their limits together can still exceed the
cap within two days; the rate bounds each tenant, not their sum. The
limiter keeps its state in each pod's memory, so a tenant admin who
restarts its collector pods over and over gains, per pod start, up to 5
MiB or one second of the pod's share of the rate, whichever is larger;
closing that needs a limiter with shared state (the processor's
gubernator mode), which is out of scope here. A sender whose connection
stays on one replica gets that replica's share of the rate. In central
mode the receiver refuses a request larger than 3 MiB: over OTLP/HTTP
with a 400, which senders do not retry, where the receiver default is 20
MiB, and over gRPC, where the default is 4 MiB. Changing the rate
upgrades every release in every tenant once, since it travels in the
Secret they all read. Until CI builds the image,
`tracingCollector.image` in `values.yaml` still names the upstream
distribution, which has no rate limiter, so a central-mode collector
from an unbuilt tree does not start; CI and releases stamp the built
image, and the image repository in the build registry has to be public
before a cluster can pull it.
**Upgrade.** Collectors in both modes move to the built image, which
carries the same components at the same version; central collectors also
roll for the new processor. The replica cap fails the render of a
central tenant's collector with more than 10 replicas, while a collector
writing to the tenant's own store keeps any replica count; the collector
has shipped only in the v1.7.0 alphas. Each central tenant's existing
VMUser loses its read route, and a second VMUser and Secret appear for
Grafana, whose datasource moves to the new login in the same release, so
the tenant's trace reads may fail until vmauth loads the new user.
Nothing else changes for clusters that do not set the rate: the platform
writes no new key, so no other release upgrades.
**Testing.** helm-unittest covers the platform writing the rate into
`_cluster` only for a positive whole number and refusing a malformed
one, numbers Helm reads as float64 included, the collector's limiter
(the configured rate, the split across replicas, burst, the receiver's
request cap in central mode and not in local mode, the default when
unset and on a word, zero or a negative value, its place right after the
memory limiter, with and without redaction, its absence in local mode),
the replica cap for a central tenant's collector and its absence for a
local one, the image key no longer passing through, the two logins (the
collector's with only the write route, Grafana's with only the read
route, separate Secrets and passwords), the Grafana image not passing
through, and only `auth.ini` being mounted from a `secretRef` Secret for
a central tenant; every term was checked by mutation. On a test cluster,
with the chart's rendered VMAuth, VMUsers, VTCluster, Grafana datasource
and collector: with one login for both, a Grafana Viewer's POST to
`../insert` through the datasource proxy landed a span in the shared
store; with the two logins vmauth answered `400 missing route` for every
encoding of the traversal tried, reads went through, and the collector's
login could write but not read. With the built collector at 100%
sampling and the receiver capped at 4 MiB, a fresh pod accepted a
request just under that cap made of 53-byte spans and one made of
ID-only spans, refused a larger one with a 400, and answered 429 once
the bucket was drained; the 53-byte spans it accepted were read back
through the read-only login; the 3 MiB cap the chart renders has not
been run on a cluster. A pod mounting a Secret through the chart's
`secretRef` volume as rendered for a central tenant got only `auth.ini`,
and one naming the collector's Secret, which has no such key, did not
start. The design's Testing section also asks for a sustained span flood
against a disk-capped store to show that the oldest spans are evicted
and ingest is not blocked; there is no end-to-end lane for
shared-central tracing yet, so that flood test is a follow-up, like the
cross-tenant isolation test #3775 names. On a test cluster, under the
network policies the tenant chart renders, the built image validated the
rendered config in both modes and wrote and read through the shared
store as before; a tenant given a small limit got 200 for its first
OTLP/HTTP requests and 429 for the rest of a burst, with the processor
logging each refusal, while another tenant under the default rate got
200 for all of the same requests.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#729
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
The new platform key `monitoring.tracingCentralTenantBytesPerSecond`
belongs in the hand-written `platform-package.md` reference on the
website, tracked in cozystack/website#729. The Monitoring schema and the
Tenant's `tracingCentral` change only descriptions; the provider models
neither, so it needs nothing. The central-mode section #3772 adds to the
tracing guide covers the rate.
### Release note
```release-note
feat(monitoring): hold each shared-central tracing tenant to a write rate. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and each central tenant's collector refuses spans above it, which bounds what one tenant writes into the store every tenant shares. The traces collector is now built by cozystack from the upstream components plus Elastic's rate-limit processor, and its image can no longer be set from a Monitoring application. A central tenant's Grafana reads the shared store with a login that cannot write, and its image can no longer be set from a Monitoring application.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Central tracing now applies a configurable per-tenant write-rate limit
shared across collector replicas. The default is approximately 97 KiB
per second, and collector replica counts are limited to 10.
* Each starting replica can temporarily exceed its rate share by up to 5
MiB, or by one second of that share when it is larger. Tenants share
central storage, so combined writes can fill the store and cause older
traces to be evicted.
* **Documentation**
* OTLP/HTTP is recommended for central tracing: excess traffic is
retried over HTTP, while OTLP/gRPC traffic may be dropped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->35 files changed
Lines changed: 971 additions & 124 deletions
File tree
- api/apps/v1alpha1/tenant
- hack
- packages
- apps/tenant
- core/platform
- templates
- tests
- extra/monitoring
- templates
- tests
- system
- monitoring-rd/cozyrds
- monitoring
- images/otelcol-tracing
- templates
- grafana
- vtraces
- tests
- tenant-rd/cozyrds
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
68 | | - | |
69 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
70 | 72 | | |
71 | 73 | | |
72 | | - | |
73 | | - | |
| 74 | + | |
| 75 | + | |
74 | 76 | | |
75 | 77 | | |
76 | | - | |
| 78 | + | |
| 79 | + | |
77 | 80 | | |
78 | 81 | | |
79 | 82 | | |
| |||
195 | 198 | | |
196 | 199 | | |
197 | 200 | | |
198 | | - | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
199 | 206 | | |
200 | 207 | | |
201 | | - | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
202 | 211 | | |
203 | 212 | | |
204 | 213 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| |||
0 commit comments