Repository navigation
Commit 052a73c
committed
feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.
The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.
Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.
Signed-off-by: Alexey Artamonov <[email protected]>1 parent 8f421a9 commit 052a73c
10 files changed
Lines changed: 237 additions & 18 deletions
File tree
- packages
- core/platform
- templates
- tests
- extra/monitoring
- system
- monitoring-rd/cozyrds
- monitoring
- templates/vtraces
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
256 | 262 | | |
257 | 263 | | |
258 | 264 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
662 | 662 | | |
663 | 663 | | |
664 | 664 | | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
665 | 673 | | |
666 | 674 | | |
667 | 675 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
382 | 382 | | |
383 | 383 | | |
384 | 384 | | |
385 | | - | |
| 385 | + | |
386 | 386 | | |
387 | 387 | | |
388 | 388 | | |
| |||
410 | 410 | | |
411 | 411 | | |
412 | 412 | | |
413 | | - | |
| 413 | + | |
414 | 414 | | |
415 | 415 | | |
416 | 416 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
114 | | - | |
| 114 | + | |
115 | 115 | | |
116 | 116 | | |
| 117 | + | |
117 | 118 | | |
118 | 119 | | |
119 | 120 | | |
120 | 121 | | |
121 | 122 | | |
122 | 123 | | |
123 | | - | |
| 124 | + | |
124 | 125 | | |
125 | 126 | | |
126 | 127 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
| 47 | + | |
46 | 48 | | |
47 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
48 | 66 | | |
49 | 67 | | |
50 | 68 | | |
| |||
75 | 93 | | |
76 | 94 | | |
77 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
78 | 105 | | |
79 | 106 | | |
80 | 107 | | |
| |||
116 | 143 | | |
117 | 144 | | |
118 | 145 | | |
119 | | - | |
| 146 | + | |
120 | 147 | | |
121 | 148 | | |
122 | 149 | | |
| |||
0 commit comments