You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6789907
Browse filesBrowse the repository at this point in the historyBrowse files
docs(observability): describe the write rate and logins central tracing ships
The guide described one VMUser for a central tenant's writes and reads
and a rate bound in terms the shipped limiter no longer matches. A
central tenant now has a write-only login for its collector and a
read-only one for Grafana, the rate is counted before sampling, a
request above 3 MiB gets a 400 that exporters do not retry, each pod
start lets up to 5 MiB through above the rate, and central mode allows
at most 10 collector replicas. Operators sizing a shared store and
tenants choosing an exporter need those numbers, and the trust-boundary
section needs to say why a tenant can write only through its
collector.
Signed-off-by: Alexey Artamonov <[email protected]>
Copy file name to clipboardExpand all lines: docs/observability/distributed-tracing.md
+16-6Lines changed: 16 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,7 @@ env:
33
33
value: "deployment.environment.name=production"
34
34
```
35
35
36
-
The variables are defined by the OpenTelemetry specification, and the official SDKs and agents read them, but support for each variable differs per language: check your SDK against the [specification compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md). For a JVM workload you can add the OpenTelemetry Java agent (`-javaagent:/otel/opentelemetry-javaagent.jar`), and the same variables drive it with no code changes. To use gRPC instead, set the endpoint to `http://otel-traces:4317` **and** `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`; with [shared-central tracing](#shared-central-tracing), stay on HTTP, since spans refused over the tenant's write rate are retried only over HTTP.
36
+
The variables are defined by the OpenTelemetry specification, and the official SDKs and agents read them, but support for each variable differs per language: check your SDK against the [specification compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md). For a JVM workload you can add the OpenTelemetry Java agent (`-javaagent:/otel/opentelemetry-javaagent.jar`), and the same variables drive it with no code changes. To use gRPC instead, set the endpoint to `http://otel-traces:4317` **and** `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`; with [shared-central tracing](#shared-central-tracing), stay on HTTP, since spans refused over the tenant's write rate are retried only over HTTP, and keep each export request below 3 MiB.
37
37
38
38
You do not need to tag your tenant: the backend a tenant writes to is its own, and in shared-central mode the platform attributes every span to its tenant itself. `service.namespace` is for grouping your own services (for example by team), not for the tenant.
39
39
@@ -77,10 +77,18 @@ Instead of a VictoriaTraces backend per tenant, tenants can send their spans to
77
77
78
78
tenant-root opts in with `tracingCentralHost: true` on its Monitoring application, next to a `tracingStorages` entry named `generic` in `cluster` mode; that store is then shared with every tenant that opts in. A vmauth in tenant-root fronts it, and every central tenant's collector and Grafana go through that vmauth.
79
79
80
-
All central tenants share the store's disk and retention with tenant-root, and nothing reserves space per tenant. Every traces store is bounded by disk usage: without `retentionDiskUsageBytes` it drops its oldest day once its volume is 80% full. VictoriaTraces always keeps the last two days, though, so the disk cap alone does not stop a fast writer from filling the volume. Each tenant is therefore also held to a write rate, which the platform sets with `monitoring.tracingCentralTenantBytesPerSecond` in the platform package. By default it is about 97 KiB/s, at which one tenant alone cannot fill the disk cap of a default 10Gi store within two days, so raise it with the store. A change reaches every tenant's collector through the `cozystack-values` Secret, and upgrades every release in every tenant once. Each tenant's collector enforces it and refuses data above it. An OTLP/HTTP exporter gets a 429 and retries; an OTLP/gRPC exporter gets `RESOURCE_EXHAUSTED` without the `RetryInfo` the OTLP specification requires for a retry, so it drops those spans. Applications of central tenants should therefore export over HTTP. Several tenants writing at their limits together can still exceed the cap, so size the volume for the tenants you admit.
80
+
All central tenants share the store's disk and retention with tenant-root, and nothing reserves space per tenant. Every traces store is bounded by disk usage: without `retentionDiskUsageBytes` it drops its oldest day once its volume is 80% full. VictoriaTraces always keeps the last two days, though, so the disk cap alone does not stop a fast writer from filling the volume.
81
81
82
82
There is no per-tenant allow-list: with hosting on, every tenant that sets `tracingCentral` is admitted, nested tenants included. A tenant-root that needs to limit who shares its store keeps `tracingCentralHost` off.
83
83
84
+
### The per-tenant write rate
85
+
86
+
Because the disk cap alone does not bound a fast writer, each central tenant is also held to a write rate, which the platform sets with `monitoring.tracingCentralTenantBytesPerSecond` in the platform package, as a whole number of bytes per second up to 1099511627776 (1 TiB/s); any other value fails the platform render. By default it is about 97 KiB/s, at which one tenant writes at most the disk cap of a default 10Gi store over the two days, so raise it with the store. The rate is counted uncompressed and before the collector samples, so the store receives only the sampled share of it and keeps that compressed. A change reaches every tenant's collector through the `cozystack-values` Secret, and upgrades every release in every tenant once.
87
+
88
+
The collector Cozystack ships is built with a rate-limit processor, and each central tenant's collector enforces the rate, split across its replicas: a sender whose connection stays on one replica gets that replica's share. Data above the rate is refused. An OTLP/HTTP exporter gets a 429 and retries; an OTLP/gRPC exporter gets `RESOURCE_EXHAUSTED` without the `RetryInfo` the OTLP specification requires for a retry, so it drops those spans. A request larger than 3 MiB, decompressed, is refused with a 400, which exporters do not retry. Applications of central tenants should therefore export over HTTP, in batches below 3 MiB.
89
+
90
+
The limiter keeps its state in each collector pod's memory, and every pod starts with a full allowance, so a pod start lets through above the rate up to 5 MiB, or one second of the pod's share of the rate when that is larger. Shared-central tracing therefore allows at most 10 collector replicas, and the tenant's Monitoring render fails above that. The rate bounds each tenant, not their sum: several tenants writing at their limits together can still exceed the cap, so size the volume for the tenants you admit.
91
+
84
92
### Opting a tenant in
85
93
86
94
Set `tracingCentral: true` on the Tenant, which needs `monitoring: true` on the same tenant. The tenant chart then renders the egress rules the collector and Grafana need to reach the vmauth, and switches the tenant's Monitoring to the shared store. If tenant-root does not host the store, the tenant's Monitoring render fails with a message naming the prerequisite, rather than dropping spans.
@@ -89,25 +97,27 @@ In the tenant's Grafana the shared store appears as the `traces-central` datasou
89
97
90
98
### Isolation and the trust boundary
91
99
92
-
The store itself authorizes nothing; it takes the tenant from request headers. Each central tenant gets a VMUser that pins an account derived from its namespace name and UID, for its collector's writes and its Grafana's reads alike, and vmauth replaces any account a client sends. Tenant pods can reach only the vmauth proxy port, and only from pods carrying the collector's or Grafana's label, with the tenant's credentials. So tenants sharing the store cannot read or write each other's traces.
100
+
The store itself authorizes nothing; it takes the tenant from request headers. Each central tenant gets two VMUsers that pin one account, derived from its namespace name and UID: the collector's, which may only write, and its Grafana's, which may only read, each with its own Secret (`traces-central-credentials` and `traces-central-reader-credentials`). vmauth replaces any account a client sends. Tenant pods can reach only the vmauth proxy port, and only from pods carrying the collector's or Grafana's label, with the tenant's credentials. So tenants sharing the store cannot read or write each other's traces.
101
+
102
+
Within its own account, a tenant writes only through its collector, so the write rate holds. Grafana's login cannot write, so a write sent through the Grafana datasource proxy, which vmauth would otherwise route to the store, finds no route. The collector and Grafana images are the platform's and cannot be set from the Monitoring application, and of a Secret named in `oidc.customConfig.secretRef` only its `auth.ini` key is mounted into Grafana, so the collector's password does not reach a pod the tenant can shape.
93
103
94
104
tenant-root is inside the store's trust boundary. Its workloads reach the store directly, and one that learns a tenant's account can read and write it. Whoever tenant-root admits to its Grafana can read every account through tenant-root's own datasource for the store, which does not go through vmauth. Who that is follows tenant-root Grafana's OIDC mode: with `System`, tenant-root's own groups, which already hold the same access level in every tenant under it; with `CustomConfig` or `None`, whoever that configuration admits.
95
105
96
106
### Lifecycle
97
107
98
108
- **The account follows the namespace.** It is derived from the namespace name and UID, so a re-created namespace, a restored one included, starts a new account and does not see the spans stored before.
99
-
- **Turning central on** keeps the tenant's local stores and their datasources. Right after it, spans the collector sends before the vmauth has loaded the tenant's login get a 401, which the exporter does not retry, so those spans are dropped. The same happens after the tenant's `traces-central-credentials` Secret is re-created with a new password.
109
+
- **Turning central on** keeps the tenant's local stores and their datasources. Right after it, spans the collector sends before the vmauth has loaded the tenant's login get a 401, which the exporter does not retry, so those spans are dropped. The same happens after the tenant's `traces-central-credentials` Secret is re-created with a new password; after `traces-central-reader-credentials` is, the tenant's Grafana gets a 401 on reads until the vmauth loads the new login.
100
110
- **Turning central off** removes the tenant's login to the shared store. The spans it wrote there stay until the store's retention drops them and cannot be read from the tenant while it is off; turning it on again on the same namespace makes them readable again. Spans sent between turning it off and the collector switching back to a local store may be dropped, since the egress rule to the shared store goes first.
101
111
- **Deleting a tenant** leaves its spans in the shared store until retention drops them.
102
112
103
113
### Stopping hosting
104
114
105
-
Turn `tracingCentral` off on every central tenant and let its Monitoring reconcile before tenant-root stops hosting. tenant-root's render refuses to stop hosting while a central tenant's VMUser remains and lists the tenants still holding one. Removing tenant-root's Monitoring altogether is not a render and is not refused; until the central tenants turn central off, every Monitoring render of theirs fails, their metrics, logs and alerting changes included.
115
+
Turn `tracingCentral` off on every central tenant and let its Monitoring reconcile before tenant-root stops hosting. tenant-root's render refuses to stop hosting while a central tenant's VMUsers remain and lists the tenants still holding them. Removing tenant-root's Monitoring altogether is not a render and is not refused; until the central tenants turn central off, every Monitoring render of theirs fails, their metrics, logs and alerting changes included.
106
116
107
117
The two refusals read each other's objects when they render, so a tenant turning central on while tenant-root stops hosting can pass both. That tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning the tenant's `tracingCentral` off instead stops the silent drop, and turning it on again is then refused until tenant-root hosts.
108
118
109
119
## Prerequisites
110
120
111
121
- The tenant's Monitoring application with a `tracingStorages` entry, or shared-central tracing turned on for the tenant, and `tracingCollector.enabled` left on (the default).
112
-
- For shared-central tracing: tenant-root hosting the store, as described above.
122
+
- For shared-central tracing: tenant-root hosting the store, as described above, and at most 10 collector replicas (`tracingCollector.replicas`).
113
123
- A workload with OpenTelemetry instrumentation (SDK or auto-instrumentation agent).
0 commit comments