You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 385fb8d
Browse filesBrowse the repository at this point in the historyBrowse files
fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one
login it writes with was also the login of the tenant's Grafana
datasource, and it could both read and write. vmauth cleans the request
path before it picks a route, so any user the tenant admits to its
Grafana, a Viewer included, could POST through the datasource proxy to
../insert and write into the shared store past the limiter.
Each central tenant now has two VMUsers on its one account: the
collector's, which may only write, and Grafana's, which may only read,
each with its own chart-owned Secret and password. A namespace name
holds no dot, so the reader's username never matches another tenant's.
On upgrade the datasource moves to the new login in the same release
that creates it, so reads may fail until vmauth loads the new user.
Signed-off-by: Alexey Artamonov <[email protected]>
Copy file name to clipboardExpand all lines: packages/extra/monitoring/values.schema.json
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -382,7 +382,7 @@
382
382
}
383
383
},
384
384
"tracingCentralHost": {
385
-
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
385
+
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUsers carry) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
386
386
"type": "boolean",
387
387
"default": false
388
388
},
@@ -410,7 +410,7 @@
410
410
}
411
411
},
412
412
"replicas": {
413
-
"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.",
413
+
"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's write limit of four per vmauth replica, so more replicas do not raise the tenant's throughput; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.",
Copy file name to clipboardExpand all lines: packages/extra/monitoring/values.yaml
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -111,7 +111,7 @@ logsStorages:
111
111
## @param {[]TracingStorage} tracingStorages - Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector, while tracingCollector.enabled is true, without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed.
112
112
tracingStorages: []
113
113
114
-
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
114
+
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUsers carry) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
115
115
tracingCentralHost: false
116
116
117
117
##
@@ -120,7 +120,7 @@ tracingCentralHost: false
120
120
121
121
## @typedef {struct} TracingCollector - Per-tenant OpenTelemetry Collector that fronts the traces backend, the tenant's own or, with shared-central tracing, the shared store in tenant-root. It is the app-facing OTLP ingest gateway and the point where sampling/redaction are enforced. Tenant isolation does not come from this workload: for a local backend it comes from the tenant NetworkPolicies, and with shared-central tracing from the vmauth in tenant-root, which pins the tenant's account to its credentials whatever the collector sends. With shared-central tracing it also holds the tenant to the platform's write rate (monitoring.tracingCentralTenantBytesPerSecond) and refuses data above it: an OTLP/HTTP sender gets a 429 and retries, while an OTLP/gRPC sender gets RESOURCE_EXHAUSTED without RetryInfo and drops those spans, so applications of central tenants should export over HTTP.
122
122
## @field {bool} enabled=true - Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction); with shared-central tracing they have no path to it, so no new spans arrive, while the traces already stored stay readable. The collector fronts a single backend, and the limit holds with shared-central tracing too, since turning that off makes it export to the local entry again: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled.
123
-
## @field {int} replicas=1 - Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.
123
+
## @field {int} replicas=1 - Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's write limit of four per vmauth replica, so more replicas do not raise the tenant's throughput; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.
124
124
## @minimum 1
125
125
## @maximum 10
126
126
## @field {int} samplingPercentage=10 - Head-sampling percentage applied at the collector before export.
Copy file name to clipboardExpand all lines: packages/system/monitoring/templates/vtraces/vmauth.yaml
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -29,7 +29,7 @@
29
29
{{- end }}
30
30
{{- end }}
31
31
{{- with $users }}
32
-
{{- fail (printf "monitoring: tenant-root cannot stop hosting shared-central tracing while these tenants still use it: %s. Turn tracingCentral off on them first, then wait until each has reconciled its Monitoring and its VMUser traces-central is gone." (join ", " (sortAlpha (uniq .)))) }}
32
+
{{- fail (printf "monitoring: tenant-root cannot stop hosting shared-central tracing while these tenants still use it: %s. Turn tracingCentral off on them first, then wait until each has reconciled its Monitoring and its VMUsers traces-central and traces-central-reader are gone." (join ", " (sortAlpha (uniq .)))) }}
33
33
{{- end }}
34
34
{{- end }}
35
35
{{- if include "monitoring.tracingCentralStore" . }}
0 commit comments