Skip to content

Commit 385fb8d

Browse files
committed
fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one login it writes with was also the login of the tenant's Grafana datasource, and it could both read and write. vmauth cleans the request path before it picks a route, so any user the tenant admits to its Grafana, a Viewer included, could POST through the datasource proxy to ../insert and write into the shared store past the limiter. Each central tenant now has two VMUsers on its one account: the collector's, which may only write, and Grafana's, which may only read, each with its own chart-owned Secret and password. A namespace name holds no dot, so the reader's username never matches another tenant's. On upgrade the datasource moves to the new login in the same release that creates it, so reads may fail until vmauth loads the new user. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 6f67c5f commit 385fb8d

14 files changed

Lines changed: 224 additions & 89 deletions

‎packages/extra/monitoring/README.md‎

Lines changed: 24 additions & 24 deletions
Large diffs are not rendered by default.

‎packages/extra/monitoring/values.schema.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -382,7 +382,7 @@
382382
}
383383
},
384384
"tracingCentralHost": {
385-
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
385+
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUsers carry) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
386386
"type": "boolean",
387387
"default": false
388388
},
@@ -410,7 +410,7 @@
410410
}
411411
},
412412
"replicas": {
413-
"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.",
413+
"description": "Number of collector replicas. Head sampling is trace-id-deterministic, so replicas\u003e1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's write limit of four per vmauth replica, so more replicas do not raise the tenant's throughput; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.",
414414
"type": "integer",
415415
"default": 1,
416416
"maximum": 10,

‎packages/extra/monitoring/values.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ logsStorages:
111111
## @param {[]TracingStorage} tracingStorages - Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector, while tracingCollector.enabled is true, without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed.
112112
tracingStorages: []
113113

114-
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
114+
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together. Each tenant is held to the platform's monitoring.tracingCentralTenantBytesPerSecond, which by default lets one tenant write at most the cap of a default 10Gi store, counted uncompressed, over the two days VictoriaTraces always keeps, so raise it with the store; several tenants writing at their limit together still can fill it. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUsers carry) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
115115
tracingCentralHost: false
116116

117117
##
@@ -120,7 +120,7 @@ tracingCentralHost: false
120120

121121
## @typedef {struct} TracingCollector - Per-tenant OpenTelemetry Collector that fronts the traces backend, the tenant's own or, with shared-central tracing, the shared store in tenant-root. It is the app-facing OTLP ingest gateway and the point where sampling/redaction are enforced. Tenant isolation does not come from this workload: for a local backend it comes from the tenant NetworkPolicies, and with shared-central tracing from the vmauth in tenant-root, which pins the tenant's account to its credentials whatever the collector sends. With shared-central tracing it also holds the tenant to the platform's write rate (monitoring.tracingCentralTenantBytesPerSecond) and refuses data above it: an OTLP/HTTP sender gets a 429 and retries, while an OTLP/gRPC sender gets RESOURCE_EXHAUSTED without RetryInfo and drops those spans, so applications of central tenants should export over HTTP.
122122
## @field {bool} enabled=true - Deploy the OTLP collector. When disabled, applications must send OTLP straight to the traces backend (no central sampling/redaction); with shared-central tracing they have no path to it, so no new spans arrive, while the traces already stored stay readable. The collector fronts a single backend, and the limit holds with shared-central tracing too, since turning that off makes it export to the local entry again: while it is enabled, tracingStorages must hold at most one entry, and adding a second one fails the whole Monitoring render until the extra entry is removed or the collector is disabled.
123-
## @field {int} replicas=1 - Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's limit of four per vmauth replica, which its Grafana reads share, so more replicas do not raise the tenant's throughput and can slow its reads; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.
123+
## @field {int} replicas=1 - Number of collector replicas. Head sampling is trace-id-deterministic, so replicas>1 are safe and remove the single-ingest-path SPOF. With shared-central tracing each replica sends up to two requests at a time against this tenant's write limit of four per vmauth replica, so more replicas do not raise the tenant's throughput; the tenant's write rate (the platform's monitoring.tracingCentralTenantBytesPerSecond) is split across them as well, so a sender whose connection stays on one replica gets that replica's share, and each replica that starts lets up to 4 MiB through above the rate.
124124
## @minimum 1
125125
## @maximum 10
126126
## @field {int} samplingPercentage=10 - Head-sampling percentage applied at the collector before export.

‎packages/system/monitoring-rd/cozyrds/monitoring.yaml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/system/monitoring/templates/_helpers.tpl‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -291,15 +291,18 @@
291291
every template of one render shares.
292292
*/ -}}
293293
{{- define "monitoring.tracingCentralPassword" -}}
294-
{{- if not (hasKey .Values "_tracingCentralPassword") -}}
294+
{{- $ctx := index . 0 -}}
295+
{{- $secret := index . 1 -}}
296+
{{- $key := printf "_tracingCentralPassword_%s" $secret -}}
297+
{{- if not (hasKey $ctx.Values $key) -}}
295298
{{- $password := "" -}}
296-
{{- with (index (lookup "v1" "Secret" .Release.Namespace "traces-central-credentials") "data") -}}
299+
{{- with (index (lookup "v1" "Secret" $ctx.Release.Namespace $secret) "data") -}}
297300
{{- with .password }}{{ $password = b64dec . }}{{ end -}}
298301
{{- end -}}
299302
{{- if not $password }}{{ $password = randAlphaNum 32 }}{{ end -}}
300-
{{- $_ := set .Values "_tracingCentralPassword" $password -}}
303+
{{- $_ := set $ctx.Values $key $password -}}
301304
{{- end -}}
302-
{{- index .Values "_tracingCentralPassword" -}}
305+
{{- index $ctx.Values $key -}}
303306
{{- end -}}
304307

305308
{{- /* tenant-root hosts the shared store only when it opts in with
Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,25 @@
11
{{- /*
2-
The central tenant's login at the tenant-root vmauth, read by its collector
3-
and its Grafana datasource and referenced by its VMUser. Owned here because
2+
The central tenant's logins at the tenant-root vmauth: one the collector
3+
writes with, one the Grafana datasource reads with, each referenced by its
4+
VMUser. Grafana's must not be able to write, or a write sent through its
5+
datasource proxy would pass the collector's rate limit. Owned here because
46
the operator publishes a VMUser's generated Secret only while some VMAuth
57
selects that user, so the collector would otherwise wait on a Secret that
6-
never appears when tenant-root has no central store. The password is kept
8+
never appears when tenant-root has no central store. Each password is kept
79
across renders (monitoring.tracingCentralPassword).
810
*/ -}}
911
{{- if include "monitoring.tracingCentral" . }}
12+
{{- range $secret, $username := dict "traces-central-credentials" $.Release.Namespace "traces-central-reader-credentials" (printf "%s.reader" $.Release.Namespace) }}
1013
---
1114
apiVersion: v1
1215
kind: Secret
1316
metadata:
14-
name: traces-central-credentials
17+
name: {{ $secret }}
1518
labels:
16-
apps.cozystack.io/application.name: {{ .Release.Name }}
19+
apps.cozystack.io/application.name: {{ $.Release.Name }}
1720
type: Opaque
1821
stringData:
19-
username: {{ .Release.Namespace | quote }}
20-
password: {{ include "monitoring.tracingCentralPassword" . | quote }}
22+
username: {{ $username | quote }}
23+
password: {{ include "monitoring.tracingCentralPassword" (list $ $secret) | quote }}
24+
{{- end }}
2125
{{- end }}

‎packages/system/monitoring/templates/vtraces/collector.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,8 @@ exporters:
133133
{{- if $central }}
134134
auth:
135135
authenticator: basicauth/vmauth
136-
# The VMUser allows four requests in flight per vmauth replica, shared with
137-
# the tenant's Grafana reads; two senders keep writes under it.
136+
# The write VMUser allows four requests in flight per vmauth replica; two
137+
# senders keep writes under it.
138138
sending_queue:
139139
num_consumers: 2
140140
{{- end }}
@@ -194,7 +194,7 @@ spec:
194194
checksum/config: {{ include "monitoring.tracingCollectorConfig" . | sha256sum }}
195195
{{- if eq (include "monitoring.tracingCentral" .) "true" }}
196196
{{- /* The login is read from env at start, so a new one needs a new pod. */}}
197-
checksum/credentials: {{ include "monitoring.tracingCentralPassword" . | sha256sum }}
197+
checksum/credentials: {{ include "monitoring.tracingCentralPassword" (list . "traces-central-credentials") | sha256sum }}
198198
{{- end }}
199199
spec:
200200
securityContext:

‎packages/system/monitoring/templates/vtraces/grafana-datasource.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ spec:
3131
{{- end }}
3232
{{- if include "monitoring.tracingCentral" . }}
3333
{{- /* Central mode reads the shared store only through the tenant-root vmauth,
34-
as this tenant's VMUser (vmuser.yaml). The operator substitutes ${key}
34+
as this tenant's read-only VMUser (vmuser.yaml). The operator substitutes ${key}
3535
with that key of the referenced Secret. */}}
3636
---
3737
apiVersion: grafana.integreatly.org/v1beta1
@@ -43,12 +43,12 @@ spec:
4343
- targetPath: basicAuthUser
4444
valueFrom:
4545
secretKeyRef:
46-
name: traces-central-credentials
46+
name: traces-central-reader-credentials
4747
key: username
4848
- targetPath: secureJsonData.basicAuthPassword
4949
valueFrom:
5050
secretKeyRef:
51-
name: traces-central-credentials
51+
name: traces-central-reader-credentials
5252
key: password
5353
datasource:
5454
access: proxy

‎packages/system/monitoring/templates/vtraces/vmauth.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
{{- end }}
3030
{{- end }}
3131
{{- with $users }}
32-
{{- fail (printf "monitoring: tenant-root cannot stop hosting shared-central tracing while these tenants still use it: %s. Turn tracingCentral off on them first, then wait until each has reconciled its Monitoring and its VMUser traces-central is gone." (join ", " (sortAlpha (uniq .)))) }}
32+
{{- fail (printf "monitoring: tenant-root cannot stop hosting shared-central tracing while these tenants still use it: %s. Turn tracingCentral off on them first, then wait until each has reconciled its Monitoring and its VMUsers traces-central and traces-central-reader are gone." (join ", " (sortAlpha (uniq .)))) }}
3333
{{- end }}
3434
{{- end }}
3535
{{- if include "monitoring.tracingCentralStore" . }}

0 commit comments

Comments
 (0)