Skip to content

Commit 360f03b

Browse files
authored
feat(monitoring,tenant): shared-central tracing topology (phase 4) (#3775)
## What this PR does Adds the opt-in shared-central traces topology from the distributed tracing design (cozystack/community `design-proposals/distributed-tracing`, rollout item 4): a tenant can send its spans to one VictoriaTraces store in tenant-root instead of running its own, with both write and read isolation between the tenants that share it. **Switching it on.** A tenant opts in with the new Tenant field `tracingCentral`, which needs the tenant's own Monitoring. The tenant chart publishes it to the tenant's monitoring charts as `_namespace.tracingCentral` in the `cozystack-values` Secret, which the apps API does not let an application write, and renders the two network paths the topology needs under the same condition. tenant-root decides whether it hosts the store, with the new Monitoring field `tracingCentralHost` next to a cluster-mode `tracingStorages` entry named `generic`. **Isolation.** VictoriaTraces takes the tenant account from the `AccountID`/`ProjectID` headers and does no authorization of its own, so other tenants reach the shared store only through a vmauth in tenant-root. Each central tenant gets a VMUser that pins an account derived from its namespace name and UID, for both its collector's writes and its Grafana reads, and vmauth overrides any account a client sends. The collector and Grafana log in with credentials the chart generates and keeps. The tenant egress rules allow only pods carrying the collector's or Grafana's label to reach the vmauth proxy port, and a pod that carries such a label without the tenant's credentials gets only 401 from it; the vmauth internal routes, whose `/metrics` lists every tenant's user, are served on a port tenants cannot reach. The tenant-root namespace itself is inside the trust boundary, as it already is for the metrics of tenants that rely on a parent's monitoring: its workloads reach the store directly, and one that learns a tenant's account ID can read and write that account, and whoever tenant-root admits to its Grafana reads the store through a datasource that bypasses vmauth (with the System oidc mode, tenant-root's own groups, which already hold the same access level in every tenant under it); the `tracingCentralHost` description says so. **Sharing the store.** Each tenant's VMUser is capped at four requests in flight per vmauth replica, reads and writes together, and each collector replica exports with two senders, so more collector replicas do not raise a tenant's throughput; above the cap vmauth queues for up to 10s and answers 429, which the exporter retries. The store has one disk and one retention for all of its tenants and nothing reserves space per tenant; `tracingCentralHost` states that, since hosting is tenant-root's decision. The vmauth runs two replicas spread across nodes under a PodDisruptionBudget, so draining one node does not stop every central tenant at once. Every traces store is also disk-bounded now, as the design requires: without `retentionDiskUsageBytes` it drops its oldest day once its filesystem is 80% full. That alone does not keep one tenant from filling the shared disk, since VictoriaTraces always keeps the last two days. **States that would drop spans silently are refused, one race aside.** A live render fails, with a message naming the prerequisite, when a tenant turns central on while tenant-root hosts nothing, and when tenant-root stops hosting while central tenants remain; tenant-root's render also refuses `tracingCentralHost` without a store to host. The second check does not wait on a VMUser that is already being deleted, since its tenant is leaving. Each check reads the other side's objects when it renders, so a tenant turning central on while tenant-root stops hosting can pass both and export to a vmauth that is gone; the `tracingCentralHost` description names this race and its recovery. At runtime, tenant-root's Monitoring release is not Ready while the vmauth has failed. These checks run on a render, so removing the tenant-root Monitoring altogether is not refused, and from then on every Monitoring render of a central tenant fails until it turns central off; the `tracingCentralHost` description says to turn central off on the tenants first. Turning central on keeps the tenant's own local stores and their datasources, so traces stored before the switch stay readable until the tenant removes them; turning it off again leaves the spans written meanwhile in the shared store, unreadable from the tenant until it turns central on again, and spans sent while the collector switches back may be dropped; a deleted tenant's spans likewise stay until retention drops them. Right after a tenant turns central on, or after its credentials Secret is re-created, spans its collector sends before the vmauth has loaded the new login get a 401, which the exporter does not retry, so they are dropped; the tracing guide in #3772 covers it. **Where this departs from the design.** The design has the per-tenant collector inject the account headers and adds exactly one egress rule. Here the account is pinned by the tenant's VMUser at the vmauth and the collector sends none, because the collector runs in the tenant namespace and whatever runs with its pod label could otherwise choose an account; vmauth is also what the design already requires for reads, so both directions go through it. That needs a second egress rule, for the tenant's Grafana to reach the vmauth, beside the collector's. The per-tenant write rate the design gives the collector is not in this PR: no OpenTelemetry distribution has a rate-limiting processor, so #4648 builds the collector with one and holds each central tenant to a rate tenant-root publishes. Until it lands, a tenant writing faster than the disk cap allows over two days can fill the disk the other tenants share. **Consent is per host, not per tenant.** `tracingCentralHost` lets every tenant that sets `tracingCentral` in, nested tenants included, and each gets its own VMUser and its own request cap; there is no allow-list. A tenant-root that needs to limit who shares its store keeps `tracingCentralHost` off. **Upgrade.** Tenants that do not opt in render the same objects as before, and their `cozystack-values` Secret is unchanged. The monitoring charts render the same objects too, with one exception: a VTSingle or VTCluster without `retentionDiskUsageBytes` gains `retention.maxDiskUsagePercent: "80"` in its `extraArgs`, so its spec changes and its storage pods restart once, and a store already more than 80% full drops its oldest days when it starts. The inner monitoring HelmRelease's values gain the new `tracingCentralHost: false` default, as with any new field, and its `healthCheckExprs` gain a `VMAuth` entry, which matches nothing outside a hosting tenant-root. Both change the HelmRelease spec, so every tenant's monitoring release upgrades once, rendering the same objects. A tenant-root that has a cluster-mode `generic` store gets no vmauth until it sets `tracingCentralHost`. **Testing.** helm-unittest covers the three charts the change touches (tenant, monitoring, extra monitoring) and every guard was checked by mutation; a bats test covers what helm-unittest cannot, that the generated password and the collector's rollout checksum agree on a first install. On a test cluster: a spoofed `AccountID` from a tenant is overridden by vmauth, one tenant cannot read another's spans, the vmauth starts under the real tenant network policies (and stays in init without its apiserver label), the guards refuse as described, and the rendered collector config passes `otelcol validate` in the pinned image. There is no end-to-end test yet; one that covers cross-tenant isolation is planned as a follow-up. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: covered by #3772, the tracing guide - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#64 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The Tenant gains the user-facing field `tracingCentral` (bool, default false), which the trigger map sends to terraform-provider-cozystack: its hand-written Tenant schema and model need the field. The follow-up is cozystack/terraform-provider-cozystack#64. The Monitoring gains `tracingCentralHost` (bool, default false); the provider's `cozystack_tenant_module` is a marker without values, so it needs no change. The website reference pages for both are generated from the READMEs on release. The tracing user guide in #3772 covers the per-tenant backend only. A central-mode section in the tracing guide of #3772 (hosting and opting in, the shared disk and retention, turning central off on the tenants before tenant-root stops hosting, and the recovery from the opt-in/stop-hosting race) also takes the lifecycle details the `tracingCentral` description no longer carries. ### Release note ```release-note feat(monitoring,tenant): add an opt-in shared-central traces store. A tenant sets tracingCentral to send its spans to one VictoriaTraces store in tenant-root instead of its own, and tenant-root opts in to hosting it with tracingCentralHost on its Monitoring. Tenants sharing the store are isolated on write and read through a vmauth in tenant-root, and each is capped in the requests it can hold in flight. ```
2 parents 2d0c359 + 68ee973 commit 360f03b

42 files changed

Lines changed: 2092 additions & 59 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎api/apps/v1alpha1/tenant/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/usr/bin/env bats
2+
# The central traces collector reads its vmauth password at start and is rolled
3+
# by a checksum of it. On a first install the password is random, and it is
4+
# rendered twice, into the Secret and into that checksum: if the two renders
5+
# disagree, the collector rolls on every reconcile and the checksum proves
6+
# nothing. helm-unittest asserts one template at a time and cannot compare them,
7+
# so this renders both together and compares.
8+
#
9+
# Run via hack/cozytest.sh from the repo root (make bats-unit-tests).
10+
11+
MONITORING_CHART=packages/system/monitoring
12+
13+
@test "a first install hashes the same password it stores" {
14+
out=$(mktemp -d)
15+
for run in 1 2 3; do
16+
helm template monitoring "$MONITORING_CHART" --namespace tenant-foo \
17+
--show-only templates/vtraces/central-credentials.yaml \
18+
--show-only templates/vtraces/collector.yaml \
19+
--set _namespace.tracingCentral=true --set _namespace.host=example.org \
20+
> "$out/render-$run.yaml"
21+
password=$(yq eval 'select(.kind == "Secret") | .stringData.password' "$out/render-$run.yaml")
22+
checksum=$(yq eval 'select(.kind == "Deployment") | .spec.template.metadata.annotations["checksum/credentials"]' "$out/render-$run.yaml")
23+
# Separate commands: under set -e a failure on the left of && is not fatal.
24+
[ -n "$password" ]
25+
[ "$password" != "null" ]
26+
[ "$(printf '%s' "$password" | sha256sum | cut -d' ' -f1)" = "$checksum" ]
27+
done
28+
rm -rf "$out"
29+
}

‎packages/apps/tenant/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ tenant-u1
7979
| `host` | The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). | `string` | `""` |
8080
| `etcd` | Deploy own Etcd cluster. | `bool` | `false` |
8181
| `monitoring` | Deploy own Monitoring Stack. | `bool` | `false` |
82+
| `tracingCentral` | Send this tenant's traces to the shared VictoriaTraces store in tenant-root instead of a per-tenant backend; its Grafana reads them through a vmauth there that pins the tenant's own account, so tenants sharing the store cannot read each other's traces. It needs the tenant's own Monitoring (`monitoring: true`) and tenant-root hosting the store (`tracingCentralHost: true` on its Monitoring, with a `cluster`-mode tracingStorages entry named `generic`); without that the tenant's monitoring render fails rather than dropping spans. tenant-root, and whoever it admits to its Grafana, are inside the store's trust boundary. | `bool` | `false` |
8283
| `ingress` | Deploy own Ingress Controller. | `bool` | `false` |
8384
| `gateway` | Deploy own Gateway API Gateway (backed by Cilium Gateway API controller). When unset (the default) or set to `false`, the tenant does not own a Gateway and instead publishes through the nearest ancestor tenant that owns one, regardless of whether `host` is empty or custom (`false` currently behaves the same as unset and does not opt out of that inheritance). Set to `true` to give the tenant its own Gateway with its own LoadBalancer IP and Certificate. A tenant with a custom (non-derived) `host` that needs its apex publicly reachable must set `true` explicitly, because an ancestor's TLS certificate does not cover a custom apex. Note: leave the key absent (do not write `gateway: null`): the chart distinguishes "unset" via missing-key, not via null value, to satisfy the JSON schema generated from this comment. | `bool` | `false` |
8485
| `gatewayClass` | GatewayClass for this tenant's own Gateway. Empty uses the platform default (`gateway.className`). Only the classes the operator lists in the platform's `gateway.tenantSelectableClasses`, plus that default, may be named here — anything else fails this tenant's own gateway release and reaches no other tenant. Read only when this tenant owns a Gateway; an inheriting tenant uses its ancestor's. Whether a class terminates TLS itself or at its provider's edge is declared once per class on the platform, not here. | `string` | `""` |

‎packages/apps/tenant/templates/_helpers.tpl‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,3 +92,11 @@ false
9292
{{- if .Values.gateway -}}true{{- else -}}false{{- end -}}
9393
{{- end -}}
9494
{{- end -}}
95+
96+
{{- /* The shared-central tracing hops out of a tenant: an opted-in tenant with
97+
Monitoring (which deploys the collector and Grafana that use them),
98+
never tenant-root, which hosts the shared store. Both egress rules read
99+
this one condition so the write and read paths cannot be split. */}}
100+
{{- define "tenant.tracingCentralEgress" -}}
101+
{{- if and (ne (include "tenant.name" .) "tenant-root") .Values.tracingCentral .Values.monitoring -}}true{{- end -}}
102+
{{- end -}}

‎packages/apps/tenant/templates/namespace.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,9 @@ stringData:
125125
monitoring: {{ $monitoring | quote }}
126126
seaweedfs: {{ $seaweedfs | quote }}
127127
host: {{ $computedHost | quote }}
128+
{{- if include "tenant.tracingCentralEgress" . }}
129+
tracingCentral: true
130+
{{- end }}
128131
{{- with $schedulingClass }}
129132
schedulingClass: {{ . | quote }}
130133
{{- end }}
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{{- if include "tenant.tracingCentralEgress" . }}
2+
# Read hop of shared-central tracing: Grafana reaches the tenant-root vmauth,
3+
# never vtselect, which serves any account a client names.
4+
apiVersion: cilium.io/v2
5+
kind: CiliumClusterwideNetworkPolicy
6+
metadata:
7+
name: {{ include "tenant.name" . }}-egress-traces-central-read
8+
spec:
9+
endpointSelector:
10+
matchLabels:
11+
"k8s:io.kubernetes.pod.namespace": "{{ include "tenant.name" . }}"
12+
"k8s:app": grafana
13+
egress:
14+
- toEndpoints:
15+
- matchLabels:
16+
"k8s:app.kubernetes.io/name": vmauth
17+
"k8s:app.kubernetes.io/instance": traces-central
18+
"k8s:io.kubernetes.pod.namespace": tenant-root
19+
toPorts:
20+
- ports:
21+
- port: "8427"
22+
protocol: TCP
23+
{{- end }}
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{{- if include "tenant.tracingCentralEgress" . }}
2+
# Write hop of shared-central tracing: the collector reaches the tenant-root
3+
# vmauth, never vtinsert, which takes the account from the writer's headers.
4+
# vmauth sets it from the tenant's VMUser login, so a pod that only carries
5+
# the collector's label reaches no account but this tenant's.
6+
apiVersion: cilium.io/v2
7+
kind: CiliumClusterwideNetworkPolicy
8+
metadata:
9+
name: {{ include "tenant.name" . }}-egress-traces-central
10+
spec:
11+
endpointSelector:
12+
matchLabels:
13+
"k8s:io.kubernetes.pod.namespace": "{{ include "tenant.name" . }}"
14+
"k8s:app.kubernetes.io/name": "otel-traces"
15+
egress:
16+
- toEndpoints:
17+
- matchLabels:
18+
"k8s:app.kubernetes.io/name": vmauth
19+
"k8s:app.kubernetes.io/instance": traces-central
20+
"k8s:io.kubernetes.pod.namespace": tenant-root
21+
toPorts:
22+
- ports:
23+
- port: "8427"
24+
protocol: TCP
25+
{{- end }}
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
suite: shared-central tracing couples the collector backend to tracingCentral
2+
# _namespace.tracingCentral is the only switch to the shared store; it is
3+
# published under the same condition as the egress rules, and the monitoring
4+
# HelmRelease's spec.values stays the Monitoring application's.
5+
templates:
6+
- templates/monitoring.yaml
7+
- templates/namespace.yaml
8+
9+
release:
10+
name: tenant-ktj
11+
namespace: tenant-root
12+
13+
tests:
14+
- it: publishes tracingCentral=true for an opted-in tenant with monitoring
15+
set:
16+
monitoring: true
17+
tracingCentral: true
18+
template: templates/namespace.yaml
19+
documentSelector:
20+
path: metadata.name
21+
value: cozystack-values
22+
asserts:
23+
- matchRegex:
24+
path: stringData["values.yaml"]
25+
pattern: "(?m)^_namespace:\\n( .*\\n)* tracingCentral: true$"
26+
27+
# Every app in the tenant reads this Secret, so a key that is always written
28+
# would re-upgrade all of them on the platform upgrade.
29+
- it: leaves the flag out when the tenant did not opt in
30+
set:
31+
monitoring: true
32+
template: templates/namespace.yaml
33+
documentSelector:
34+
path: metadata.name
35+
value: cozystack-values
36+
asserts:
37+
- notMatchRegex:
38+
path: stringData["values.yaml"]
39+
pattern: "tracingCentral"
40+
41+
# The egress rules exist only for the tenant that opts in and select only its
42+
# own namespace, so a child inheriting the flag from a central parent would
43+
# export to the shared store with no path to it, every object Ready.
44+
- it: does not pass a central parent's flag on to a child with its own monitoring
45+
release:
46+
name: tenant-bar
47+
namespace: tenant-foo
48+
set:
49+
monitoring: true
50+
_namespace:
51+
tracingCentral: true
52+
monitoring: tenant-foo
53+
template: templates/namespace.yaml
54+
documentSelector:
55+
path: metadata.name
56+
value: cozystack-values
57+
asserts:
58+
- notMatchRegex:
59+
path: stringData["values.yaml"]
60+
pattern: "tracingCentral"
61+
62+
- it: leaves the flag out without monitoring, which deploys no collector
63+
set:
64+
monitoring: false
65+
tracingCentral: true
66+
template: templates/namespace.yaml
67+
documentSelector:
68+
path: metadata.name
69+
value: cozystack-values
70+
asserts:
71+
- notMatchRegex:
72+
path: stringData["values.yaml"]
73+
pattern: "tracingCentral"
74+
75+
- it: leaves the monitoring HelmRelease values to the Monitoring application
76+
set:
77+
monitoring: true
78+
tracingCentral: true
79+
template: templates/monitoring.yaml
80+
asserts:
81+
- notExists:
82+
path: spec.values
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
suite: shared-central traces egress reaches tenant-root at any depth
2+
# The shared store lives in tenant-root, not in the parent, so a tenant nested
3+
# below a non-root parent must still target tenant-root.
4+
templates:
5+
- templates/networkpolicy-traces-central.yaml
6+
release:
7+
name: tenant-deep
8+
namespace: tenant-ktj
9+
set:
10+
tracingCentral: true
11+
monitoring: true
12+
tests:
13+
- it: targets tenant-root even for a tenant nested below a non-root parent
14+
documentSelector:
15+
path: metadata.name
16+
value: tenant-ktj-deep-egress-traces-central
17+
asserts:
18+
- equal:
19+
path: spec.egress
20+
value:
21+
- toEndpoints:
22+
- matchLabels:
23+
"k8s:app.kubernetes.io/name": vmauth
24+
"k8s:app.kubernetes.io/instance": traces-central
25+
"k8s:io.kubernetes.pod.namespace": tenant-root
26+
toPorts:
27+
- ports:
28+
- port: "8427"
29+
protocol: TCP
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
suite: shared-central traces egress is off by default
2+
# The cross-boundary egress exists only for a tenant that opted in; by default
3+
# no tenant may reach the shared store.
4+
templates:
5+
- templates/networkpolicy-traces-central.yaml
6+
release:
7+
name: tenant-ktj
8+
namespace: tenant-root
9+
tests:
10+
- it: does not render the egress rule by default (tracingCentral unset)
11+
asserts:
12+
- containsDocument:
13+
apiVersion: cilium.io/v2
14+
kind: CiliumClusterwideNetworkPolicy
15+
name: tenant-ktj-egress-traces-central
16+
not: true
17+
18+
# Without Monitoring there is no collector to grant the hop to.
19+
- it: does not render the egress rule when monitoring is off (no collector)
20+
set:
21+
tracingCentral: true
22+
monitoring: false
23+
asserts:
24+
- containsDocument:
25+
apiVersion: cilium.io/v2
26+
kind: CiliumClusterwideNetworkPolicy
27+
name: tenant-ktj-egress-traces-central
28+
not: true
29+
30+
- it: renders the egress rule when both tracingCentral and monitoring are on
31+
set:
32+
tracingCentral: true
33+
monitoring: true
34+
asserts:
35+
- containsDocument:
36+
apiVersion: cilium.io/v2
37+
kind: CiliumClusterwideNetworkPolicy
38+
name: tenant-ktj-egress-traces-central

0 commit comments

Comments
 (0)