Skip to content

Commit ea425df

Browse files
scooby87scooby87
committed
feat(monitoring,tenant): shared-central tracing topology (phase 4)
Phase 4 of distributed tracing (design cozystack/community#38, epic #3761): the opt-in shared-central topology where per-tenant collectors export to a single shared VictoriaTraces in tenant-root instead of a per-tenant backend. - collector: tracingCollector.backend=local|central. In central mode it exports over OTLP/HTTP to vtinsert-generic (ExternalName -> tenant-root) and a resource processor stamps tenant=<namespace>, so all tenants share account 0 while staying distinguishable. No local backend/datasource is rendered in central mode (the #3181 empty-list and multi-backend guards become local-only). - tenant NetworkPolicy: a narrow CiliumClusterwideNetworkPolicy scoped to the otel-traces collector pods only, allowing egress to the ancestor vtinsert (mirrors the vminsert egress block but endpoint-scoped, so no other tenant workload gains the cross-boundary hop). Non-root tenants only. - cozystack-basics: vtinsert-generic ExternalName in cozy-monitoring -> tenant-root (mirrors vlinsert-generic), gated on monitoring-enabled. - backend enum in the schema; helm-unittest for central export + tenant stamp, central-mode local-backend/datasource skip, and the collector-scoped egress rule; regenerated schema/README/ApplicationDefinition. Read-side isolation on the shared backend (vmauth) is deferred future work per the design. Stacks on the phase-2 collector PR #3763. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent b000ef7 commit ea425df

23 files changed

Lines changed: 322 additions & 35 deletions

‎api/apps/v1alpha1/tenant/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/tenant/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ tenant-u1
7979
| `host` | The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). | `string` | `""` |
8080
| `etcd` | Deploy own Etcd cluster. | `bool` | `false` |
8181
| `monitoring` | Deploy own Monitoring Stack. | `bool` | `false` |
82+
| `tracingCentral` | Opt into the shared-central tracing topology: allow this tenant's OTLP collector to export traces to the shared VictoriaTraces in tenant-root (account 0, distinguished by a `tenant` attribute) instead of a per-tenant backend. Renders a narrow, collector-scoped Cilium egress rule. Set alongside the monitoring chart's `tracingCollector.backend=central`. The default per-tenant backend gives stronger isolation; see design-proposals/distributed-tracing in cozystack/community. | `bool` | `false` |
8283
| `ingress` | Deploy own Ingress Controller. | `bool` | `false` |
8384
| `gateway` | Deploy own Gateway API Gateway (backed by Cilium Gateway API controller). When unset (the default), the chart auto-enables the Gateway for tenants whose apex is derived from the parent (i.e. `host` is empty), and leaves it off for tenants with a custom non-derived apex. Set to `true` or `false` explicitly to override that auto-behaviour. Note: leave the key absent (do not write `gateway: null`) — the chart distinguishes "unset" via missing-key, not via null value, to satisfy the JSON schema generated from this comment. | `bool` | `false` |
8485
| `seaweedfs` | Deploy own SeaweedFS. | `bool` | `false` |

‎packages/apps/tenant/templates/networkpolicy-traces-central.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
{{- if and (ne (include "tenant.name" .) "tenant-root") .Values.tracingCentral }}
2+
# Shared-central tracing (opt-in via .Values.tracingCentral): allow ONLY the
3+
# per-tenant OTLP collector pods to reach the shared VictoriaTraces insert
4+
# (vtinsert) in tenant-root — the fixed location the vtinsert-generic
5+
# ExternalName in cozy-monitoring points at, for tenants at any depth. Egress is
6+
# endpoint-scoped to app.kubernetes.io/name=otel-traces, so no other tenant
7+
# workload gains this cross-boundary hop. Gated on tracingCentral so the
8+
# capability exists only for tenants that opted into the shared backend; the
9+
# operator sets it alongside the monitoring chart's tracingCollector.backend=central.
10+
#
11+
# Known limitation of the account-0 shared model: selection is by pod label, so
12+
# a tenant able to create pods could label one otel-traces, obtain this egress,
13+
# and write spans with an arbitrary `tenant` resource attribute into the shared
14+
# store. Real write- and read-side isolation on a shared backend needs an
15+
# authenticating proxy (vmauth) that derives the tenant from identity — deferred
16+
# future work per the design. Use the default per-tenant backend for hard
17+
# isolation.
18+
#
19+
# Kept in its own template file (one document, no empty guarded blocks) so the
20+
# opt-in/off-by-default contract is reliably assertable in helm-unittest.
21+
apiVersion: cilium.io/v2
22+
kind: CiliumClusterwideNetworkPolicy
23+
metadata:
24+
name: {{ include "tenant.name" . }}-egress-traces-central
25+
spec:
26+
endpointSelector:
27+
matchLabels:
28+
"k8s:io.kubernetes.pod.namespace": "{{ include "tenant.name" . }}"
29+
"k8s:app.kubernetes.io/name": "otel-traces"
30+
egress:
31+
- toEndpoints:
32+
- matchLabels:
33+
"k8s:app.kubernetes.io/name": "vtinsert"
34+
"k8s:io.kubernetes.pod.namespace": tenant-root
35+
{{- end }}

‎packages/apps/tenant/tests/networkpolicy_traces_central_depth_test.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
suite: shared-central traces egress reaches tenant-root at any depth
2+
# Regression guard: the shared backend is a fixed location (tenant-root, via the
3+
# vtinsert-generic ExternalName), so the collector egress must target tenant-root
4+
# regardless of how deep the tenant is. An earlier version derived the target
5+
# from the namespace ancestry and silently dropped traces for tenants below the
6+
# direct children of tenant-root. This renders a deeper tenant and pins the
7+
# target to tenant-root.
8+
templates:
9+
- templates/networkpolicy-traces-central.yaml
10+
release:
11+
name: tenant-deep
12+
namespace: tenant-ktj
13+
set:
14+
tracingCentral: true
15+
tests:
16+
- it: targets tenant-root even for a tenant nested below a non-root parent
17+
documentSelector:
18+
path: metadata.name
19+
value: tenant-ktj-deep-egress-traces-central
20+
asserts:
21+
- equal:
22+
path: spec.egress[0].toEndpoints[0].matchLabels["k8s:app.kubernetes.io/name"]
23+
value: vtinsert
24+
- equal:
25+
path: spec.egress[0].toEndpoints[0].matchLabels["k8s:io.kubernetes.pod.namespace"]
26+
value: tenant-root

‎packages/apps/tenant/tests/networkpolicy_traces_central_gate_test.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
suite: shared-central traces egress is off by default
2+
# Security control: the cross-boundary egress to the shared account-0 store must
3+
# NOT exist unless the tenant explicitly opted into central mode. If it rendered
4+
# by default, every non-root tenant would silently gain write access to the
5+
# shared backend (the over-grant this gate fixes). The opted-in positive path is
6+
# covered by networkpolicy_traces_central_test.yaml.
7+
templates:
8+
- templates/networkpolicy-traces-central.yaml
9+
release:
10+
name: tenant-ktj
11+
namespace: tenant-root
12+
tests:
13+
- it: does not render the egress rule by default (tracingCentral unset)
14+
asserts:
15+
- containsDocument:
16+
apiVersion: cilium.io/v2
17+
kind: CiliumClusterwideNetworkPolicy
18+
name: tenant-ktj-egress-traces-central
19+
not: true

‎packages/apps/tenant/tests/networkpolicy_traces_central_root_test.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
suite: shared-central traces egress never renders for tenant-root
2+
# tenant-root hosts the shared backend, so it must never get the cross-boundary
3+
# egress rule — even when tracingCentral is set. Pin the ne-tenant-root guard.
4+
templates:
5+
- templates/networkpolicy-traces-central.yaml
6+
release:
7+
name: tenant-root
8+
namespace: tenant-root
9+
set:
10+
tracingCentral: true
11+
tests:
12+
- it: is excluded even when tracingCentral is opted in
13+
asserts:
14+
- containsDocument:
15+
apiVersion: cilium.io/v2
16+
kind: CiliumClusterwideNetworkPolicy
17+
name: tenant-root-egress-traces-central
18+
not: true

‎packages/apps/tenant/tests/networkpolicy_traces_central_test.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
suite: tenant collector egress to the shared-central traces backend
2+
# Shared-central tracing exports spans across the tenant boundary to the shared
3+
# VictoriaTraces (vtinsert) in an ancestor namespace. Only the per-tenant OTLP
4+
# collector pods may make that hop — the rule is endpoint-scoped to
5+
# app.kubernetes.io/name=otel-traces so no other tenant workload gains
6+
# cross-boundary egress. Pin the selector and target so a widened scope or a
7+
# dropped rule fails a test rather than silently regressing tenant isolation.
8+
templates:
9+
- templates/networkpolicy-traces-central.yaml
10+
release:
11+
# A nested tenant (own namespace tenant-ktj) provisioned from tenant-root. The
12+
# egress target below is a fixed tenant-root, so it is correct at any depth —
13+
# not derived from the namespace ancestry.
14+
name: tenant-ktj
15+
namespace: tenant-root
16+
set:
17+
tracingCentral: true
18+
tests:
19+
- it: scopes the cross-boundary egress to only the collector pods
20+
documentSelector:
21+
path: metadata.name
22+
value: tenant-ktj-egress-traces-central
23+
asserts:
24+
- isKind:
25+
of: CiliumClusterwideNetworkPolicy
26+
- equal:
27+
path: spec.endpointSelector.matchLabels["k8s:app.kubernetes.io/name"]
28+
value: otel-traces
29+
- equal:
30+
path: spec.endpointSelector.matchLabels["k8s:io.kubernetes.pod.namespace"]
31+
value: tenant-ktj
32+
33+
- it: allows egress only to the shared vtinsert in tenant-root
34+
documentSelector:
35+
path: metadata.name
36+
value: tenant-ktj-egress-traces-central
37+
asserts:
38+
- equal:
39+
path: spec.egress[0].toEndpoints[0].matchLabels["k8s:app.kubernetes.io/name"]
40+
value: vtinsert
41+
- equal:
42+
path: spec.egress[0].toEndpoints[0].matchLabels["k8s:io.kubernetes.pod.namespace"]
43+
value: tenant-root

‎packages/apps/tenant/values.schema.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,11 @@
1717
"type": "boolean",
1818
"default": false
1919
},
20+
"tracingCentral": {
21+
"description": "Opt into the shared-central tracing topology: allow this tenant's OTLP collector to export traces to the shared VictoriaTraces in tenant-root (account 0, distinguished by a `tenant` attribute) instead of a per-tenant backend. Renders a narrow, collector-scoped Cilium egress rule. Set alongside the monitoring chart's `tracingCollector.backend=central`. The default per-tenant backend gives stronger isolation; see design-proposals/distributed-tracing in cozystack/community.",
22+
"type": "boolean",
23+
"default": false
24+
},
2025
"ingress": {
2126
"description": "Deploy own Ingress Controller.",
2227
"type": "boolean",

‎packages/apps/tenant/values.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ etcd: false
1111
## @param {bool} monitoring - Deploy own Monitoring Stack.
1212
monitoring: false
1313

14+
## @param {bool} tracingCentral - Opt into the shared-central tracing topology: allow this tenant's OTLP collector to export traces to the shared VictoriaTraces in tenant-root (account 0, distinguished by a `tenant` attribute) instead of a per-tenant backend. Renders a narrow, collector-scoped Cilium egress rule. Set alongside the monitoring chart's `tracingCollector.backend=central`. The default per-tenant backend gives stronger isolation; see design-proposals/distributed-tracing in cozystack/community.
15+
tracingCentral: false
16+
1417
## @param {bool} ingress - Deploy own Ingress Controller.
1518
ingress: false
1619

0 commit comments

Comments
 (0)