Skip to content

Commit 3ede7c4

Browse files
authored
feat(observability-proxy): include platform exporter series in the namespace root (#5277)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Builds on #5219 and #5220, both merged. The namespace metrics root `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/...` now also returns series that a platform exporter attributes to `<ns>`. Example: the ClickHouse operator's exporter runs in `cozy-clickhouse-operator`. vmagent scrapes it without `honorLabels`, so its series carry `namespace="cozy-clickhouse-operator"`, and the tenant namespace the exporter reported moves to `exported_namespace="<ns>"`. Until now, any client of the namespace root (kubectl, automation, the console) never saw those series. ### Behavior changes - Namespace root, every endpoint (`query`, `query_range`, `series`, `labels`, `label/<name>/values`): - was: one proxy-set `extra_filters[]={namespace="<ns>"}`; - now: two values, `{namespace="<ns>"}` and `{namespace=~"cozy-.*",exported_namespace="<ns>"}`. vmselect ORs them. - Label APIs without a client `match[]`: the proxy already repeats the scope in `match[]` for `-search.ignoreExtraFiltersAtLabelsAPI`; it now repeats both values. vmselect ANDs every `match[]` with every `extra_filters[]`. Each cross pair is narrower than one of its halves, so the result is still exactly the union. - Client `extra_filters`, `extra_filters[]` and `extra_label` are still dropped. - New flag `--metrics-platform-namespace-prefix` (default `cozy-`) and chart value `observabilityProxy.metrics.platformNamespacePrefix`. An empty value turns the second branch off and restores the single-value scope. The proxy quotes the prefix and anchors it, so the value is matched literally and cannot become a wider regex. The flag accepts only namespace name characters, because any other value would match no namespace and silently drop the exporter series. - Exporter series keep `namespace="cozy-..."`. A query that also selects `namespace="<ns>"` excludes them; select them by `exported_namespace` or by the exporter's own labels (for ClickHouse, `chi`). - The branch is not specific to ClickHouse. For example, kube-state-metrics in the platform monitoring namespace sets `exported_namespace` from `metadata.namespace` on the Flux `gotk_resource_info` series (`packages/system/monitoring-agents/values.yaml`), so a namespace root now also returns the state of that namespace's own HelmReleases and Kustomizations. This is the namespace's own data, but it is a visible change with the default prefix. - Queries grouped `by (namespace)` can now return an extra group labelled with the exporter's platform namespace. The exporter's series also carry its own target labels (`pod`, `instance`, `job`) into the tenant's results. - An empty scope is refused with 500 instead of forwarding a request without `extra_filters[]`. - The application roots of #5220 are not affected: they replace the scope with their own selectors. ### Threat model 1. **Scrape path.** vmagent sets `namespace` from service discovery, while `exported_namespace` is whatever the exporter wrote. Any tenant pod can write a `namespace` label, which arrives as `exported_namespace`, so the extra branch only trusts it on series scraped from platform namespaces. Tenant roles (`packages/system/cozystack-basics/templates/clusterroles.yaml`) grant no access to `operator.victoriametrics.com` and no verbs on `Namespace`. A tenant can therefore neither run a target in `cozy-*` nor point a scrape object at one. 2. **Remote-write writers.** A tenant's nested Kubernetes cluster runs its own vmagent, which remote-writes into the same store when the tenant has no monitoring of its own, and the labels it writes are not enforced. Inside that cluster the tenant can create `cozy-*` namespaces or any other, so it can already write `namespace="<any ns>"` and pass the existing single filter. This is an existing limitation that this PR neither introduces nor widens: anything that can write `namespace="cozy-x",exported_namespace="<ns>"` can write `namespace="<ns>"` directly. Closing it needs enforcement on the write path and is out of scope here. 3. **Impact of a forged series.** The scope only adds series to a namespace's results; it never lets a caller read another namespace. A forged series can spoil or flood another namespace's graphs, up to the series limits that answer 422. ### Rejected alternatives - **Bare `{exported_namespace="<ns>"}`.** Every namespace is scraped by default, so any tenant pod could inject series into another namespace's root. - **Relabeling the ClickHouse operator's scrape so that `namespace` carries the tenant namespace.** It fixes one exporter at a time, each in its own chart, and it changes the label set of every existing series, which splits their history at the change and breaks any query that selects them by the operator's namespace. The proxy-side branch covers every platform exporter and leaves stored series as they are. - **Scraping the ClickHouse pods directly.** The metric names differ from the operator's exporter, so the shipped ClickHouse dashboard (`dashboards/clickhouse/altinity-clickhouse-operator-dashboard.json`), which reads the exporter's `chi_clickhouse_*` series, would have to be rewritten. - **A free regex flag instead of a prefix.** One typo (`.*`) would trust `exported_namespace` from every namespace. ### Tests - Table of namespace-root requests: exactly two `extra_filters[]` values in order with the default prefix, one value with an empty prefix, on all five endpoints. Client `extra_filters[]` still dropped. - A model of vmselect's selection (OR over `extra_filters[]`, AND with each `match[]`) shows that `namespace=cozy-x, exported_namespace=<ns>` is visible, and that `namespace=tenant-y, exported_namespace=<ns>` and `namespace=xcozy-a, ...` are not. - Escaping of `<ns>` in `exported_namespace`, literal prefix quoting, empty scope refused, flag validation, and a chart test that an empty value reaches the binary as an explicit empty flag. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `cmd/observability-proxy`, `internal/observabilityproxy` and the system package `packages/system/observability-proxy` (a new value and flag, no app, platform values, CRD, `hack/` or naming change). None of the triggers in `docs/agents/contributing.md` under "Downstream Repositories" apply. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(observability-proxy): the namespace metrics root also returns series that exporters in platform (`cozy-*`) namespaces attribute to the namespace through `exported_namespace`, such as the ClickHouse operator's. The prefix is set by `--metrics-platform-namespace-prefix` / `observabilityProxy.metrics.platformNamespacePrefix`; an empty value disables it. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Metrics queries now include series attributed to a tenant namespace by exporters in platform namespaces with the configured prefix. The default prefix is `cozy-`; setting it to an empty value disables this behavior. * Namespace scoping is applied across supported metrics endpoints, including label lookups. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b34f0c3 + b24a9b4 commit 3ede7c4

12 files changed

Lines changed: 341 additions & 21 deletions

File tree

‎cmd/observability-proxy/main.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ func parseFlags(args []string) (options, error) {
101101
fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 30*24*time.Hour, "how far before the window an application metrics query may read; one reading further back, or by an unknown amount, answers 422")
102102
fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422")
103103
fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422")
104+
fs.StringVar(&o.metrics.PlatformNamespacePrefix, "metrics-platform-namespace-prefix", "cozy-", "namespaces with this prefix run platform exporters, whose exported_namespace label adds series to that namespace's scope; empty disables")
104105
fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables")
105106
fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained client requests per second per user; an application metrics request makes three store calls (pod lookup, PVC lookup, query); 0 disables")
106107
fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user")
@@ -121,6 +122,11 @@ func parseFlags(args []string) (options, error) {
121122
return o, fmt.Errorf("invalid metrics store name %q", store)
122123
}
123124
}
125+
// The prefix is matched literally, so one no namespace name can start
126+
// with would turn the exporter branch off without a word.
127+
if p := o.metrics.PlatformNamespacePrefix; p != "" && len(validation.IsDNS1123Label(p+"x")) > 0 {
128+
return o, fmt.Errorf("invalid metrics platform namespace prefix %q", p)
129+
}
124130
switch {
125131
case o.rootTenant == "":
126132
return o, errors.New("--root-tenant must not be empty")

‎cmd/observability-proxy/main_test.go‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,10 @@ func TestParseFlagsDefaults(t *testing.T) {
4646
if m.RootTenant != o.rootTenant {
4747
t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant)
4848
}
49+
// The prefix every platform namespace carries.
50+
if m.PlatformNamespacePrefix != "cozy-" {
51+
t.Errorf("platform namespace prefix = %q, want cozy-", m.PlatformNamespacePrefix)
52+
}
4953
}
5054

5155
func TestParseFlagsOverrides(t *testing.T) {
@@ -67,16 +71,23 @@ func TestParseFlagsMetricsOverrides(t *testing.T) {
6771
"--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow",
6872
"--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s",
6973
"--metrics-app-lookback=2h", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3",
74+
"--metrics-platform-namespace-prefix=sys-",
7075
})
7176
if err != nil {
7277
t.Fatal(err)
7378
}
7479
m := o.metrics
7580
if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour ||
7681
m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second ||
77-
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 {
82+
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 ||
83+
m.PlatformNamespacePrefix != "sys-" {
7884
t.Errorf("overrides not applied: %+v", m)
7985
}
86+
// An empty prefix switches the exporter branch of the scope off.
87+
o, err = parseFlags([]string{"--metrics-platform-namespace-prefix="})
88+
if err != nil || o.metrics.PlatformNamespacePrefix != "" {
89+
t.Errorf("empty platform namespace prefix: %q, %v", o.metrics.PlatformNamespacePrefix, err)
90+
}
8091
}
8192

8293
func TestParseFlagsInvalid(t *testing.T) {
@@ -112,6 +123,10 @@ func TestParseFlagsInvalid(t *testing.T) {
112123
{"--metrics-app-lookback=0s"},
113124
{"--metrics-app-max-pods=0"},
114125
{"--metrics-app-max-pvcs=0"},
126+
// No namespace name starts with these, so they would match nothing.
127+
{"--metrics-platform-namespace-prefix=Cozy-"},
128+
{"--metrics-platform-namespace-prefix=cozy.*"},
129+
{"--metrics-platform-namespace-prefix=-"},
115130
{"--unknown"},
116131
// The proxy would cut the request while vmselect is still within
117132
// the timeout it was given.

‎internal/observabilityproxy/metricsparams.go‎

Lines changed: 37 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import (
2222
"net/http"
2323
"net/url"
2424
"regexp"
25+
"slices"
2526
"strconv"
2627
"strings"
2728
"time"
@@ -47,6 +48,9 @@ type MetricsConfig struct {
4748
ShortTermRetention time.Duration
4849
Limits MetricsLimits
4950
App AppScopeLimits
51+
// PlatformNamespacePrefix names the namespaces whose exporters may
52+
// attribute a series to another namespace; empty trusts none.
53+
PlatformNamespacePrefix string
5054
// UpstreamURL maps a store name to its Prometheus API base URL, without
5155
// a trailing slash; nil means the in-cluster vmselect Service.
5256
UpstreamURL func(store string) string
@@ -103,15 +107,37 @@ func namespaceSelector(ns string) string {
103107
return "{namespace=" + strconv.Quote(ns) + "}"
104108
}
105109

110+
// namespaceScope selects the series of namespace ns. Besides its own, it
111+
// takes the series an exporter in a platform namespace attributes to ns:
112+
// vmagent sets namespace to the scraped target's and, without honorLabels,
113+
// moves the exporter's own namespace label to exported_namespace. Any pod
114+
// can write that label, so it is trusted only where the platform's vmagent
115+
// set namespace to one with the platform prefix, in which tenants cannot run
116+
// pods or scrape objects. Series remote-written from a tenant's nested
117+
// cluster carry whatever namespace that cluster chose, so they are outside
118+
// this guarantee for both values alike.
119+
func (c MetricsConfig) namespaceScope(ns string) []string {
120+
scope := []string{namespaceSelector(ns)}
121+
if c.PlatformNamespacePrefix != "" {
122+
// MetricsQL anchors =~ at both ends.
123+
platform := strconv.Quote(regexp.QuoteMeta(c.PlatformNamespacePrefix) + ".*")
124+
scope = append(scope, "{namespace=~"+platform+",exported_namespace="+strconv.Quote(ns)+"}")
125+
}
126+
return scope
127+
}
128+
106129
// BuildMetricsParams rebuilds the upstream parameters from an allowlist and
107-
// adds the namespace scope as the only extra_filters[] value. vmselect ORs
108-
// repeated extra_filters, so a client value would widen the scope; it ANDs
109-
// the single one into every selector of the query and every match[], which
110-
// is what lets the query pass through unparsed.
111-
func BuildMetricsParams(endpoint string, in url.Values, namespace string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) {
130+
// sends scope as the only extra_filters[] values. vmselect ORs repeated
131+
// extra_filters, so a client value would widen the scope; it ANDs each into
132+
// every selector of the query and every match[], which is what lets the
133+
// query pass through unparsed.
134+
func BuildMetricsParams(endpoint string, in url.Values, scope []string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) {
112135
req := metricsRequest{path: endpoint, params: url.Values{}}
136+
if len(scope) == 0 {
137+
return req, errorf(http.StatusInternalServerError, "no metrics scope")
138+
}
113139
out := req.params
114-
out.Set("extra_filters[]", namespaceSelector(namespace))
140+
out["extra_filters[]"] = slices.Clone(scope)
115141
timeout := lim.MaxTimeout
116142
if v := in.Get("timeout"); v != "" {
117143
d, herr := parseMetricsDuration("timeout", v)
@@ -226,8 +252,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
226252
}
227253
// vmselect started with -search.ignoreExtraFiltersAtLabelsAPI drops
228254
// extra_filters[] on the label APIs unless a match[] is present
229-
// (getCommonParamsInternal), so the scope is repeated here.
230-
out.Set("match[]", namespaceSelector(namespace))
255+
// (getCommonParamsInternal), so the scope is repeated here. Where both
256+
// are sent, vmselect ANDs every match[] with every extra_filters[];
257+
// a cross pair is narrower than one of its halves, so the union is
258+
// still the scope.
259+
out["match[]"] = slices.Clone(scope)
231260
req.scopedMatch = true
232261
}
233262
return req, nil

‎internal/observabilityproxy/metricsparams_test.go‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ package observabilityproxy
1919
import (
2020
"net/http"
2121
"net/url"
22+
"slices"
2223
"strconv"
2324
"strings"
2425
"testing"
@@ -29,21 +30,28 @@ func testMetricsLimits() MetricsLimits {
2930
return MetricsLimits{MaxRange: 30 * 24 * time.Hour, MaxPoints: 11000, MaxTimeout: 30 * time.Second}
3031
}
3132

32-
const wantMetricsScope = `{namespace="tenant-a"}`
33+
// wantMetricsScope is the namespace root's scope under the default
34+
// platform prefix: the namespace's own series, then the series a platform
35+
// exporter attributes to it.
36+
var wantMetricsScope = []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`}
37+
38+
func testScope() []string {
39+
return MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope("tenant-a")
40+
}
3341

3442
func unix(t time.Time) string { return strconv.FormatInt(t.Unix(), 10) }
3543

3644
func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest {
3745
t.Helper()
38-
req, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits())
46+
req, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits())
3947
if err != nil {
4048
t.Fatalf("BuildMetricsParams(%s, %v): %v", endpoint, in, err)
4149
}
4250
return req
4351
}
4452

4553
func metricsStatus(endpoint string, in url.Values) int {
46-
_, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits())
54+
_, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits())
4755
if err == nil {
4856
return http.StatusOK
4957
}
@@ -84,8 +92,8 @@ func TestBuildMetricsParamsAllowlist(t *testing.T) {
8492
t.Errorf("parameter %q forwarded: %v", k, req.params)
8593
}
8694
}
87-
if got := req.params["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope {
88-
t.Errorf("extra_filters[] = %q, want exactly [%s]", got, wantMetricsScope)
95+
if got := req.params["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) {
96+
t.Errorf("extra_filters[] = %q, want exactly %q", got, wantMetricsScope)
8997
}
9098
for _, m := range req.params["match[]"] {
9199
if strings.Contains(m, "tenant-b") {
@@ -116,8 +124,8 @@ func TestNamespaceSelectorEscapes(t *testing.T) {
116124
func TestBuildMetricsParamsLabelEndpointsAlwaysMatch(t *testing.T) {
117125
for _, endpoint := range []string{"api/v1/labels", "api/v1/label/__name__/values"} {
118126
req := buildMetrics(t, endpoint, url.Values{})
119-
if got := req.params["match[]"]; len(got) != 1 || got[0] != wantMetricsScope {
120-
t.Errorf("%s without match[] sends %q, want the namespace selector", endpoint, got)
127+
if got := req.params["match[]"]; !slices.Equal(got, wantMetricsScope) {
128+
t.Errorf("%s without match[] sends %q, want the namespace scope %q", endpoint, got, wantMetricsScope)
121129
}
122130
req = buildMetrics(t, endpoint, url.Values{"match[]": {"up", `{job="x"}`}})
123131
if got := req.params["match[]"]; len(got) != 2 || got[0] != "up" || got[1] != `{job="x"}` {
@@ -198,7 +206,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) {
198206
})
199207
want := url.Values{
200208
"query": {"up"}, "start": {"2026-11-04T11:00:00.250Z"}, "end": {"2026-11-04T12:00:00.000Z"}, "step": {"90"},
201-
"timeout": {"30"}, "extra_filters[]": {wantMetricsScope},
209+
"timeout": {"30"}, "extra_filters[]": wantMetricsScope,
202210
}
203211
if req.params.Encode() != want.Encode() {
204212
t.Errorf("params = %v\nwant %v", req.params, want)

0 commit comments

Comments
 (0)