Skip to content

Commit 2f1bed8

Browse files
committed
docs(tenant-resource-metrics-api): fold the approving-review follow-ups
Correct the tenant Grafana network path (the egress allowlist does not admit the proxy's namespace, so add an allow rule or run it in an allow-listed one), guard the per-application boundary against a forged identity label (the scrape relabel must win over honorLabels) and test it, note the slash as well as the dots in the sanitized label keys, drop lookback from the out-of-bounds case now that the lookup window is fixed, and add an e2e test pinning the label-endpoint filter behaviour against the shipped vmselect. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
1 parent 867bb07 commit 2f1bed8

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

  • design-proposals/tenant-resource-metrics-api

‎design-proposals/tenant-resource-metrics-api/README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ The proxy serves two Kubernetes-shaped roots, with the native Prometheus API (`a
114114

115115
On the application root, the proxy additionally scopes the query to one application. The scope is defined by **what it returns**: the application's series within the namespace.
116116

117-
- **Where series carry the application's identity labels, it matches on them.** The labels are `apps_cozystack_io_application_{group,kind,name}` (the lineage keys with dots replaced by underscores), written directly on the series by the `scrapeClass` relabel; this is the core mechanism, not an add-on. Exporter series inside an application's pods get them by a scrape-time relabel from the pod's lineage labels (a `scrapeClass` on the cluster-wide and tenant `VMAgent`s). Note this is the series-label form; the pod/PVC lookup below instead reads kube-state-metrics' `kube_pod_labels`, where the same keys appear in KSM's prefixed `label_apps_cozystack_io_application_*` form. Node-level series such as cAdvisor and kube-state-metrics do not carry them yet; platform-side enrichment for those is part of the broader work.
117+
- **Where series carry the application's identity labels, it matches on them.** The labels are `apps_cozystack_io_application_{group,kind,name}` (the lineage keys with the slash and the dots replaced by underscores), written directly on the series by the `scrapeClass` relabel; this is the core mechanism, not an add-on. Exporter series inside an application's pods get them by a scrape-time relabel from the pod's lineage labels (a `scrapeClass` on the cluster-wide and tenant `VMAgent`s). The relabel must win over any exporter-supplied label: application scrapes must not set `honorLabels: true` (today only the ingress module does), or the scrape class must drop an exporter-supplied `apps_cozystack_io_application_*` before setting its own, so a pod cannot forge another application's identity. Note this is the series-label form; the pod/PVC lookup below instead reads kube-state-metrics' `kube_pod_labels`, where the same keys appear in KSM's prefixed `label_apps_cozystack_io_application_*` form. Node-level series such as cAdvisor and kube-state-metrics do not carry them yet; platform-side enrichment for those is part of the broader work.
118118
- **Until a series carries the labels, a pod/PVC lookup covers it, and becomes a fallback once it does.** The proxy resolves the application's pods and PVCs over the request window from `kube_pod_labels` / `kube_persistentvolumeclaim_labels` (via the label-values API, so pods gone before `end` are included) and ORs a `pod=~"^(<pods>)$"` branch and a `persistentvolumeclaim=~"^(<pvcs>)$"` branch, each also carrying `namespace="<ns>"`. An empty list omits its branch (never `pod=~"^()$"`, which would match series without a `pod` label). Lists are capped; above the cap the proxy answers `422` and asks for a shorter range.
119119
- **Kind versus plural.** Lineage labels carry the title-cased kind (`Postgres`); paths and RBAC use the plural (`postgreses`). The proxy maps one to the other through ApplicationDefinitions, in one place, reused by the label match and the lookup.
120120
- **No kind-specific branches.** A VM's host-side CPU, memory, and pod-network usage is covered by the pod branch through its `virt-launcher` pod; guest-view `kubevirt_vmi_*` series join the scope once they carry the identity labels. The proxy holds no per-kind knowledge.
@@ -138,7 +138,7 @@ The forced namespace matcher keeps reads safe, but a `namespace` label in the sh
138138

139139
### 8. Deployment
140140

141-
One small stateless Go service plus a Service, running in a platform namespace (`cozy-monitoring`). The dashboard's nginx routes the two roots to it ahead of `/apis`; a tenant's Grafana reaches it through the tenant namespace's allow-listed egress to `cozy-system`/platform services, as the dashboard already does for `vlselect`. No change to `cozystack-api`, and no new Kubernetes API surface.
141+
One small stateless Go service plus a Service. The dashboard's nginx (in `cozy-dashboard`) routes the two roots to it ahead of `/apis`, so the SPA needs no change to tenant egress. A tenant's Grafana, however, reaches it only if network policy allows it: the tenant egress allowlist (`packages/apps/tenant/templates/networkpolicy.yaml`) today admits `cozy-dashboard`, `cozy-linstor`, `cozy-keycloak`, `cozy-grafana-operator`, `cozy-kubevirt-cdi`, and `cozy-kubevirt`, but not the namespace the proxy runs in, so either add an allow rule for the proxy shaped like `allow-to-dashboard`, or run the proxy in a namespace that is already allow-listed. No change to `cozystack-api`, and no new Kubernetes API surface.
142142

143143
## User-facing changes
144144

@@ -156,7 +156,7 @@ One small stateless Go service plus a Service, running in a platform namespace (
156156

157157
- **Identity and policy are Kubernetes.** `TokenReview` and `SubjectAccessReview` on `apps.cozystack.io`/`cozystack.io`; the proxy stores no credentials of its own for callers and holds only its own ServiceAccount to reach `vmselect`.
158158
- **Forced namespace matcher** is the baseline isolation hinge: injected server-side into every selector and `match[]`, with the upstream request rebuilt from an allowlist (URL and POST form) so a caller cannot smuggle `extra_filters` / `extra_filters[]` / `extra_label` to widen scope.
159-
- **The application endpoint is a boundary for a narrowed grant.** A user whose grant is limited (via `resourceNames`) to one application cannot read another application's series in the same namespace, through `query`, `series`, or the label endpoints. A test asserts this.
159+
- **The application endpoint is a boundary for a narrowed grant.** A user whose grant is limited (via `resourceNames`) to one application cannot read another application's series in the same namespace, through `query`, `series`, or the label endpoints. This rests on the scrape-time relabel winning over exporter-supplied labels (the `honorLabels` note in 4a), so a pod cannot forge another application's identity label. A test asserts both: the narrowed grant cannot reach another application, and an exporter emitting another application's identity label stays out of that application's scope.
160160
- **Input.** `<plural>` must be a kind of `apps.cozystack.io`, `<name>` a DNS-1123 subdomain; all values entering filters are quoted and regex-escaped.
161161
- **Write-path trust** is a documented weakness of the shared store (Design 6): a tenant can write another namespace's label. It cannot leak data but can fabricate or inflate another tenant's series; the fix is a write-path control chosen in the broader work, and exposure timing is a rollout decision.
162162
- **Why not `services/proxy` to `vmselect`:** coarse authz, no forced matcher; it reads every tenant's series on the shared store.
@@ -169,7 +169,7 @@ One small stateless Go service plus a Service, running in a platform namespace (
169169
- Caller lacks the permission for the requested scope: 403 before any query runs.
170170
- Caller tries to drop or widen the namespace matcher (`extra_filters`, `extra_filters[]`, `extra_label`, extra `match[]`): impossible; the request is rebuilt from an allowlist with the matcher injected.
171171
- Application scope with no pods or PVCs: the empty branch is not sent, so it returns only the application's own labelled series, never the whole namespace.
172-
- Range, step, or lookback out of bounds: rejected by the cost limits (`422`).
172+
- Range or step out of bounds: rejected by the cost limits (`422`).
173173
- Token invalid or expired: `TokenReview` fails, 401.
174174

175175
## Testing
@@ -178,7 +178,7 @@ One small stateless Go service plus a Service, running in a platform namespace (
178178
- **Per-application boundary:** a user holding only a grant for one application (via `resourceNames`) cannot read another application's series in the same namespace through `query`, `series`, or the label endpoints.
179179
- **Application scope:** an application without PVCs (or without pods) does not see neighbour series (the empty branch is not sent); a `<name>` such as `a"} or {namespace=~".+` is rejected with `400`; an expression on a restarted application returns the series of its earlier pods within the lookback window.
180180
- **Integration:** `TokenReview` + `SubjectAccessReview` allow `<tenant>-view` in the tenant namespace and deny a foreign tenant; a POST query is authorized the same as a GET.
181-
- **e2e:** two tenants against the shared `tenant-root` store; tenant A reads only A's series; a VM/database graph renders end to end in the dashboard; a tenant's Grafana datasource against the proxy returns only that tenant's series.
181+
- **e2e:** two tenants against the shared `tenant-root` store; tenant A reads only A's series; a VM/database graph renders end to end in the dashboard; a tenant's Grafana datasource against the proxy returns only that tenant's series. An e2e test also pins the label-endpoint filter behaviour against the shipped `vmselect` version (it applies the injected filters on the label endpoints while `match[]` is present, which the `-search.ignoreExtraFiltersAtLabelsAPI` help text appears to contradict), so a version bump cannot widen label reads silently.
182182

183183
## Rollout
184184

0 commit comments

Comments
 (0)