You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4705ae0
Browse filesBrowse the repository at this point in the historyBrowse files
docs(tenant-application-logs): move Merge_Log_Key to Phase 1 and tighten the read-path bounds
Per review: move the Merge_Log_Key forgery hardening into Phase 1 ahead of proxy exposure, since it is a read-isolation prerequisite independent of the lineage labels and application scope. Require a finite, clamped time window for query/hits/stats_query_range (VictoriaLogs otherwise scans the whole retention), name the per-endpoint response formats instead of a blanket NDJSON, map plural to Kind via ApplicationDefinitions, narrow the application-scoped post-mortem guarantee for windows predating the lineage labels (the namespace endpoint still serves them), note the monitoring-ownership-change store gap, state the bounded tail resume honestly, soften the (_stream_id,_time) dedup claim to at-least-once, and mark the Merge_Log_Key field move as a breaking field-schema change.
Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
Copy file name to clipboardExpand all lines: design-proposals/tenant-application-logs/README.md
+14-12Lines changed: 14 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -49,7 +49,7 @@ Empty while the proposal is still intent. -->
49
49
## Goals
50
50
51
51
- A tenant can read the logs of a resource it owns, over a chosen time window, and follow them live, rendered in the dashboard, with the live tail surviving past 60 seconds.
52
-
- Logs of a deleted pod or application remain readable (post-mortem) as long as they are within VictoriaLogs retention.
52
+
- Logs of a deleted pod or application remain readable (post-mortem) as long as they are within VictoriaLogs retention, through the namespace endpoint; application-scoped post-mortem additionally needs the §6 lineage labels or a resolvable pod, so for windows older than metrics retention and predating §6 it falls back to the namespace endpoint.
53
53
- A tenant can never read another namespace's logs, enforced server-side by a forced scope the client cannot override.
54
54
- The dashboard consumes the native VictoriaLogs NDJSON with a standard client, with no Kubernetes envelope around the stream.
55
55
- Minimal implementation: the same proxy as metrics, no `cozystack-api` change.
@@ -72,7 +72,7 @@ GET /namespaces/<ns>/log/select/logsql/{query,hits,stats_query_range,tail}
72
72
GET /namespaces/<ns>/<plural>/<name>/log/select/logsql/{query,hits,stats_query_range,tail}
73
73
```
74
74
75
-
The client sends LogsQL; the proxy adds the scope and returns VictoriaLogs' native NDJSON unchanged, flushing after every line on `tail`. No `cozystack-api` change, no new API group served, no Kubernetes envelope. Metrics live under the sibling prefix `.../metrics/api/v1/...` of the same scope.
75
+
The client sends LogsQL; the proxy adds the scope and returns VictoriaLogs' native response unchanged, with each endpoint's own content type: `query` and `tail` stream JSON lines, `hits` returns a JSON object, and `stats_query_range` returns Prometheus-compatible matrix JSON. On `tail` the proxy flushes after every line. No `cozystack-api` change, no new API group served, no Kubernetes envelope. Metrics live under the sibling prefix `.../metrics/api/v1/...` of the same scope.
76
76
77
77
```mermaid
78
78
flowchart TD
@@ -120,28 +120,29 @@ extra_filters=(cozystack_app_kind:=<Kind> cozystack_app_name:=<name>) or kuberne
120
120
```
121
121
122
122
-`<pods>` are the application's pods over the request window, from the label values API on `kube_pod_labels` (same lookup as the metrics application scope), capped. The pod branch covers lines collected before the §6 change; an empty list is omitted.
123
-
- The `<pods>` lookup reads `tenant-root`'s VictoriaMetrics even when the logs live in a tenant `VLCluster`; for windows beyond metrics retention only the `cozystack_app_*` branch applies.
124
-
-`<plural>` must be a kind of `apps.cozystack.io`, `<name>` a DNS-1123 subdomain; values are quoted. Anything else is `400`.
123
+
- The `<pods>` lookup reads `tenant-root`'s VictoriaMetrics even when the logs live in a tenant `VLCluster`; for windows beyond metrics retention only the `cozystack_app_*` branch applies. So application-scoped post-mortem has a bounded reach: a line written before the §6 labels existed and older than metrics retention carries neither `cozystack_app_*` nor a resolvable pod, and the application endpoint cannot select it. Such lines stay reachable through the namespace endpoint; the post-mortem guarantee is narrowed accordingly (see Goals and Failure cases).
124
+
-`<plural>` must be a kind of `apps.cozystack.io`, `<name>` a DNS-1123 subdomain; values are quoted. Anything else is `400`. The proxy maps `<plural>` to the lineage `Kind` by reading the ApplicationDefinitions (the same mapping as #94's application scope) and uses the mapped value in `cozystack_app_kind`, never the raw plural.
125
125
- The upstream request is built from scratch: only allowlisted parameters (`query`, `start`, `end`, `limit` (clamped), `step`, and on `tail` only `since` or `start_offset`, both capped) and no client headers. VictoriaLogs takes the tenant from `AccountID` / `ProjectID` headers; the proxy always sets them. Client `extra_filters` / `extra_stream_filters` are dropped. `/select/tenant_ids` and every non-allowlisted path are not proxied.
126
+
-**Bounded time window.** For `query`, `hits`, and `stats_query_range` the proxy requires or derives a finite `[start, end]` and clamps the maximum span; VictoriaLogs otherwise uses the oldest stored timestamp when `start` is omitted and would scan the whole retention. Inverted ranges are rejected with `400`. The `tail` resume cap (§5) is a separate bound and does not restrict a client's `start`/`end` on the other endpoints.
126
127
- Queries using `options(ignore_global_time_filter=true)` are rejected, detected by a plain substring match on `ignore_global_time_filter` anywhere in `query`, subqueries included: it disables the time bounds the proxy relies on, and on `tail` it rescans the whole retention every tick.
127
128
- Platform sources are out of reach: kube-apiserver audit and Kubernetes events carry no `kubernetes_namespace_name` in the current pipeline, so the stream filter excludes them.
128
129
129
130
### 4. Store resolution
130
131
131
-
As in #94: `namespace.cozystack.io/monitoring` names the `VLCluster`; empty means `tenant-root`. Logs of a tenant with its own monitoring are moved there, not copied, so exactly one store answers.
132
+
As in #94: `namespace.cozystack.io/monitoring` names the `VLCluster`; empty means `tenant-root`. Logs of a tenant with its own monitoring are moved there, not copied, so exactly one store answers. If the label changes during the retention window, records already written to the previous `VLCluster` are not copied, so logs from before the change are not visible through the new store; routing should be treated as immutable for the retention period, or the proxy must keep routing history and query both stores (open question).
132
133
133
134
### 5. Live tail
134
135
135
136
-`tail` is VictoriaLogs live tailing with the scope applied. VictoriaLogs rejects pipes that compute stats, reorder, or drop lines; the proxy passes its `400` through.
136
-
-`tail` has no absolute start: it starts at `now - offset - start_offset`. To resume after a disconnect the client sends `since=<last _time>`; the proxy sets `start_offset = (now - offset) - since`, capped (proposed: 15 min). Client `start`/`end` on `tail` are ANDed into each tick as usual.
137
-
- Lines around the resume point may repeat; clients deduplicate by `(_stream_id, _time)`. Queries whose pipes drop these fields (`fields`, `delete`) get no deduplication guarantee.
137
+
-`tail` has no absolute start: it starts at `now - offset - start_offset`. To resume after a disconnect the client sends `since=<last _time>`; the proxy sets `start_offset = (now - offset) - since`, capped (proposed: 15 min). A reconnect later than the cap resumes at the cap, not at `since`: the older gap is not replayed on `tail` and needs a separate historical `query`. The proxy does not promise gap-free resume beyond the cap. Client `start`/`end` on `tail` are ANDed into each tick as usual.
138
+
- Lines around the resume point may repeat. Delivery is at-least-once; VictoriaLogs does not define `(_stream_id, _time)` as a per-entry identity (entries in one stream can share a `_time`), so a client dedup on that pair is best-effort at the boundary, not a uniqueness guarantee, and a stable per-entry cursor is needed if exactly-once matters. Queries whose pipes drop these fields (`fields`, `delete`) get no deduplication at all.
138
139
-`tail` goes through the proxy only, so there is no request-timeout cut. The proxy caps open tails per user and in total and closes a tail after a configurable idle period.
139
140
140
141
### 6. Filtering to an application, and the write-path hardening (collection change)
141
142
142
143
Fluent Bit keeps the lineage labels `apps.cozystack.io/application.{group,kind,name}` as `cozystack_app_group`, `cozystack_app_kind`, `cozystack_app_name` (a second `nest` lift on `kubernetes_labels`, renamed before `Remove_wildcard kubernetes_`, like the three kept fields today) and adds them to `_stream_fields`. These give the application scope of §3 its labels. The webhook sets these labels on pods whose owning application it resolves.
143
144
144
-
**Hardening.**`Merge_Log On` adds the JSON keys of an application's own log line to the top level of the record, ahead of the filter's own `kubernetes` map and without deduplication. A line such as `{"kubernetes":{"namespace_name":"<other>"}}` therefore yields a second `kubernetes` map; the `nest` lift lifts every map under that key, so a forged `kubernetes_namespace_name` (and `kubernetes_labels`) sits next to the real one, first, and `rewrite_tag` routes on it. `log_source`, `tenant`, and `cluster` can be forged the same way, since `Add` never overwrites. Set `Merge_Log_Key` (for example `Merge_Log_Key log_processed`) so a parsed line lands under one key and cannot add top-level keys; this also covers `K8S-Logging.Parser`. Trade-off: fields of JSON log lines searched at the top level today (for example `level`) become `log_processed.<field>` in VictoriaLogs; nothing in the current pipeline or charts reads them. A test asserts that a crafted line cannot appear under another namespace or application.
145
+
**Hardening.** `Merge_Log On` adds the JSON keys of an application's own log line to the top level of the record, ahead of the filter's own `kubernetes` map and without deduplication. A line such as `{"kubernetes":{"namespace_name":"<other>"}}` therefore yields a second `kubernetes` map; the `nest` lift lifts every map under that key, so a forged `kubernetes_namespace_name` (and `kubernetes_labels`) sits next to the real one, first, and `rewrite_tag` routes on it. `log_source`, `tenant`, and `cluster` can be forged the same way, since `Add` never overwrites. Set `Merge_Log_Key` (for example `Merge_Log_Key log_processed`) so a parsed line lands under one key and cannot add top-level keys; this also covers `K8S-Logging.Parser`. This hardening is a read-isolation prerequisite and lands in Phase 1, before the proxy is exposed; it does not depend on the lineage labels or the application scope (which are Phase 2), because without it any pod can forge `kubernetes_namespace_name` and surface in another namespace's view that Phase 1 already serves. Trade-off, a breaking field-schema change: fields of JSON log lines searched at the top level today (for example `level`) become `log_processed.<field>` in VictoriaLogs; in-repo charts do not read them, but any external dashboard or LogsQL query that reads a top-level field must move to `log_processed.<field>` or run a dual-read during migration. A test asserts that a crafted line cannot appear under another namespace or application.
145
146
146
147
## User-facing changes
147
148
@@ -166,7 +167,7 @@ Fluent Bit keeps the lineage labels `apps.cozystack.io/application.{group,kind,n
166
167
## Failure and edge cases
167
168
168
169
- Namespace has no dedicated store (empty label): resolves to `tenant-root` where the logs are; the common case, not a failure.
169
-
- Target resource deleted: logs still stream (post-mortem) within retention; the SAR passes on permission, not existence; the application scope matches by `cozystack_app_*` labels and the earlier pods' names.
170
+
- Target resource deleted: logs still stream (post-mortem) within retention; the SAR passes on permission, not existence; the application scope matches by `cozystack_app_*` labels and the earlier pods' names. For a window older than metrics retention and predating the §6 labels, the application endpoint cannot select those lines, but the namespace endpoint still returns them.
170
171
- Caller lacks `get <res>/log` in `observability.cozystack.io`: 403 before any query.
171
172
- Client tries to drop or widen the scope (`extra_filters`, `extra_stream_filters`, `AccountID` header, a `| union` / `:in(...)` subquery, `options(ignore_global_time_filter=true)`): impossible; the request is rebuilt from an allowlist and such queries are rejected.
172
173
-`tail` resume with `since`: returns the lines after the cut, with duplicates only at the boundary.
@@ -179,12 +180,13 @@ Fluent Bit keeps the lineage labels `apps.cozystack.io/application.{group,kind,n
179
180
- A `<name>` with LogsQL metacharacters is rejected with `400`.
180
181
- A crafted JSON line from a host pod carrying another namespace's `kubernetes_namespace_name` does not show up under that namespace.
181
182
- Logs of a deleted application, and of a restarted pod, are readable through the application scope within retention.
182
-
- Resume with `since` returns the lines after the cut, with duplicates only at the boundary.
183
+
- A query with an omitted, over-wide, or inverted `start`/`end` is clamped or rejected rather than scanning the whole retention.
184
+
- Resume with `since` within the cap returns the lines after the cut, with duplicates only at the boundary; a resume older than the cap starts at the cap and does not replay the older gap.
183
185
184
186
## Rollout
185
187
186
-
-**Phase 1: the proxy endpoints (namespace scope) and the RBAC rule.** Serve `/namespaces/<ns>/log/...` on the shared proxy with the forced `extra_stream_filters`, and ship the aggregated `*/log` rule at `use`. Land the write-path control before exposing the proxy.
187
-
-**Phase 2: the collection change (§6) and the application scope.** The Fluent Bit lineage labels, `Merge_Log_Key` hardening, and the `/namespaces/<ns>/<plural>/<name>/log/...` endpoints.
188
+
-**Phase 1: the proxy endpoints (namespace scope), the RBAC rule, and the `Merge_Log_Key` hardening.** Serve `/namespaces/<ns>/log/...` on the shared proxy with the forced `extra_stream_filters`, ship the aggregated `*/log` rule at `use`, and set `Merge_Log_Key` in Fluent Bit before exposing the proxy. `Merge_Log_Key` is a read-isolation prerequisite (without it a crafted line forges `kubernetes_namespace_name` and surfaces in another namespace's view) and depends on neither the lineage labels nor the application scope, so it belongs here, not in Phase 2. Land the guest-cluster write-path control in the same phase.
189
+
-**Phase 2: the lineage labels (§6) and the application scope.** The Fluent Bit `cozystack_app_*` labels and the `/namespaces/<ns>/<plural>/<name>/log/...` endpoints.
188
190
-**Phase 3: the dashboard log viewer** with live tail.
0 commit comments