Skip to content

Commit c1ba055

Browse files
authored
feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does Makes Fluent Bit keep the pod lineage labels `apps.cozystack.io/application.{group,kind,name}` on container log lines, as the stream fields `apps_cozystack_io_application_{group,kind,name}`, for both the default container-log output and the per-tenant routed output. Why: the per-application logs scope in the design proposal [Tenant application logs](cozystack/community#95) (section 6) filters by these stream fields. Today the kubernetes filter fetches pod labels, but they are dropped together with the rest of the `kubernetes_*` keys before output. What changes in the pipeline: - A second `nest` lift of the pod labels, then a rename of the three lineage keys to their final names before the existing `Remove_wildcard kubernetes_` runs. All other labels are still removed. Every lifted key is prefixed, so no other label can collide into the identity names. - The three keys are added to `_stream_fields` of both container-log outputs. Events and audit pipelines are unchanged. - A parsed JSON log line is kept under `log_processed` (already on `main`), so a line's content cannot create the identity keys. - Guest Kubernetes clusters are not affected: their monitoring-agents release replaces the filters and outputs wholesale. A regression test in `packages/apps/kubernetes` pins that guest streams carry no identity fields. Effect on existing streams: VictoriaLogs treats a missing stream field like an empty one, and the rename writes nothing when the label is absent. Pods without lineage labels keep exactly the same stream. Pods with lineage labels get a new stream once after the rollout; their earlier lines stay reachable by namespace. #### How it was tested - helm-unittest for `monitoring-agents` (lift order, rename before `Remove_wildcard`, stream fields on both outputs, events and audit untouched) and for `kubernetes` (guest streams unchanged). The events/audit guard was mutation-checked. - Verified on a test cluster: real pod log lines arrive, Redis pods' lines carry the three `apps_cozystack_io_application_*` stream fields, and Fluent Bit logs show no errors. #### Known limitation Guest clusters write to the parent log store without authentication, so a guest administrator can forge these stream fields as well as the namespace field, within the store their tenant shares. This is the write-path trust risk already documented in the proposal. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff changes `packages/system/monitoring-agents/values.yaml` (a system package with no schema) and adds tests; the file under `packages/apps/kubernetes` is a test only, so the app's `values.yaml` and `values.schema.json` are unchanged. ### Release note ```release-note feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Container-log streams now include application group, kind, and name metadata for root and tenant stores. * Guest monitoring streams include pod name, container name, and namespace fields. * **Bug Fixes** * Application metadata is added to container-log streams without changing the existing stream fields for event and audit outputs. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents e2283b3 + 95dc591 commit c1ba055

3 files changed

Lines changed: 123 additions & 2 deletions

File tree

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
suite: guest fluent-bit never ships application identity stream fields
2+
3+
templates:
4+
- templates/helmreleases/monitoring-agents.yaml
5+
6+
values:
7+
- values/common.yaml
8+
9+
release:
10+
name: kubernetes-test
11+
namespace: tenant-test
12+
13+
# Pods in a guest cluster are tenant-controlled, so their
14+
# apps.cozystack.io/application.* labels are not a trustworthy application
15+
# identity, and the platform must not promote them to the stream fields the
16+
# parent scopes application logs by. The guest release replaces the
17+
# monitoring-agents filters and outputs, so it is the guest overrides that keep
18+
# the parent's label lift out. This does not stop a guest admin who edits the
19+
# guest's own fluent-bit or writes to the parent's vlinsert directly; that is
20+
# the write-path control the logs proposal leaves open.
21+
22+
tests:
23+
- it: does not lift pod labels or declare lineage stream fields in the guest
24+
set:
25+
addons.monitoringAgents.enabled: true
26+
_namespace.monitoring: tenant-test
27+
asserts:
28+
- isKind:
29+
of: HelmRelease
30+
- notMatchRegex:
31+
path: spec.values.fluent-bit.config.filters
32+
pattern: 'kubernetes_labels|apps_cozystack_io_application_'
33+
- notMatchRegex:
34+
path: spec.values.fluent-bit.config.outputs
35+
pattern: 'apps_cozystack_io_application_'
36+
- matchRegex:
37+
path: spec.values.fluent-bit.config.outputs
38+
pattern: '_stream_fields=stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&'
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
suite: fluent-bit keeps the application identity of a container log line
2+
3+
# The lineage webhook labels a pod with apps.cozystack.io/application.{group,
4+
# kind,name}. A container log line keeps them as top-level
5+
# apps_cozystack_io_application_{group,kind,name}, the names the metrics side
6+
# uses, and they are stream fields so a reader can scope to one application
7+
# with extra_stream_filters. The kubernetes map, labels included, is otherwise
8+
# dropped by Remove_wildcard kubernetes_, so the three keys have to be lifted
9+
# and renamed out of the kubernetes_ prefix before that filter runs.
10+
templates:
11+
- charts/fluent-bit/templates/configmap.yaml
12+
release:
13+
name: monitoring-agents
14+
namespace: cozy-monitoring
15+
set:
16+
global:
17+
target: tenant-root
18+
kubernetesProvider:
19+
scheme:
20+
"v1/Namespace":
21+
gvr:
22+
version: v1
23+
resource: namespaces
24+
namespaced: false
25+
objects:
26+
- kind: Namespace
27+
apiVersion: v1
28+
metadata:
29+
name: tenant-a
30+
labels:
31+
namespace.cozystack.io/monitoring: tenant-a
32+
tests:
33+
- it: lifts the pod labels after the kubernetes map is lifted
34+
asserts:
35+
- matchRegex:
36+
path: data["fluent-bit.conf"]
37+
pattern: '(?s)Operation lift\n\s+Nested_under kubernetes\n.*Name nest\n\s+Match kube\.\*\n\s+Operation lift\n\s+Nested_under kubernetes_labels\n\s+Add_prefix kubernetes_labels_\n'
38+
39+
- it: renames the three lineage labels out of the kubernetes_ prefix before Remove_wildcard
40+
asserts:
41+
- matchRegex:
42+
path: data["fluent-bit.conf"]
43+
pattern: '(?s)Nested_under kubernetes_labels\n.*Rename kubernetes_labels_apps\.cozystack\.io/application\.group apps_cozystack_io_application_group\n\s+Rename kubernetes_labels_apps\.cozystack\.io/application\.kind apps_cozystack_io_application_kind\n\s+Rename kubernetes_labels_apps\.cozystack\.io/application\.name apps_cozystack_io_application_name\n.*Remove_wildcard kubernetes_\n'
44+
45+
- it: makes the lineage keys stream fields on the default container-log output
46+
asserts:
47+
- matchRegex:
48+
path: data["fluent-bit.conf"]
49+
pattern: 'Match kube\.\*\n\s+Host vlinsert-generic\.tenant-root\.svc\n\s+port 9481\n\s+compress gzip\n\s+uri /insert/jsonline\?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&'
50+
51+
- it: makes the lineage keys stream fields on a routed tenant's container-log output
52+
asserts:
53+
- matchRegex:
54+
path: data["fluent-bit.conf"]
55+
pattern: 'Match tenant\.tenant-a\.\*\n\s+Host vlinsert-generic\.tenant-a\.svc\n\s+port 9481\n\s+compress gzip\n\s+uri /insert/jsonline\?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&'
56+
57+
- it: leaves the events and audit outputs without lineage stream fields
58+
asserts:
59+
- matchRegex:
60+
path: data["fluent-bit.conf"]
61+
pattern: 'uri /insert/jsonline\?_stream_fields=log_source,reason,metadata_namespace,metadata_name&'
62+
- matchRegex:
63+
path: data["fluent-bit.conf"]
64+
pattern: 'uri /insert/jsonline\?_stream_fields=log_source,stage,user_username,verb,requestUri&'
65+
- notMatchRegex:
66+
path: data["fluent-bit.conf"]
67+
pattern: 'Match (events\.\*|audit\.\*|\*)\n[^\[]*Nested_under kubernetes_labels'

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -411,7 +411,7 @@ fluent-bit:
411411
Host vlinsert-generic.{{ .Values.global.target }}.svc
412412
port 9481
413413
compress gzip
414-
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
414+
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&_msg_field=log&_time_field=date
415415
format json_lines
416416
json_date_format iso8601
417417
header AccountID 0
@@ -445,7 +445,7 @@ fluent-bit:
445445
Host vlinsert-generic.{{ $target }}.svc
446446
port 9481
447447
compress gzip
448-
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
448+
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&_msg_field=log&_time_field=date
449449
format json_lines
450450
json_date_format iso8601
451451
header AccountID 0
@@ -467,6 +467,19 @@ fluent-bit:
467467
Operation lift
468468
Nested_under kubernetes
469469
Add_prefix kubernetes_
470+
{{- /*
471+
The pod's lineage labels name the application it belongs to. lift
472+
cannot pick keys, so every label comes up; all but these three keep
473+
the kubernetes_ prefix and fall to Remove_wildcard below. An absent
474+
label stays absent, and VictoriaLogs builds _stream from the stream
475+
fields that are present, so an unlabelled pod keeps its stream.
476+
*/}}
477+
[FILTER]
478+
Name nest
479+
Match kube.*
480+
Operation lift
481+
Nested_under kubernetes_labels
482+
Add_prefix kubernetes_labels_
470483
[FILTER]
471484
Name modify
472485
Match kube.*
@@ -507,6 +520,9 @@ fluent-bit:
507520
Rename kubernetes_pod_name keep_kubernetes_pod_name
508521
Rename kubernetes_namespace_name keep_kubernetes_namespace_name
509522
Rename kubernetes_container_name keep_kubernetes_container_name
523+
Rename kubernetes_labels_apps.cozystack.io/application.group apps_cozystack_io_application_group
524+
Rename kubernetes_labels_apps.cozystack.io/application.kind apps_cozystack_io_application_kind
525+
Rename kubernetes_labels_apps.cozystack.io/application.name apps_cozystack_io_application_name
510526
[FILTER]
511527
Name modify
512528
Match kube.*

0 commit comments

Comments
 (0)