Proposal: Integrate Coroot as a zero-instrumentation (eBPF) observability option for Cozystack #3028

Summary

I'd like to propose adding Coroot as a
first-class observability option in Cozystack. Coroot is an open-source (Apache-2.0),
eBPF-based observability platform that provides a service map, RED metrics, logs,
distributed traces, continuous profiling, SLOs, and cost monitoring — with
zero application instrumentation. Because it auto-discovers workloads and
generates telemetry from kernel-level events, it fits Cozystack's "batteries-included,
no manual wiring" philosophy well.

Crucially, Coroot does not have to bring its own parallel stack: Cozystack already
ships the backends Coroot needs — victoria-metrics-operator (Prometheus-compatible
TSDB) and clickhouse-operator (trace/log/profile store) — so an integration can lean
on existing platform components instead of duplicating them.

Motivation

Cozystack's current observability stack (packages/system/monitoring,
monitoring-agents, grafana-operator, victoria-metrics-operator, opencost,
opensearch) is solid for metrics + dashboards + logs, but it leaves gaps that
matter to platform operators and tenants alike:

• No out-of-the-box service map / dependency graph. Operators can't see, without
  manual dashboards, how workloads talk to each other (and to managed services like
  Postgres, Kafka, Redis).
• No zero-code distributed tracing. Today tracing requires the application owner to
  instrument code; many tenant workloads never will. eBPF closes that gap.
• No continuous profiling for "why is this pod burning CPU" investigations.
• Latency/error attribution across the mesh and to backing stores is hard to get
  from raw PromQL.

Coroot delivers all of the above automatically, which is especially valuable in a
multi-tenant PaaS where the platform can't assume tenants instrument their own apps.

This is complementary to (not a replacement for) the existing OpenTelemetry idea in
#1780: OTel is the
instrumentation/pipeline standard; Coroot is a turnkey eBPF-based analysis layer that
can consume telemetry without requiring tenants to wire up collectors.

Fit with Cozystack principles

Beyond the feature set, Coroot lines up well with how Cozystack is built and governed:

• Fully open source (Apache-2.0). Both the Coroot server
  (coroot/coroot) and its Kubernetes operator
  (coroot/coroot-operator) are licensed
  under Apache-2.0 — the same permissive, CNCF-friendly licensing Cozystack itself uses.
  There's no open-core bait-and-switch on the core observability capabilities, so it's a
  safe fit for an open platform and for downstream redistribution.
• Operator-driven deployment — Cozystack's native pattern. Coroot ships an official
  Kubernetes/OpenShift operator that manages its components and storage via CRDs.
  Cozystack is overwhelmingly operator-centric — it already bundles
  victoria-metrics-operator, grafana-operator, clickhouse-operator,
  prometheus-operator-crds, and many others. Packaging Coroot via its operator is
  therefore idiomatic: lifecycle, upgrades, and configuration follow the exact pattern
  Cozystack already uses for every other stateful component, rather than a bespoke set
  of manifests.

What Coroot adds

AI-assisted root cause analysis

Coroot integrates with LLMs to perform automated root cause analysis — it correlates
its eBPF-collected metrics, traces, logs, and the service map and produces a plain-language
explanation of what broke and why, dramatically shortening incident triage. This is very
much in the spirit of Cozystack as a modern, batteries-included platform: it turns the
raw telemetry the platform already has into actionable answers, rather than leaving
operators to hand-correlate dashboards. (The RCA layer is optional and pluggable, so it
can be enabled per environment and pointed at a chosen model/provider.)

Coroot components

A Coroot install consists of:

• coroot – the server/UI (Apache-2.0).
• coroot-node-agent – a privileged DaemonSet that loads eBPF programs on each node
  to collect metrics, logs, traces, and profiles.
• coroot-cluster-agent – discovers and scrapes cluster-level signals.
• Storage backends – a Prometheus-compatible TSDB for metrics and ClickHouse for
  logs/traces/profiles.

Proposed positioning (two complementary modes)

Based on Cozystack's package layout, I propose targeting both of the following,
likely in phases:

1. Platform / operator mode — Coroot as a system-level component
   (packages/system or packages/extra/monitoring), managed by the Cozystack
   operator, giving cluster admins fleet-wide observability across all tenants and
   managed services. This is the higher-leverage starting point.

2. Tenant-app mode — Coroot packaged as a tenant-deployable application under
   packages/apps (like postgres, kafka, clickhouse), so a tenant gets
   self-service, namespace-scoped observability for their own workloads, integrated
   with the tenant dashboard and Keycloak SSO.

The two modes share most of the chart work; tenant-app mode mainly adds scoping,
RBAC/network-policy isolation, and per-tenant UI wiring.

Incremental adoption: agents-only / bring-your-own-UI

Even if maintainers decide Coroot as a whole product is not desirable for Cozystack
(extra UI, ownership, overlap with existing tooling), the coroot-node-agent is
valuable on its own. The eBPF node-agent works as a standalone telemetry source: it
exposes Prometheus-compatible metrics (and can emit OTLP traces), so its signals can be
scraped straight into the existing VictoriaMetrics + Grafana stack without running
the Coroot UI at all.

This means there's a low-commitment path:

• Minimal: deploy only coroot-node-agent, scrape it with the existing VMAgent,
  build/curate Grafana dashboards on top. No Coroot server, no new UI to own.
• Full: the complete Coroot experience (service map, traces, profiling, inspections)
  as described above.

There is real-world precedent for running the agents decoupled from the Coroot UI — e.g.
the ecom.tech talk on their experience adopting OpenTelemetry, Qryn and Coroot
(https://youtu.be/z-68KyE90jk), which covers feeding this kind of telemetry into a
non-Coroot backend/UI. Unfortunately I wasn't able to find many other public examples of
this standalone-agent pattern, so it's worth flagging as an area to validate.

Two reasons this matters for Cozystack specifically:

1. Full-fledged monitoring with minimal new surface. The agents alone meaningfully
   close today's gaps (network/L7 RED metrics, connection-level visibility) while
   reusing the platform's existing storage and dashboards.
2. Foundation for security monitoring. Because coroot-node-agent observes
   L7 traffic, network flows, and process/syscall-level activity via eBPF, it provides a
   data source that can later underpin security observability (anomalous connections,
   unexpected egress, process behavior) — not just performance monitoring. This is a
   useful long-term building block independent of whether the Coroot UI is adopted.

Backend integration — open question (input wanted)

Coroot can either bring its own Prometheus + ClickHouse, or reuse what Cozystack
already runs. I'd like maintainer input on where to land. The two ends of the spectrum:

• Reuse existing platform backends (my leaning): Coroot remote-writes metrics into
  the existing VictoriaMetrics, and stores traces/logs/profiles in a ClickHouse
  instance provisioned via the existing clickhouse-operator. Pros: no duplication,
  consistent with Cozystack conventions, single source of truth for metrics. Cons:
  requires validating Coroot against VM's Prometheus-compatibility surface and wiring
  ClickHouse provisioning.
• Bundle Coroot's own stack: simplest to ship and matches upstream's default Helm
  chart, but duplicates storage, increases footprint, and complicates the multi-tenant
  story.

A hybrid is likely best (reuse VM for metrics; dedicated ClickHouse per
scope via the operator), but I'm explicitly leaving this open for discussion rather
than prescribing it.

Multi-tenancy considerations

• Isolation: in tenant-app mode, each tenant's Coroot must only see its own
  namespaces/workloads — needs scoping at the agent/query layer plus NetworkPolicy
  (Cilium) and RBAC boundaries.
• eBPF privilege: coroot-node-agent is a privileged, host-level DaemonSet. In
  platform mode this is an operator concern; in tenant mode the node-agent should remain
  platform-owned (shared) and feed per-tenant views, rather than letting tenants run
  privileged DaemonSets.
• SSO: integrate Coroot auth with the existing Keycloak setup.
• Ingress/DNS: follow the grafana.<tenant-host> pattern for the Coroot UI.

Open questions

1. Reuse VM + clickhouse-operator vs. bundle Coroot's own backends (see above)?
2. Overlap with OpenCost for cost monitoring — coexist, or let one defer to the other?
3. Relationship to logging: ClickHouse-based logs (Coroot) vs. existing OpenSearch
   — coexist, migrate, or keep separate?
4. Is the per-tenant eBPF privilege model acceptable, or should tenant mode be
   metrics/traces-only via a shared platform agent?
5. Sequencing: ship platform mode first, tenant-app mode second?
6. Appetite for an agents-only first step (just coroot-node-agent →
   VictoriaMetrics + Grafana, no Coroot UI) as a low-commitment entry point?

Alternatives considered

• Grafana Beyla + Tempo + Pyroscope assembled by hand — more moving parts, no unified
  service map / inspections UX.
• Pure OpenTelemetry (Proposal: Native support for OpenTelemetry (Collector) integration and per-tenant telemetry isolation #1780) — complementary, but requires instrumentation/pipeline
  setup and doesn't provide turnkey eBPF auto-discovery or the analysis UI.
• Hubble (Cilium) — Cozystack already runs Cilium; Hubble gives flow visibility but
  not the metrics/traces/profiles/SLO analysis Coroot provides. Could be a data source.

Proposed phased plan (for discussion)

1. Phase 0 – this discussion: validate interest and agree on positioning + backend strategy.
2. Phase 1 – agents-only PoC (optional, low-commitment): deploy just
   coroot-node-agent, scrape it with the existing VMAgent into VictoriaMetrics, and
   curate Grafana dashboards — no Coroot server/UI. Validates value and footprint with
   minimal new surface, and stands alone if the full product isn't adopted.
3. Phase 1 – platform mode PoC: Helm package under packages/extra/monitoring (or
   packages/system) that deploys Coroot via the official coroot-operator (CRD-driven,
   matching Cozystack's operator pattern) — node-agent + cluster-agent + server, metrics
   into VM, ClickHouse via the existing operator. Validate VM Prometheus-compatibility
   and resource footprint.
4. Phase 2 – platform hardening: SSO, ingress, dashboards, docs, resource defaults.
5. Phase 3 – tenant-app mode: packages/apps/coroot with isolation, RBAC, per-tenant UI.

Asks

• Is there interest in Coroot as an official Cozystack observability option?
• Preference on positioning (platform component, tenant app, or both as proposed)?
• Preference on backend integration (reuse vs. bundle vs. hybrid)?
• Even if the full Coroot product is out of scope, is there interest in an
  agents-only integration (coroot-node-agent feeding the existing VM + Grafana
  stack), including as a building block for future security monitoring?
• If yes, I'm happy to open a kind/feature issue and start on a PoC PR (agents-only or
  platform mode, whichever you prefer first).