|
| 1 | +# Integrate Coroot as a zero-instrumentation (eBPF) observability option |
| 2 | + |
| 3 | +- **Title:** `Integrate Coroot as a zero-instrumentation (eBPF) observability option for Cozystack` |
| 4 | +- **Author(s):** `@gecube` |
| 5 | +- **Date:** `2026-06-24` |
| 6 | +- **Status:** Draft |
| 7 | + |
| 8 | +> Migrated from discussion [cozystack/cozystack#3028](https://github.com/cozystack/cozystack/discussions/3028) to the design-proposal process for review. |
| 9 | +
|
| 10 | +## Overview |
| 11 | + |
| 12 | +This proposal adds [Coroot](https://coroot.com/) — an Apache-2.0, open-source, eBPF-based observability platform — as a native option in Cozystack. Coroot auto-discovers workloads and generates telemetry from kernel-level events without requiring application code changes, which aligns with Cozystack's batteries-included philosophy. |
| 13 | + |
| 14 | +The decisive fit: Cozystack already operates the backends Coroot needs — VictoriaMetrics and ClickHouse — so adding Coroot does not duplicate storage. |
| 15 | + |
| 16 | +## Scope and related proposals |
| 17 | + |
| 18 | +- Migrated together with the **[self-hosted in-cluster registry for air-gapped Cozystack](../airgap-in-cluster-registry/)** proposal (from discussion [#3029](https://github.com/cozystack/cozystack/discussions/3029)). The two are independent in substance; they are submitted as a pair. |
| 19 | +- Overlaps to manage with existing components: **OpenCost** (cost monitoring) and the **OpenSearch** logging integration (vs. Coroot's ClickHouse-based logs). See open questions. |
| 20 | + |
| 21 | +## Context |
| 22 | + |
| 23 | +Cozystack ships a solid metrics/logging stack, but several observability capabilities are missing today: |
| 24 | + |
| 25 | +- No automatic service mapping or dependency graphs. |
| 26 | +- Distributed tracing requires manual application instrumentation. |
| 27 | +- No continuous profiling. |
| 28 | +- Hard to attribute latency/errors across services. |
| 29 | + |
| 30 | +These gaps are particularly painful on a multi-tenant platform, where tenants frequently do not instrument their own applications. |
| 31 | + |
| 32 | +### The problem |
| 33 | + |
| 34 | +A platform operator (or a tenant) wanting a service map, traces, or profiles today has to instrument applications by hand — something tenants often will not do. The result is blind spots in cross-service latency and error attribution that the platform cannot fill on the tenant's behalf. |
| 35 | + |
| 36 | +## Goals |
| 37 | + |
| 38 | +- Offer eBPF-based, zero-instrumentation observability (service maps, tracing, profiling) as a native Cozystack option. |
| 39 | +- Reuse existing backends (VictoriaMetrics, ClickHouse) rather than bundling duplicate storage. |
| 40 | +- Follow the existing operator-driven deployment pattern used by other Cozystack components. |
| 41 | +- Provide a low-commitment entry point that does not require adopting the full Coroot UI. |
| 42 | + |
| 43 | +### Non-goals |
| 44 | + |
| 45 | +- Replacing the existing metrics/logging stack. |
| 46 | +- Mandating Coroot for all clusters; it is an option, not a default. |
| 47 | +- Committing up front to the per-tenant eBPF privilege model (its feasibility is an open question). |
| 48 | + |
| 49 | +## Design |
| 50 | + |
| 51 | +**Why Coroot fits Cozystack** |
| 52 | +- Fully open-source (Apache-2.0), no open-core restrictions. |
| 53 | +- Operator-driven deployment, matching existing components. |
| 54 | +- Reuses VictoriaMetrics and ClickHouse — no storage duplication. |
| 55 | + |
| 56 | +**Dual deployment modes** |
| 57 | +1. **Platform/operator mode** — fleet-wide visibility for the platform team. |
| 58 | +2. **Tenant-app mode** — self-service, namespace-scoped observability for tenants. |
| 59 | + |
| 60 | +**Low-commitment alternative** |
| 61 | +Deploy only the Coroot node-agent, feeding metrics into the existing VictoriaMetrics + Grafana, bypassing the Coroot UI entirely. This is the cheapest way to validate value before committing to the full platform. |
| 62 | + |
| 63 | +```mermaid |
| 64 | +flowchart LR |
| 65 | + subgraph node[Each node] |
| 66 | + A[Coroot node-agent<br/>eBPF] |
| 67 | + end |
| 68 | + A -->|metrics| VM[(VictoriaMetrics)] |
| 69 | + A -->|traces/logs| CH[(ClickHouse)] |
| 70 | + VM --> G[Grafana] |
| 71 | + VM --> UI[Coroot UI optional] |
| 72 | + CH --> UI |
| 73 | +``` |
| 74 | + |
| 75 | +## User-facing changes |
| 76 | + |
| 77 | +- A deployable Coroot option (platform component and/or tenant app), following existing Cozystack packaging. |
| 78 | +- In the agents-only mode: new metrics in VictoriaMetrics and Grafana dashboards, with no new UI surface. |
| 79 | +- In full mode: a Coroot UI (SSO/ingress addressed in Phase 2). |
| 80 | + |
| 81 | +## Upgrade and rollback compatibility |
| 82 | + |
| 83 | +Coroot is additive: it introduces no changes to existing CRDs or backends beyond writing telemetry into them. Rollback is removal of the component; existing VictoriaMetrics/ClickHouse data and the rest of the stack are unaffected. |
| 84 | + |
| 85 | +## Security |
| 86 | + |
| 87 | +eBPF agents run privileged on each node — a new trust boundary that must be evaluated, especially for the tenant-app mode where namespace-scoped tenants would gain access to kernel-level telemetry. The per-tenant eBPF privilege model is an explicit open question; tenant-app mode is deferred to a later phase partly for this reason. SSO and ingress hardening for the UI land in Phase 2. |
| 88 | + |
| 89 | +## Failure and edge cases |
| 90 | + |
| 91 | +- **Coroot UI / control plane down** → agents keep shipping to VictoriaMetrics/ClickHouse; dashboards in Grafana remain usable (agents-only mode is unaffected by UI availability). |
| 92 | +- **eBPF unsupported on a node kernel** → agent should degrade/skip rather than crash-loop; document the minimum kernel. |
| 93 | +- **Backend overlap** → Coroot writing logs to ClickHouse alongside the OpenSearch integration risks duplicate log pipelines; pick one per cluster (open question). |
| 94 | + |
| 95 | +## Testing |
| 96 | + |
| 97 | +- Unit/packaging: operator install/uninstall is clean and idempotent. |
| 98 | +- Integration: agents-only mode emits expected metrics into VictoriaMetrics and renders in Grafana. |
| 99 | +- e2e: full platform mode produces a correct service map and traces for a known multi-service workload. |
| 100 | +- Manual: tenant-app mode privilege scoping (once that phase is in scope). |
| 101 | + |
| 102 | +## Rollout |
| 103 | + |
| 104 | +- **Phase 0** — Community discussion and consensus (this proposal). |
| 105 | +- **Phase 1** — Agents-only PoC (or platform-mode PoC) reusing existing backends. |
| 106 | +- **Phase 2** — Hardening: SSO, ingress, documentation. |
| 107 | +- **Phase 3** — Tenant-app (namespace-scoped) mode. |
| 108 | + |
| 109 | +## Open questions |
| 110 | + |
| 111 | +1. Backend strategy: reuse existing VictoriaMetrics/ClickHouse vs. bundling dedicated instances? |
| 112 | +2. How to manage overlap with OpenCost for cost monitoring? |
| 113 | +3. ClickHouse logs vs. the existing OpenSearch integration — one or both? |
| 114 | +4. Is a per-tenant eBPF privilege model feasible? |
| 115 | +5. Phasing preference — platform-first? |
| 116 | +6. Interest in an agents-only preliminary implementation as the entry point? |
| 117 | + |
| 118 | +## Alternatives considered |
| 119 | + |
| 120 | +- **Manual application instrumentation (status quo).** Rejected as the only option: tenants often will not instrument, leaving permanent blind spots the platform cannot fill. |
| 121 | +- **Bundling dedicated VictoriaMetrics/ClickHouse for Coroot.** Possible but duplicates storage the cluster already runs; kept as an open question rather than the default. |
| 122 | +- **Full Coroot UI from day one.** Heavier commitment than needed to validate value; the agents-only mode is offered as the low-risk entry point instead. |
0 commit comments