Skip to content

Commit 29692a9

Browse files
committed
Add design proposal: Coroot eBPF observability option for Cozystack
Migrated from discussion cozystack/cozystack#3028 to the design-proposal process for review. Signed-off-by: Gaál György <[email protected]>
1 parent fbfc6ba commit 29692a9

1 file changed

Lines changed: 122 additions & 0 deletions

File tree

  • design-proposals/coroot-ebpf-observability
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Integrate Coroot as a zero-instrumentation (eBPF) observability option
2+
3+
- **Title:** `Integrate Coroot as a zero-instrumentation (eBPF) observability option for Cozystack`
4+
- **Author(s):** `@gecube`
5+
- **Date:** `2026-06-24`
6+
- **Status:** Draft
7+
8+
> Migrated from discussion [cozystack/cozystack#3028](https://github.com/cozystack/cozystack/discussions/3028) to the design-proposal process for review.
9+
10+
## Overview
11+
12+
This proposal adds [Coroot](https://coroot.com/) — an Apache-2.0, open-source, eBPF-based observability platform — as a native option in Cozystack. Coroot auto-discovers workloads and generates telemetry from kernel-level events without requiring application code changes, which aligns with Cozystack's batteries-included philosophy.
13+
14+
The decisive fit: Cozystack already operates the backends Coroot needs — VictoriaMetrics and ClickHouse — so adding Coroot does not duplicate storage.
15+
16+
## Scope and related proposals
17+
18+
- Migrated together with the **[self-hosted in-cluster registry for air-gapped Cozystack](../airgap-in-cluster-registry/)** proposal (from discussion [#3029](https://github.com/cozystack/cozystack/discussions/3029)). The two are independent in substance; they are submitted as a pair.
19+
- Overlaps to manage with existing components: **OpenCost** (cost monitoring) and the **OpenSearch** logging integration (vs. Coroot's ClickHouse-based logs). See open questions.
20+
21+
## Context
22+
23+
Cozystack ships a solid metrics/logging stack, but several observability capabilities are missing today:
24+
25+
- No automatic service mapping or dependency graphs.
26+
- Distributed tracing requires manual application instrumentation.
27+
- No continuous profiling.
28+
- Hard to attribute latency/errors across services.
29+
30+
These gaps are particularly painful on a multi-tenant platform, where tenants frequently do not instrument their own applications.
31+
32+
### The problem
33+
34+
A platform operator (or a tenant) wanting a service map, traces, or profiles today has to instrument applications by hand — something tenants often will not do. The result is blind spots in cross-service latency and error attribution that the platform cannot fill on the tenant's behalf.
35+
36+
## Goals
37+
38+
- Offer eBPF-based, zero-instrumentation observability (service maps, tracing, profiling) as a native Cozystack option.
39+
- Reuse existing backends (VictoriaMetrics, ClickHouse) rather than bundling duplicate storage.
40+
- Follow the existing operator-driven deployment pattern used by other Cozystack components.
41+
- Provide a low-commitment entry point that does not require adopting the full Coroot UI.
42+
43+
### Non-goals
44+
45+
- Replacing the existing metrics/logging stack.
46+
- Mandating Coroot for all clusters; it is an option, not a default.
47+
- Committing up front to the per-tenant eBPF privilege model (its feasibility is an open question).
48+
49+
## Design
50+
51+
**Why Coroot fits Cozystack**
52+
- Fully open-source (Apache-2.0), no open-core restrictions.
53+
- Operator-driven deployment, matching existing components.
54+
- Reuses VictoriaMetrics and ClickHouse — no storage duplication.
55+
56+
**Dual deployment modes**
57+
1. **Platform/operator mode** — fleet-wide visibility for the platform team.
58+
2. **Tenant-app mode** — self-service, namespace-scoped observability for tenants.
59+
60+
**Low-commitment alternative**
61+
Deploy only the Coroot node-agent, feeding metrics into the existing VictoriaMetrics + Grafana, bypassing the Coroot UI entirely. This is the cheapest way to validate value before committing to the full platform.
62+
63+
```mermaid
64+
flowchart LR
65+
subgraph node[Each node]
66+
A[Coroot node-agent<br/>eBPF]
67+
end
68+
A -->|metrics| VM[(VictoriaMetrics)]
69+
A -->|traces/logs| CH[(ClickHouse)]
70+
VM --> G[Grafana]
71+
VM --> UI[Coroot UI optional]
72+
CH --> UI
73+
```
74+
75+
## User-facing changes
76+
77+
- A deployable Coroot option (platform component and/or tenant app), following existing Cozystack packaging.
78+
- In the agents-only mode: new metrics in VictoriaMetrics and Grafana dashboards, with no new UI surface.
79+
- In full mode: a Coroot UI (SSO/ingress addressed in Phase 2).
80+
81+
## Upgrade and rollback compatibility
82+
83+
Coroot is additive: it introduces no changes to existing CRDs or backends beyond writing telemetry into them. Rollback is removal of the component; existing VictoriaMetrics/ClickHouse data and the rest of the stack are unaffected.
84+
85+
## Security
86+
87+
eBPF agents run privileged on each node — a new trust boundary that must be evaluated, especially for the tenant-app mode where namespace-scoped tenants would gain access to kernel-level telemetry. The per-tenant eBPF privilege model is an explicit open question; tenant-app mode is deferred to a later phase partly for this reason. SSO and ingress hardening for the UI land in Phase 2.
88+
89+
## Failure and edge cases
90+
91+
- **Coroot UI / control plane down** → agents keep shipping to VictoriaMetrics/ClickHouse; dashboards in Grafana remain usable (agents-only mode is unaffected by UI availability).
92+
- **eBPF unsupported on a node kernel** → agent should degrade/skip rather than crash-loop; document the minimum kernel.
93+
- **Backend overlap** → Coroot writing logs to ClickHouse alongside the OpenSearch integration risks duplicate log pipelines; pick one per cluster (open question).
94+
95+
## Testing
96+
97+
- Unit/packaging: operator install/uninstall is clean and idempotent.
98+
- Integration: agents-only mode emits expected metrics into VictoriaMetrics and renders in Grafana.
99+
- e2e: full platform mode produces a correct service map and traces for a known multi-service workload.
100+
- Manual: tenant-app mode privilege scoping (once that phase is in scope).
101+
102+
## Rollout
103+
104+
- **Phase 0** — Community discussion and consensus (this proposal).
105+
- **Phase 1** — Agents-only PoC (or platform-mode PoC) reusing existing backends.
106+
- **Phase 2** — Hardening: SSO, ingress, documentation.
107+
- **Phase 3** — Tenant-app (namespace-scoped) mode.
108+
109+
## Open questions
110+
111+
1. Backend strategy: reuse existing VictoriaMetrics/ClickHouse vs. bundling dedicated instances?
112+
2. How to manage overlap with OpenCost for cost monitoring?
113+
3. ClickHouse logs vs. the existing OpenSearch integration — one or both?
114+
4. Is a per-tenant eBPF privilege model feasible?
115+
5. Phasing preference — platform-first?
116+
6. Interest in an agents-only preliminary implementation as the entry point?
117+
118+
## Alternatives considered
119+
120+
- **Manual application instrumentation (status quo).** Rejected as the only option: tenants often will not instrument, leaving permanent blind spots the platform cannot fill.
121+
- **Bundling dedicated VictoriaMetrics/ClickHouse for Coroot.** Possible but duplicates storage the cluster already runs; kept as an open question rather than the default.
122+
- **Full Coroot UI from day one.** Heavier commitment than needed to validate value; the agents-only mode is offered as the low-risk entry point instead.

0 commit comments

Comments
 (0)