Brokoli

The open-source data platform for building, running, and explaining data pipelines.

Website · Documentation · Quick Start · Features · Examples

---

It starts as a single ~30MB binary with a visual editor, parallel DAG execution, built-in data quality, real-time monitoring, and a Python SDK — no infrastructure required beyond the binary itself.

The important part is the complete operating loop: define a pipeline, run it, see the physical plan and evidence behind the result, and trace the data that was affected. Start locally, then keep the same pipeline when you need more workers or stronger governance.

---

Quick Start

Option A: Install Script

curl -fsSL https://raw.githubusercontent.com/Tnsor-Labs/brokoli/main/install.sh | sh
brokoli serve

Open http://localhost:8080, create your admin account, choose the Hello World template, and click Run. You should have a successful run, an output CSV, and an execution record in a few minutes.

For the complete first-success path, see the five-minute getting started guide.

Option B: Download Binary

# Linux x86_64
curl -L https://github.com/Tnsor-Labs/brokoli/releases/latest/download/brokoli_Linux_x86_64.tar.gz | tar xz
./brokoli serve

Option C: Python SDK

pip install brokoli

from brokoli import Pipeline, task, source_api, quality_check, sink_file

with Pipeline("my_pipeline", schedule="0 6 * * *") as p:

    data = source_api("Fetch", url="https://api.example.com/data", retries=3)

    @task("Transform")
    def clean(raw):
        return [r for r in raw if r.get("status") == "active"]

    cleaned = clean(data)
    quality_check("Validate", cleaned, rules=["not_null(id)", "unique(id)"])
    cleaned >> sink_file("Save", path="/tmp/output.csv")

brokoli deploy my_pipeline.py --server http://localhost:8080

TypeScript code nodes are available when Node.js 20 or newer is installed on the machine running the Brokoli server. Node is optional: Python nodes and non-code pipelines work without it, and Brokoli never downloads or bundles a runtime. The server resolves the existing node executable at startup; when it finds none, /api/capabilities stops advertising typescript, and the SDKs refuse to deploy a TypeScript node against that host rather than letting it fail at run time. See the TypeScript example for a dependency-free end-to-end run.

The smallest working example is also in examples/hello-world.py.

From Source

bash build-ui.sh                        # builds the React UI onto web/dist
go build -tags minicore_disabled -o brokoli . && ./brokoli serve

build-ui.sh is the step that places the built UI on the embed target. Building in ui/ alone leaves it in ui/apps/community/dist, and the binary then starts with no UI and says so on its first page.

---

Features

Single binary, no infrastructure. Runs on any Linux/macOS machine with no external dependencies. SQLite by default; swap to PostgreSQL with a connection string.

Visual pipeline editor. Drag-and-drop canvas with 15 node types, zoom/pan, inline node config, dry-run preview, and YAML import/export. No code required for common ETL patterns.

Built-in data quality. 10 quality rules (not_null, unique, min/max, range, regex, row_count, type_check, freshness, no_blank), auto-profiling on every node output, and schema drift detection across runs.

Parallel DAG execution. Wave-based Kahn's algorithm, per-node retry with exponential backoff, resume from failure, cross-pipeline dependencies with trigger-mode chaining, condition branching.

Explainable operations. Live run status and log streaming over WebSocket, physical execution plans, run evidence, artifacts, profiling, a run calendar heatmap, and cross-pipeline lineage.

Secrets and connections. AES-256-GCM encryption at rest, typed database, API, file, and cloud-storage connections, typed variables with ${var.key} resolution in any config field.

Authentication. httpOnly cookie sessions, JWT auth, API key support, role-based access (admin/editor/viewer).

Pipeline versioning. Auto-snapshot on every save, full version history, one-click rollback.

Webhook triggers. Per-pipeline tokens — integrate with GitHub Actions, dbt, Kafka, or any HTTP source.

---

Brokoli Cloud

Brokoli Cloud builds on the open-source core with team and governance features — SSO/OIDC, audit logging, workspaces with RBAC, Git Sync, data contracts, column-level lineage, PII detection, OpenLineage export, and Kubernetes execution. Free tier available.

Try it now

---

Configuration

Flag	Default	Description
--port / -p	8080	HTTP server port
--db	./brokoli.db	SQLite path or postgres://... URI
--api-key	—	Enable API key authentication

Environment variable	Description
BROKOLI_APP_URL	Base URL for Slack deep links
BROKOLI_JWT_SECRET	Persistent JWT signing secret
BROKOLI_CORS_ORIGINS	Allowed CORS origins (comma-separated)
BROKOLI_MAX_CONCURRENT_RUNS	Max parallel pipeline runs (default: 4)

---

License

Apache 2.0 — see LICENSE.

Enterprise features are available under a separate commercial license. Contact us for pricing.