A working cold-chain monitoring lakehouse, built from scratch one layer at a time. Freezer sensors stream temperature and door-open events over MQTT; a change -data-capture stream keeps a device registry in sync; everything lands in Apache Iceberg tables on object storage and is queryable with SQL through Dremio.
It is deliberately end-to-end and deliberately small — sized to run on a single 16 GB laptop — so every moving part is visible and every guarantee is one you can break and watch fail.
Phase 1 of a longer infrastructure roadmap: this repo (the data pipeline), then self-hosting it, then Kubernetes + Helm.
Which customers had a temperature breach this week, attributed to the site the device was at at the time of the breach?
That "at the time" is the whole point. Devices get reassigned between sites and customers; a naive join against the current registry would mis-attribute a breach that happened last Tuesday to wherever the device sits today. Answering it correctly needs change-data-capture (to record registry history) and Iceberg time travel (to query the world as it was) — which is why those technologies are in the stack rather than bolted on for show.
sensors/<id>/telemetry
sim/sensor.py ──MQTT──▶ Mosquitto ──▶ bridge ──▶ Kafka ─┐
(cold-chain fleet) (mqtt→kafka) │
▼
Postgres registry ──WAL──▶ Debezium ──▶ Kafka ──▶ Iceberg sink ──▶ Iceberg tables
(device→site→customer) (Kafka Connect) on MinIO (S3)
│
▼
Dremio ◀── SQL
Single Kafka broker (KRaft, no ZooKeeper). One Kafka Connect worker hosts both the Debezium source and the Iceberg sink — no Spark, no Flink — so the whole stack fits in memory.
| Week | Layer | State |
|---|---|---|
| 1 | Docker fundamentals + MQTT (Mosquitto, sensor sim, QoS) | ✅ done — docs/01-mqtt.md |
| 2 | Kafka (KRaft single broker) | ✅ done — docs/02-kafka.md |
| 3 | MQTT→Kafka bridge | ✅ done — docs/03-bridge.md |
| 4 | Iceberg + MinIO + REST catalog | ✅ done — docs/04-iceberg.md |
| 5 | Debezium CDC from Postgres | ✅ done — docs/05-cdc.md |
| 6 | SQL layer — DuckDB (works) + Dremio UI | ✅ done — docs/06-dremio.md |
Requires Docker Desktop (WSL2 backend) and Python 3.12+.
# 1. bring up the whole stack
docker compose up -d
# 2. register the Debezium CDC connector (once)
curl -X POST -H "Content-Type: application/json" --data @connect/devices-connector.json http://localhost:8083/connectors
# 3. python deps for the sensor sim
py -m venv .venv && .venv/Scripts/python -m pip install -r requirements.txt
# 4. stream cold-chain telemetry through the pipeline
.venv/Scripts/python sim/sensor.py --devices 3 --interval 1
# 5. run the capstone: breaches attributed to the owner at the time
docker run --rm --network telemetry-lakehouse_default -e CATALOG_URI=http://iceberg-rest:8181 -e S3_ENDPOINT=http://minio:9000 -e AWS_ACCESS_KEY_ID=admin -e AWS_SECRET_ACCESS_KEY=password -v "$PWD/sink:/app" telemetry-lakehouse-sink python -u capstone.pyConsoles: MinIO http://localhost:9001 · Kafka Connect http://localhost:8083 · Dremio http://localhost:9047
docker-compose.yml the stack, brought up one service at a time
mqtt/ broker config + observer (sub.py)
sim/ cold-chain sensor simulator (sensor.py)
bridge/ MQTT→Kafka bridge (week 3)
docs/ per-layer write-ups
docs/plans/ the week-by-week build plan
LOG.md append-only journal; read the last entry to resume
Each layer ends at a checkpoint — a command whose output proves it works — and
each is stress-tested by breaking it on purpose (killing the broker mid-publish,
etc.) and recording the result. LOG.md is the running record and the
resume-from-cold pointer.