🌊 Danube Messaging

A lightweight and scalable Cloud-Native Messaging / Streaming Platform

Danube is an open-source distributed messaging broker platform, designed to be cloud-native and cost-effective. Built with a Write-Ahead Log (WAL) architecture and persistent object storage integration, Danube delivers sub-second dispatch with cloud economics.

🚀 Get Started with Danube

Quick Start with Docker Compose - Deploy a cluster in seconds:

Create a directory and download the required files:

mkdir danube-docker && cd danube-docker

Download the docker-compose and broker configuration file:

curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/quickstart/docker-compose.yml

curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/danube_broker.yml

Start the danube cluster:

docker-compose up -d

This launches a complete Danube cluster with:

• 2 High-Availability Brokers for topics failover
• ETCD for distributed metadata management
• Prometheus for monitoring
• danube-cli to produce and consume messages

Test the setup:

Produce messages with reliable delivery

docker-compose run -it danube-cli produce \
  --service-addr http://broker1:6650 \
  --topic "/default/persistent-topic" \
  --count 100 \
  --message "Persistent message" \
  --reliable

Consume messages from the topic

docker-compose run -it danube-cli consume \
  --service-addr http://broker1:6650 \
  --topic "/default/persistent-topic" \
  --subscription "persistent-sub" \
  --sub-type exclusive

📖 Complete Docker Setup Guide →

Architecture

🏗️ Cluster & Broker Characteristics

• Stateless brokers: Metadata in ETCD and data in WAL/Object Storage
• Horizontal scaling: Add brokers in seconds with zero-downtime expansion
• Intelligent load balancing: Automatic topic placement and rebalancing across brokers
• Security-ready: TLS/mTLS support in Admin and data paths
• Leader election & HA: Automatic failover and coordination via ETCD
• Multi-tenancy: Isolated namespaces with policy controls

🌩️ Write-Ahead Log + Cloud Persistence

• Cloud-Native by Design - Danube's architecture separates compute from storage
• Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob, MinIO
• Hot path optimization: Messages served from in-memory WAL cache
• Stream per subscription: WAL + cloud storage from selected offset
• Asynchronous background uploads to S3/GCS/Azure object storage

🎯 Intelligent Load Management

• Automated rebalancing: Detects cluster imbalances and redistributes topics automatically
• Smart topic assignment: Places new topics on least-loaded brokers using configurable strategies
• Resource monitoring: Tracks CPU, memory, throughput, and backlog per broker in real-time
• Configurable policies: Conservative, balanced, or aggressive rebalancing based on workload
• Graceful topic migration: Moves topics between brokers

Core Capabilities

📨 Message Delivery

• Topics: Partitioned and non-partitioned with automatic load balancing
• Reliable Dispatch: At-least-once delivery with configurable storage backends
• Non-Reliable Dispatch: High-throughput, low-latency for real-time scenarios

🔄 Subscription Models

• Exclusive: Single consumer per subscription
• Shared: Load-balanced message distribution across consumers
• Failover: Automatic consumer failover with ordered delivery

📋 Schema Registry

• Centralized schema management: Single source of truth for message schemas across all topics
• Schema versioning: Automatic version tracking with compatibility enforcement
• Multiple formats: Bytes, String, Number, JSON Schema, Avro, Protobuf
• Validation & governance: Prevent invalid messages and ensure data quality

🤖 AI-Powered Administration

Danube features the AI-native messaging platform administration through the Model Context Protocol (MCP):

• Natural language cluster management: Manage your cluster by talking to AI assistants (Claude, Cursor, Windsurf)
• 40 intelligent tools: Full cluster operations accessible via AI - topics, schemas, brokers, diagnostics, metrics
• Automated troubleshooting: AI-guided workflows for consumer lag analysis, health checks, and performance optimization
• Multiple interfaces: CLI commands, Web UI, or AI conversation - your choice

Example: Ask Claude "What's the cluster balance?" or "Create a partitioned topic for analytics" and watch it happen.

Community & Clients

Official Clients

• Rust Client - learn more Rust examples
• Go Client - learn more Go examples
• Java Client - learn more Java examples
• Python Client - learn more Python examples

Community Contributions

Contributions in NodeJs, C / C++ / C#, Ruby, and other languages are welcome! Join our growing ecosystem.

Development & Contribution

Get involved - Danube is actively developed with new features added regularly.

🐛 Report Issues | 💡 Request Features | 📖 Development Guide

Project Structure

• danube-broker - Core messaging platform
• danube-persistent-storage - WAL and cloud storage integration
• danube-client - Async Rust client library
• danube-cli - Command-line producer/consumer tools
• danube-admin - Unified admin tool (CLI + AI/MCP + Web UI)