Skip to content

About

Apache DataFusion Ballista Distributed Query Engine

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

2.2k stars

Watchers

42 watching

Forks

Latest commit

 

History

5,459 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Ballista: Making DataFusion Applications Distributed

Crates.io PyPI Apache licensed Commit activity

logo

Ballista is a distributed query execution engine that enhances Apache DataFusion by enabling the parallelized execution of workloads across multiple nodes in a distributed environment.

It is also a lighter, Rust-native alternative to both batch engines such as Apache Spark and interactive MPP query engines such as Trino and Presto, with two execution modes that share one cluster:

  • Staged (default) runs a query the way Spark does: it is split into stages at shuffle boundaries, each stage writes shuffle files, a failed task is re-run from them, and adaptive query execution re-plans the rest of the query between stages. It suits batch and ETL jobs.
  • Pipelined (experimental) runs a query the way interactive MPP engines do: every stage is scheduled at once and stages stream record batches to each other over Arrow Flight, with nothing written to disk. It suits short, interactive queries.

On a 1 TB benchmark derived from TPC-H, Ballista is 2.2x faster than Spark in staged mode and 3.4x faster in pipelined mode. A session can switch modes between queries, so one engine can serve both ETL and interactive queries.

An existing DataFusion application can be distributed with a few lines of code changed:

+use ballista::prelude::*;
 use datafusion::prelude::*;

 #[tokio::main]
 async fn main() -> datafusion::error::Result<()> {
-    let ctx = SessionContext::new();
+    // create SessionContext with ballista support
+    // standalone context will start all required
+    // ballista infrastructure in the background as well
+    let ctx = SessionContext::standalone().await?;
     // register the table
     ctx.register_csv("example", "tests/data/example.csv", CsvReadOptions::new())
         .await?;

     // create a plan to run a SQL query
     let df = ctx
         .sql("SELECT a, MIN(b) FROM example WHERE a <= b GROUP BY a LIMIT 100")
         .await?;

     // execute and print results
     df.show().await?;
     Ok(())
 }

For documentation or more examples, please refer to the Ballista User Guide.

Performance

Ballista is measured against Apache Spark 4.1.3 on the same cluster shape: Kubernetes on AWS, with 32 executors of 8 vCPUs running the 22 TPC-H-derived queries over 1 TB of Parquet in S3. Each Ballista executor had 64 GiB of memory, and each Spark executor 74 GiB.

Total time for the 22 queries for Apache Spark, Ballista staged and Ballista pipelined

Time per query for Apache Spark, Ballista staged and Ballista pipelined

Derived from TPC-H and not an official TPC-H result. Times are the mean of two runs, including the cold first run. The Spark numbers come from an earlier run with separate scripts and its own phrasing of the queries. See the benchmarking guide for versions, settings, the full results and how to reproduce them.

Features

Query interfaces

  • SQL and DataFrames from Rust and Python, using DataFusion's APIs
  • Arrow Flight SQL, so ADBC and JDBC drivers and BI tools can query a cluster without linking Ballista
  • SQL CLI with a built-in terminal UI
  • Substrait and physical plans accepted alongside logical plans
  • DataFrame.cache() keeps intermediate results on the executors for reuse within a session (opt-in)

Execution

  • Adaptive query execution (AQE), on by default, re-optimizes the remaining plan from runtime statistics: join selection, broadcast joins, empty-stage elimination and exchange reuse, with opt-in skew-join splitting and partition coalescing
  • Two execution modes that share one cluster: fault-tolerant staged execution by default, for batch and ETL jobs, and low-latency pipelined execution (experimental), for interactive queries
  • Memory-bounded executors with an auto-sized pool by default, and spilling to disk
  • Sort-based and hash shuffle with configurable compression
  • Multi-partition tasks that share memory and broadcast state across a stage's partitions
  • Parallel window functions over range shuffles (opt-in)

Operations

  • Deployment as native binaries or Docker images, with Docker Compose and Kubernetes guides and optional KEDA autoscaling
  • Fault tolerance: failed tasks are retried, and work lost with an executor is re-run
  • Graceful shutdown and Kubernetes health probes
  • Observability: a browser-based Web TUI, a terminal UI, a REST API with an OpenAPI spec, Prometheus metrics, and a history server for completed jobs
  • Result service for clients outside the cluster network
  • Data access to Parquet, CSV and JSON on S3-compatible object stores, HTTP and local filesystems

Extensibility

  • Reusable components: embed the scheduler and executor crates in your own engine
  • Extension points for custom operators and codecs, functions, object stores, session configuration and adaptive optimizer rules

Who is Ballista for

Ballista serves several distinct audiences:

  • DataFusion users going multi-node — you already use Apache DataFusion on a single machine and have outgrown it. Ballista runs the same SQL and DataFrame workloads across a cluster with minimal code changes and the same results.
  • Spark users wanting the same execution model — you run Spark SQL or batch jobs and want a lighter, Rust-native alternative without relearning a new paradigm. Ballista keeps the familiar model: plans split into stages at shuffle boundaries, tasks over partitions, executors with vcores, and adaptive query execution (AQE). For latency-sensitive queries, the same cluster can also run them pipelined, the way interactive MPP engines do, instead of needing a separate interactive engine.
  • Teams running separate batch and interactive engines — you run Spark for ETL and an MPP engine such as Trino, Presto or Impala for interactive analytics, and want to consolidate. Ballista runs both workloads on one scheduler and one pool of executors, choosing staged or pipelined execution per query.
  • Library users building a specialized engine — you are building a bespoke distributed query engine and want reusable scheduler, executor, and plan-serialization building blocks with extension points, instead of writing distributed execution from scratch.

These audiences are documented in more detail, along with the guarantees each relies on, in the User Personas guide.

Architecture

A Ballista cluster consists of one or more scheduler processes and one or more executor processes. These processes can be run as native binaries and are also available as Docker Images, which can be easily deployed with Docker Compose or Kubernetes.

Clients submit query plans to the scheduler over gRPC, or SQL over Arrow Flight SQL. The scheduler breaks each query into stages and sends tasks to the executors, which report progress back. Executors exchange data with each other over Arrow Flight, reading shuffle files in staged mode or streaming record batches directly in pipelined mode, and return results to the client the same way.

Ballista Cluster Diagram

See the architecture guide for more details.

Execution Modes: One Engine for ETL and Interactive Queries

A lakehouse aims to merge the data warehouse and the ETL platform, but it is often still served by two kinds of engine: a batch engine for long ETL jobs, built to survive task failures, and an interactive engine for short analytical queries, built for low latency. Ballista's two execution modes let one engine serve both, from the same scheduler and executors. A session selects its mode with ballista.execution.mode and can change it between queries, so the same SQL and DataFrame code serves either workload.

The modes trade fault tolerance for latency. In staged mode, a failed task is re-run from the shuffle files its input stages wrote, which suits long-running jobs and busy shared clusters. In pipelined mode, results stream to the client as they are produced, but there are no shuffle files to recover from, so a failed task fails the whole job, and the cluster needs room to run every task of the job at once. See the Pipelined Execution Guide for details.

Sharing one pool of executors can also take less capacity than running two clusters. Batch and interactive load are often uncorrelated, so the capacity one leaves idle is available to the other, and the pool can be sized for the peak of the combined load rather than for the sum of two separate peaks.

Sharing a cluster this way is still early. Pipelined execution is experimental, and the scheduler does not prioritize one job over another, so on a busy cluster an interactive query can wait for vcores that batch jobs hold, as the limitations of pipelined execution describe. Prioritizing interactive queries over batch jobs in the scheduler is what a shared cluster still needs.

Getting Started

The easiest way to get started is to run one of the standalone or distributed examples. After that, refer to the Getting Started Guide.

Web Terminal User Interface (Web TUI)

Ballista provides a browser-based Web TUI for monitoring a running cluster. It exposes the TUI views for jobs, executors, metrics, and scheduler information directly in a web browser.

Ballista Web TUI

When the scheduler HTTP endpoint is available, opening the scheduler address in a browser, for example http://localhost:50050, redirects to a hosted Web TUI.

For more information, including how to run the Web TUI locally, see the Ballista CLI documentation.

Project Status

Ballista is part of the Apache DataFusion project at the Apache Software Foundation. It ships about every two months alongside DataFusion. Adopters include Spice AI and Coralogix.

Changes are tested on real clusters before they merge:

  • The merge queue runs TPC-H (SF10) and TPC-DS (SF1) queries on a multi-process cluster and compares the results with single-process DataFusion, including runs with multi-partition tasks and, for TPC-H, pipelined mode.
  • A fault-injection suite kills executors mid-query, with AQE on and off, and checks that the results are still correct. It runs on every code change, and its Kubernetes scenarios also run nightly.
  • Upgrade guides cover breaking changes between releases, and clients get a clear error when their major version doesn't match the scheduler's.

Ballista supports a wide range of SQL, including CTEs, joins and subqueries, and can execute complex queries at scale, but there is still a gap between DataFusion and Ballista, which may bring incompatibilities. The community is actively working to close it. Refer to the DataFusion SQL Reference for more information on supported SQL.

Cargo Features

Ballista uses Cargo features to enable optional functionality.

Cargo features for each crate

ballista (client)

Feature Default Description
standalone Yes Enables standalone mode with in-process scheduler and executor

ballista-core

Feature Default Description
arrow-ipc-optimizations Yes Enables Arrow IPC optimizations for better shuffle performance
build-binary No Required for building binary executables (AWS S3 support, CLI parsing)
force_hash_collisions No Testing-only: forces all values to hash to same value
utoipa No Derives OpenAPI schemas for scheduler REST types; enabled by ballista-scheduler's rest-api

ballista-scheduler

Feature Default Description
build-binary Yes Builds the scheduler binary with CLI and logging
substrait No Enables Substrait plan support
flight-sql No Enables the Arrow Flight SQL frontend for ADBC and JDBC clients
prometheus-metrics No Enables Prometheus metrics collection
graphviz-support No Enables execution graph visualization
keda-scaler No Kubernetes Event Driven Autoscaling integration
rest-api Yes Enables REST API endpoints, the OpenAPI spec, and the history server

ballista-executor

Feature Default Description
arrow-ipc-optimizations Yes Enables Arrow IPC optimizations
build-binary Yes Builds the executor binary with CLI and logging, using mimalloc as the allocator
jemalloc No Uses jemalloc instead of mimalloc as the allocator (not supported on MSVC targets)

ballista-cli

Feature Default Description
cli Yes Builds the ballista-cli SQL shell binary
tui Yes Enables a REST client with Terminal User Interface
web No Builds the WebAssembly Web TUI (see the Ballista CLI documentation)

Usage Examples

# Build with standalone support (default)
cargo build -p ballista

# Build with Substrait support
cargo build -p ballista-scheduler --features substrait

Contribution Guide

Please see the Contribution Guide for information about contributing to Ballista.

About

Apache DataFusion Ballista Distributed Query Engine

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

2.2k stars

Watchers

42 watching

Forks

Releases

Packages

Used by

Contributors

Languages