DataFusion Distributed development tools

This repository contains the remote benchmark platform and pull-request benchmark automation for DataFusion Distributed.

It has two independently operated components:

• benchmarks-remote/ provisions the shared EKS foundation, manages engine deployments, consumes datasets generated elsewhere, and runs benchmarks from a developer machine.
• benchmark-bot/ provisions a persistent EC2 controller that accepts trusted GitHub PR comments, builds immutable base and head revisions, runs isolated benchmark deployments on the existing EKS foundation, and reports the comparison.

The benchmark bot does not own the EKS foundation. A human provisions the foundation from benchmarks-remote/ and gives the controller narrowly scoped access to use it. Both components live in this repository so the trusted benchmark harness can evolve independently of DataFusion Distributed source revisions.

Checkout layout

Keep this repository beside a DataFusion Distributed source checkout. The remote benchmark commands read datasets from S3 and SQL queries from the source checkout's testdata/ directory. Benchmark results are saved locally under testdata/.

<parent>/
  datafusion-distributed/
  datafusion-distributed-dev-tools/

For a source worktree outside this layout, pass --source-root to datafusion-deploy and --testdata-root to datafusion-bench. Benchmark clients also accept DATAFUSION_DISTRIBUTED_ROOT for source checkout discovery.

Development

Install each component's dependencies:

npm run install:all

Validate both components from the repository root:

npm run build
npm test

Prepare datasets directly in S3 using the source project’s generation commands (see dataset preparation). Root npm commands forward operational commands and command-line options to the owning component. Each engine has its own TypeScript deploy, destroy, and benchmark entry points under benchmarks-remote/src/bin/, with Optique parsing and command-specific --help. For example:

npm run foundation-deploy
npm run datafusion-deploy
npm run datafusion-destroy -- --help
npm run datafusion-bench -- tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false --iterations 1
npm run bot-deploy
npm run bot-session

Foundation, engine, benchmark, and controller teardown remain explicit operations. Dataset generation and removal happen outside this repository. See each component's README before operating live infrastructure.