Skip to content

Commit 373fab7

Browse files
authored
Add support for running sql benchmarks with command line arguments (apache#23772)
## Which issue does this PR close? - Part of apache#21706 and apache#21937 ## Rationale for this change SQL benchmark suites expose configuration through environment variables. Contributors must consult the source or documentation to discover suite-specific settings, and Cargo does not forward custom arguments to benchmark targets. This change lets `benchmark_runner` expose those settings as validated command-line arguments. It also adds suite-specific help and a dry-run mode for inspecting resolved configuration without executing a benchmark. ## What changes are included in this PR? This PR adds TOML `.suite` metadata files that define each SQL benchmark suite’s: - Description and help examples. - Suite-specific options, defaults, environment variables, and accepted values. - Query filename pattern. - Data-path replacements. `benchmark_runner` uses this metadata to: - Generate suite-specific command-line arguments and help output. - Resolve suite options using command-line, environment-variable, and default-value precedence. - Support `--path` for suites that declare a `DATA_DIR` replacement. - Support `--result-mode` for result persistence and validation. - Validate suite names, metadata, option conflicts, query identifiers, and incompatible arguments. - Provide `--dry-run` JSON output containing the resolved options, paths, value sources, filters, and execution mode without loading benchmark definitions or executing SQL. The existing basic runner, Criterion mode, and suite-listing functionality remain in place. ## Are these changes tested? Yes. New tests cover: - Suite metadata parsing and validation. - Dynamic suite options and help output. - Option precedence between command-line arguments, environment variables, and defaults. - Path and query filename resolution. - Dry-run output and validation. - Result-mode resolution. - Invalid option combinations and malformed metadata. ## Are there any user-facing changes? Yes. Contributors can configure suite-specific settings through command-line arguments and inspect them through suite help: ```shell cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --help cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \ tpch --query 15 --format csv --path /path/to/tpch cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \ clickbench --partitioning partitioned --dry-run ``` Environment variables remain supported for compatibility with direct cargo bench invocations and existing benchmark definitions. The SQL benchmark README documents the command-line options, resolution precedence, dry-run behaviour, and .suite metadata format. I have a script that cover a large number of combinations to exercise the benchmark_runner as much as possible.
1 parent 3e3a92d commit 373fab7

22 files changed

Lines changed: 2874 additions & 179 deletions

File tree

‎Cargo.lock‎

Lines changed: 49 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎benchmarks/Cargo.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ mimalloc_extended = ["libmimalloc-sys/extended"]
4343
arrow = { workspace = true }
4444
async-trait = "0.1"
4545
bytes = { workspace = true }
46-
clap = { version = "4.6.0", features = ["derive", "env"] }
46+
clap = { version = "4.6.0", features = ["derive", "env", "string"] }
4747
criterion = { workspace = true, features = ["html_reports"] }
4848
datafusion = { workspace = true, default-features = true }
4949
datafusion-common = { workspace = true, default-features = true }
@@ -62,6 +62,7 @@ serde_json = { workspace = true }
6262
snmalloc-rs = { version = "0.7", optional = true }
6363
tokio = { workspace = true, features = ["rt-multi-thread", "parking_lot"] }
6464
tokio-util = { version = "0.7.17" }
65+
toml = "0.9.8"
6566

6667
[dev-dependencies]
6768
# `pool`/`arrow_buffer_pool` are enabled only for tests, so the benchmark

‎benchmarks/benches/sql.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,8 @@
2424
use clap::Parser;
2525
use criterion::{Criterion, criterion_group, criterion_main};
2626
use datafusion_benchmarks::sql_benchmark_runner::{
27-
BenchmarkFilter, SqlRunConfig, default_sql_benchmark_directory,
28-
run_criterion_benchmarks_impl,
27+
BenchmarkFilter, SqlRunConfig, default_criterion_replacements,
28+
default_sql_benchmark_directory, run_criterion_benchmarks_impl,
2929
};
3030
use datafusion_benchmarks::util::CommonOpt;
3131
use datafusion_common::instant::Instant;
@@ -84,6 +84,8 @@ pub fn sql(c: &mut Criterion) {
8484
subgroup: args.subgroup,
8585
query: args.query,
8686
},
87+
replacements: default_criterion_replacements(),
88+
query_filename: None,
8789
persist_results: args.persist_results,
8890
validate_results: args.validate,
8991
output: None,

‎benchmarks/sql_benchmarks/README.md‎

Lines changed: 96 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -43,24 +43,96 @@ in the community:
4343
| `tpcds` | TPC‑DS queries |
4444
| `tpch` | TPC‑H queries |
4545
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
46-
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`BENCH_SUBGROUP`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Toggle a system under test with its native `DATAFUSION_*` env var |
46+
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Configure the system under test through its DataFusion settings. |
4747

4848
# Running Benchmarks
4949

50-
The easiest way to run a benchmark is to use the `bench.sh` shell script (up one level from this document)
51-
as it takes care of configuring any required environment variables and can populate any required data files.
52-
However, it is possible to directly run a sql benchmark using the `cargo bench` command. For example:
50+
Use `benchmark_runner` to run SQL benchmarks. It reads each suite's `.suite`
51+
file and exposes the suite's configuration as command-line options. Use the
52+
`bench.sh` shell script one level above this directory to download or generate
53+
required data files.
5354

5455
```shell
55-
BENCH_NAME=tpch cargo bench --bench sql
56+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
5657
```
5758

59+
## SQL benchmark runner
60+
61+
The `benchmark_runner` binary discovers suites from this directory and exposes
62+
suite-specific options alongside the common benchmark options. The suite name
63+
must come before all options.
64+
65+
```bash
66+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- --list
67+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --help
68+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --query 15 --format csv
69+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench --partitioning partitioned --dry-run
70+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --query 1 --result-mode persist
71+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --query 1 --result-mode validate
72+
```
73+
74+
Use `--path PATH` or `-p PATH` to override `DATA_DIR` for a suite that declares
75+
that path replacement. Suites without a `DATA_DIR` replacement reject the
76+
option. Suite-specific values follow this precedence: command-line option,
77+
environment variable, then the default in the suite metadata.
78+
79+
`--dry-run` prints the resolved suite, filters, run mode, common options,
80+
suite-specific values, and path replacements as JSON. It reports source metadata
81+
for suite options and path replacements. It validates the command but does not
82+
load benchmark definitions, create a session, read datasets, execute SQL, or
83+
write benchmark results.
84+
85+
Use `--result-mode persist` to save query results or `--result-mode validate` to
86+
compare them with saved results. The default, `--result-mode none`, does neither.
87+
For compatibility with direct Criterion runs, the runner also reads
88+
`BENCH_PERSIST_RESULTS` and `BENCH_VALIDATE`. Persistence takes precedence when
89+
both variables are `true`. An explicit `--result-mode` overrides both variables.
90+
91+
### Suite metadata
92+
93+
Each discoverable suite has one TOML metadata file named
94+
`<suite>/<suite>.suite`. The runner accepts these top-level fields:
95+
96+
| Field | Required | Description |
97+
|-------|----------|-------------|
98+
| `description` | Yes | Non-empty text shown by `--list` and suite help. |
99+
| `query_pattern` | No | Relative benchmark filename pattern. It must contain exactly one `{QUERY_ID}` or `{QUERY_ID_PADDED}` placeholder and defaults to `q{QUERY_ID_PADDED}.benchmark`. |
100+
| `path_replacements` | No | Map of replacement names to paths. Relative paths resolve from the suite directory. `DATA_DIR` enables `--path/-p`. |
101+
| `options` | No | Array of suite-specific option tables described below. |
102+
| `examples` | No | Array of `command` and `description` pairs appended to suite help. Both values must contain text. |
103+
104+
Each `[[options]]` table has these fields:
105+
106+
| Field | Required | Description |
107+
|-------|----------|-------------|
108+
| `name` | Yes | Long option name without `--`; use lowercase ASCII letters, digits, and hyphens. |
109+
| `short` | No | One ASCII letter or digit without `-`. |
110+
| `env` | Yes | Environment variable that supplies the option value. |
111+
| `default` | Yes | Value used when neither the command line nor the environment supplies one. |
112+
| `values` | No | Accepted values. Omit the field to allow any value. Include `"..."` to allow the listed values plus any other value. Without `"..."`, the list is closed. |
113+
| `help` | Yes | Non-empty text shown in suite help. |
114+
115+
Option names, short names, and environment keys must be unique within a suite.
116+
An option environment key cannot also appear in `path_replacements`. Suite
117+
options cannot reuse the runner's global names: `help` (`-h`), `query` (`-q`),
118+
`subgroup`, `iterations` (`-i`), `partitions` (`-n`), `batch-size` (`-s`),
119+
`mem-pool-type`, `memory-limit`, `sort-spill-reservation-bytes`, `debug` (`-d`),
120+
`simulate-latency`, `criterion`, `list`, `output` (`-o`), `save-baseline`,
121+
`path` (`-p`), `result-mode`, or `dry-run`.
122+
58123
# Benchmark configuration
59124

60-
Sql benchmarks are configured via environment variables. Cargo's bench command and
61-
[criterion](https://github.com/criterion-rs/criterion.rs) (the underlying benchmark framework) have an unfortunate
62-
limitation in that custom command arguments cannot be passed into a benchmark. The alternative is to use environment
63-
variables to pass in arguments which is what is used here.
125+
`benchmark_runner` is the preferred interface for configuring and running SQL
126+
benchmarks. Run `<suite> --help` to see the common and suite-specific options:
127+
128+
```shell
129+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- h2o --help
130+
```
131+
132+
The runner maps suite options to the environment variables below for
133+
compatibility with benchmark files and direct Criterion runs. Direct
134+
`cargo bench --bench sql` invocations cannot accept custom arguments, so they
135+
still use environment variables.
64136

65137
The SQL benchmarking tool uses the following environment variables:
66138

@@ -76,10 +148,10 @@ The SQL benchmarking tool uses the following environment variables:
76148
| MEM_POOL_TYPE | The memory pool type to use, should be one of "fair" or "greedy". |
77149
| MEMORY_LIMIT | Memory limit (e.g. '100M', '1.5G'). If not specified, run all pre-defined memory limits for given query if there's any, otherwise run with no memory limit. |
78150

79-
Example – Run the H2O window benchmarks on the 'small' sized CSV data files:
151+
Example: run the H2O window benchmarks on the small CSV data files:
80152

81-
``` bash
82-
BENCH_NAME=h2o BENCH_SUBGROUP=window H2O_BENCH_SIZE=small H20_FILE_TYPE=csv cargo bench --bench sql
153+
```shell
154+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- h2o --subgroup window --size small --format csv
83155
```
84156

85157
Some benchmarks use custom environment variables as outlined below:
@@ -101,21 +173,22 @@ Some benchmarks use custom environment variables as outlined below:
101173

102174
## How it works
103175

104-
SQL benchmarks are run via cargo's bench command using [criterion](https://docs.rs/criterion/latest/criterion/)
105-
for running and gathering statistics of each sql being benchmarked.
176+
The runner executes SQL benchmarks with its basic runner by default. Pass
177+
`--criterion` to gather statistics with
178+
[Criterion](https://docs.rs/criterion/latest/criterion/).
106179

107180
Each individual benchmark is represented by a `<name>.benchmark` file that contains a number of directives instructing
108181
the tool on how to load data, run initializations, run assertions, run the benchmark, optionally persist and
109182
validate results, and finally run any cleanup if required.
110183

111-
Variables are supported in two forms:
184+
Benchmark files support replacement variables in two forms:
112185

113-
* string substitution based on environment variables (with default values if unset): \${ENV_VAR} and
186+
* string substitution with an optional default: \${ENV_VAR} and
114187
\${ENV_VAR:-default}.
115-
* if / else based on whether an environment variable is true or not
188+
* if / else based on whether a replacement value is true or not
116189
(\${ENV_VAR:-default|true value|false value}). In this form only the value `true` (case-insensitive) selects the
117-
true branch; any other set value selects the false branch. If ENV_VAR is unset, the valud of `default` is used to
118-
* select the branch.
190+
true branch; any other supplied value selects the false branch. If the value is absent, the parser uses `default` to
191+
select the branch.
119192

120193
Comments in files are supported with lines starting with # or --.
121194

@@ -157,8 +230,8 @@ The above showcases the use of defaults for variables: `${NAME:-default}`
157230
The name of the benchmark. This will be used as part of the display name used by criterion.<br/><br/>Example:<br/>
158231
<blockquote>name Q${QUERY_NUMBER_PADDED}</blockquote>
159232

160-
The `name` directive also makes the value available to benchmark-file replacements as `BENCH_NAME`. This is separate
161-
from the `BENCH_NAME` environment variable used to select which benchmark group to run.
233+
The `name` directive also makes the value available to benchmark-file replacements as `BENCH_NAME`. This value is
234+
separate from the suite name passed to `benchmark_runner`.
162235

163236
</td>
164237
</tr>
@@ -221,8 +294,8 @@ The run directive called during execution of the benchmark. If a path to a file
221294
the run directive that path will be parsed and any sql statements in that file will be executed during the benchmark
222295
run. If no path is specified the next line is required to be the sql statement to execute. <br/><br/> Multiple
223296
statements are allowed within a single run directive, however a benchmark file may contain only one run directive. When
224-
running with `BENCH_PERSIST_RESULTS` or `BENCH_VALIDATE`, only the last `SELECT` or `WITH` statement from that run
225-
directive will be used for comparison. <br/><br/> The run directive (including any following sql statement) must be
297+
when persisting or validating results, only the last `SELECT` or `WITH` statement from that run directive will be used
298+
for comparison. <br/><br/> The run directive (including any following sql statement) must be
226299
followed by a blank line.<br/><br/>Example:<br/>
227300
<blockquote>run sql_benchmarks/imdb/queries/${QUERY_NUMBER_PADDED}.sql</blockquote>
228301

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
description = "ClickBench analytics queries over the hits dataset"
2+
3+
query_pattern = "q{QUERY_ID_PADDED}.benchmark"
4+
5+
[path_replacements]
6+
DATA_DIR = "../../data"
7+
8+
[[options]]
9+
name = "partitioning"
10+
env = "CLICKBENCH_TYPE"
11+
default = "single"
12+
values = ["single", "partitioned"]
13+
help = "Selects the single-file or partitioned ClickBench dataset."
14+
15+
[[examples]]
16+
command = "cargo run --release --bin benchmark_runner -- clickbench"
17+
description = "Run all ClickBench queries against the single-file dataset."
18+
19+
[[examples]]
20+
command = "cargo run --release --bin benchmark_runner -- clickbench --query 7"
21+
description = "Run ClickBench query 7."
22+
23+
[[examples]]
24+
command = "cargo run --release --bin benchmark_runner -- clickbench --partitioning partitioned"
25+
description = "Run all ClickBench queries against the partitioned dataset."
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
description = "Extended ClickBench queries over the hits dataset"
2+
3+
query_pattern = "q{QUERY_ID_PADDED}.benchmark"
4+
5+
[path_replacements]
6+
DATA_DIR = "../../data"
7+
8+
[[options]]
9+
name = "partitioning"
10+
env = "CLICKBENCH_TYPE"
11+
default = "single"
12+
values = ["single", "partitioned"]
13+
help = "Selects the single-file or partitioned ClickBench dataset."
14+
15+
[[examples]]
16+
command = "cargo run --release --bin benchmark_runner -- clickbench_extended"
17+
description = "Run all extended ClickBench queries against the single-file dataset."
18+
19+
[[examples]]
20+
command = "cargo run --release --bin benchmark_runner -- clickbench_extended --query 4 --partitioning partitioned"
21+
description = "Run extended ClickBench query 4 against the partitioned dataset."
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
description = "ClickBench query over a pre-sorted hits dataset"
2+
3+
query_pattern = "q{QUERY_ID_PADDED}.benchmark"
4+
5+
[path_replacements]
6+
DATA_DIR = "../../data"
7+
8+
[[options]]
9+
name = "sort-column"
10+
env = "SORTED_BY"
11+
default = "EventTime"
12+
values = ["EventTime", "..."]
13+
help = "Selects the column used to sort the ClickBench data."
14+
15+
[[options]]
16+
name = "sort-order"
17+
env = "SORTED_ORDER"
18+
default = "ASC"
19+
values = ["ASC", "DESC"]
20+
help = "Selects the sort direction for the ClickBench data."
21+
22+
[[examples]]
23+
command = "cargo run --release --bin benchmark_runner -- clickbench_sorted"
24+
description = "Run the sorted ClickBench query ordered by EventTime ascending."
25+
26+
[[examples]]
27+
command = "cargo run --release --bin benchmark_runner -- clickbench_sorted --sort-column UserID --sort-order DESC"
28+
description = "Run the query over data sorted by UserID descending."

0 commit comments

Comments
 (0)