You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6ba3654
Browse filesBrowse the repository at this point in the historyBrowse files
Add support for running sql benchmarks with command line arguments (apache#23772)
## Which issue does this PR close?
- Part of apache#21706 and apache#21937
## Rationale for this change
SQL benchmark suites expose configuration through environment variables.
Contributors must consult the source or documentation to discover
suite-specific settings, and Cargo does not forward custom arguments to
benchmark targets.
This change lets `benchmark_runner` expose those settings as validated
command-line arguments. It also adds suite-specific help and a dry-run
mode for inspecting resolved configuration without executing a
benchmark.
## What changes are included in this PR?
This PR adds TOML `.suite` metadata files that define each SQL benchmark
suite’s:
- Description and help examples.
- Suite-specific options, defaults, environment variables, and accepted
values.
- Query filename pattern.
- Data-path replacements.
`benchmark_runner` uses this metadata to:
- Generate suite-specific command-line arguments and help output.
- Resolve suite options using command-line, environment-variable, and
default-value precedence.
- Support `--path` for suites that declare a `DATA_DIR` replacement.
- Support `--result-mode` for result persistence and validation.
- Validate suite names, metadata, option conflicts, query identifiers,
and incompatible arguments.
- Provide `--dry-run` JSON output containing the resolved options,
paths, value sources, filters, and execution mode without loading
benchmark
definitions or executing SQL.
The existing basic runner, Criterion mode, and suite-listing
functionality remain in place.
## Are these changes tested?
Yes. New tests cover:
- Suite metadata parsing and validation.
- Dynamic suite options and help output.
- Option precedence between command-line arguments, environment
variables, and defaults.
- Path and query filename resolution.
- Dry-run output and validation.
- Result-mode resolution.
- Invalid option combinations and malformed metadata.
## Are there any user-facing changes?
Yes. Contributors can configure suite-specific settings through
command-line arguments and inspect them through suite help:
```shell
cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
tpch --help
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
tpch --query 15 --format csv --path /path/to/tpch
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
clickbench --partitioning partitioned --dry-run
```
Environment variables remain supported for compatibility with direct cargo bench invocations and existing benchmark definitions.
The SQL benchmark README documents the command-line options, resolution precedence, dry-run behaviour, and .suite metadata format.
I have a script that cover a large number of combinations to exercise the benchmark_runner as much as possible.
Copy file name to clipboardExpand all lines: benchmarks/sql_benchmarks/README.md
+96-23Lines changed: 96 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -43,24 +43,96 @@ in the community:
43
43
|`tpcds`| TPC‑DS queries |
44
44
|`tpch`| TPC‑H queries |
45
45
|`wide_schema`| Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
46
-
|`predicate_eval`| Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`BENCH_SUBGROUP`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Toggle a system under test with its native `DATAFUSION_*` env var|
46
+
|`predicate_eval`| Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Configure the system under test through its DataFusion settings.|
47
47
48
48
# Running Benchmarks
49
49
50
-
The easiest way to run a benchmark is to use the `bench.sh` shell script (up one level from this document)
51
-
as it takes care of configuring any required environment variables and can populate any required data files.
52
-
However, it is possible to directly run a sql benchmark using the `cargo bench` command. For example:
50
+
Use `benchmark_runner` to run SQL benchmarks. It reads each suite's `.suite`
51
+
file and exposes the suite's configuration as command-line options. Use the
52
+
`bench.sh` shell script one level above this directory to download or generate
53
+
required data files.
53
54
54
55
```shell
55
-
BENCH_NAME=tpch cargo bench --bench sql
56
+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch
56
57
```
57
58
59
+
## SQL benchmark runner
60
+
61
+
The `benchmark_runner` binary discovers suites from this directory and exposes
62
+
suite-specific options alongside the common benchmark options. The suite name
63
+
must come before all options.
64
+
65
+
```bash
66
+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- --list
67
+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --help
Use `--path PATH` or `-p PATH` to override `DATA_DIR` for a suite that declares
75
+
that path replacement. Suites without a `DATA_DIR` replacement reject the
76
+
option. Suite-specific values follow this precedence: command-line option,
77
+
environment variable, then the default in the suite metadata.
78
+
79
+
`--dry-run` prints the resolved suite, filters, run mode, common options,
80
+
suite-specific values, and path replacements as JSON. It reports source metadata
81
+
for suite options and path replacements. It validates the command but does not
82
+
load benchmark definitions, create a session, read datasets, execute SQL, or
83
+
write benchmark results.
84
+
85
+
Use `--result-mode persist` to save query results or `--result-mode validate` to
86
+
compare them with saved results. The default, `--result-mode none`, does neither.
87
+
For compatibility with direct Criterion runs, the runner also reads
88
+
`BENCH_PERSIST_RESULTS` and `BENCH_VALIDATE`. Persistence takes precedence when
89
+
both variables are `true`. An explicit `--result-mode` overrides both variables.
90
+
91
+
### Suite metadata
92
+
93
+
Each discoverable suite has one TOML metadata file named
94
+
`<suite>/<suite>.suite`. The runner accepts these top-level fields:
95
+
96
+
| Field | Required | Description |
97
+
|-------|----------|-------------|
98
+
|`description`| Yes | Non-empty text shown by `--list` and suite help. |
99
+
|`query_pattern`| No | Relative benchmark filename pattern. It must contain exactly one `{QUERY_ID}` or `{QUERY_ID_PADDED}` placeholder and defaults to `q{QUERY_ID_PADDED}.benchmark`. |
100
+
|`path_replacements`| No | Map of replacement names to paths. Relative paths resolve from the suite directory. `DATA_DIR` enables `--path/-p`. |
101
+
|`options`| No | Array of suite-specific option tables described below. |
102
+
|`examples`| No | Array of `command` and `description` pairs appended to suite help. Both values must contain text. |
103
+
104
+
Each `[[options]]` table has these fields:
105
+
106
+
| Field | Required | Description |
107
+
|-------|----------|-------------|
108
+
|`name`| Yes | Long option name without `--`; use lowercase ASCII letters, digits, and hyphens. |
109
+
|`short`| No | One ASCII letter or digit without `-`. |
110
+
|`env`| Yes | Environment variable that supplies the option value. |
111
+
|`default`| Yes | Value used when neither the command line nor the environment supplies one. |
112
+
|`values`| No | Accepted values. Omit the field to allow any value. Include `"..."` to allow the listed values plus any other value. Without `"..."`, the list is closed. |
113
+
|`help`| Yes | Non-empty text shown in suite help. |
114
+
115
+
Option names, short names, and environment keys must be unique within a suite.
116
+
An option environment key cannot also appear in `path_replacements`. Suite
117
+
options cannot reuse the runner's global names: `help` (`-h`), `query` (`-q`),
Sql benchmarks are configured via environment variables. Cargo's bench command and
61
-
[criterion](https://github.com/criterion-rs/criterion.rs) (the underlying benchmark framework) have an unfortunate
62
-
limitation in that custom command arguments cannot be passed into a benchmark. The alternative is to use environment
63
-
variables to pass in arguments which is what is used here.
125
+
`benchmark_runner` is the preferred interface for configuring and running SQL
126
+
benchmarks. Run `<suite> --help` to see the common and suite-specific options:
127
+
128
+
```shell
129
+
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- h2o --help
130
+
```
131
+
132
+
The runner maps suite options to the environment variables below for
133
+
compatibility with benchmark files and direct Criterion runs. Direct
134
+
`cargo bench --bench sql` invocations cannot accept custom arguments, so they
135
+
still use environment variables.
64
136
65
137
The SQL benchmarking tool uses the following environment variables:
66
138
@@ -76,10 +148,10 @@ The SQL benchmarking tool uses the following environment variables:
76
148
| MEM_POOL_TYPE | The memory pool type to use, should be one of "fair" or "greedy". |
77
149
| MEMORY_LIMIT | Memory limit (e.g. '100M', '1.5G'). If not specified, run all pre-defined memory limits for given query if there's any, otherwise run with no memory limit. |
78
150
79
-
Example – Run the H2O window benchmarks on the 'small' sized CSV data files:
151
+
Example: run the H2O window benchmarks on the small CSV data files:
cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- h2o --subgroup window --size small --format csv
83
155
```
84
156
85
157
Some benchmarks use custom environment variables as outlined below:
@@ -101,21 +173,22 @@ Some benchmarks use custom environment variables as outlined below:
101
173
102
174
## How it works
103
175
104
-
SQL benchmarks are run via cargo's bench command using [criterion](https://docs.rs/criterion/latest/criterion/)
105
-
for running and gathering statistics of each sql being benchmarked.
176
+
The runner executes SQL benchmarks with its basic runner by default. Pass
0 commit comments