Skip to content

Commit 7ee8f52

Browse files
authored
refactor: consolidate benchmark options and environment settings (apache#26768)
## Which issue does this PR close? - Closes apache#21936. ## Rationale for this change Standard DataFusion environment settings can be silently overwritten by benchmark defaults. For example, `DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN=false` is overridden by the legacy runners' default CLI value and by TPC-H's initialization SQL. ## What changes are included in this PR? - Share statistics and join configuration overrides across the TPC-H, TPC-DS, and IMDB runners through `QueryOpt`. Apply only explicitly supplied CLI values after `SessionConfig::from_env()`. - Remove duplicate environment aliases for session settings and remove TPC-H initialization SQL and `bench.sh` arguments that override those settings. - Document the standard environment variables and migration. Memory limits and pool types retain their separate `RuntimeEnv` configuration. ## What is the testing strategy for this PR? - Add subprocess coverage for standard environment settings, explicit CLI precedence, obsolete aliases, and consistency with the shared SQL runner. - Add `tpch_init_preserves_session_settings`. Restoring the previous TPC-H initialization SQL makes this test fail on the overwritten join setting; restoring the fix makes it pass. - `env -u DEBUG cargo test --profile ci -p datafusion-benchmarks --lib --bins --tests`: 235 tests passed. The inherited `DEBUG=release` is removed because it conflicts with the benchmark's boolean debug option. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`, `bash -n benchmarks/bench.sh`, and `git diff --check` passed. - Run the existing NLJ Q01 benchmark on upstream main (`350ca7650`) and this branch with two partitions and three iterations each. Both return 100,000 rows every time. These are unoptimized CI-profile smoke runs, not a performance claim. ## Are there any user-facing changes? Yes. Existing CLI flags remain available. Explicit CLI values take precedence over standard DataFusion environment settings, followed by DataFusion defaults. Scripts using these environment aliases must migrate: | Removed alias | Replacement | | --- | --- | | `PARTITIONS` | `DATAFUSION_EXECUTION_TARGET_PARTITIONS` | | `BATCH_SIZE` | `DATAFUSION_EXECUTION_BATCH_SIZE` | | `SORT_SPILL_RESERVATION_BYTES` | `DATAFUSION_EXECUTION_SORT_SPILL_RESERVATION_BYTES` | | `PREFER_HASH_JOIN` | `DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN` | | `HASH_JOIN_BUFFERING_CAPACITY` | `DATAFUSION_EXECUTION_HASH_JOIN_BUFFERING_CAPACITY` | The benchmark READMEs include the updated configuration examples.
1 parent 672b811 commit 7ee8f52

32 files changed

Lines changed: 242 additions & 167 deletions

‎benchmarks/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ The benchmark runs with `prefer_hash_join == true` by default, which enforces HA
127127
To run TPCH benchmarks with join other than HASH:
128128

129129
```shell
130-
PREFER_HASH_JOIN=false ./bench.sh run tpch
130+
DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN=false ./bench.sh run tpch
131131
```
132132

133133
### Configure with environment variables

‎benchmarks/bench.sh‎

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,6 @@ DATAFUSION_DIR=${DATAFUSION_DIR:-$SCRIPT_DIR/..}
4242
DATA_DIR=${DATA_DIR:-$SCRIPT_DIR/data}
4343
CARGO_COMMAND=${CARGO_COMMAND:-"cargo run --release"}
4444
SQL_CARGO_COMMAND=${SQL_CARGO_COMMAND:-"cargo bench --bench sql"}
45-
PREFER_HASH_JOIN=${PREFER_HASH_JOIN:-true}
4645
SIMULATE_LATENCY=${SIMULATE_LATENCY:-false}
4746

4847
# Build latency arg based on SIMULATE_LATENCY setting
@@ -187,7 +186,6 @@ DATA_DIR directory to store datasets
187186
CARGO_COMMAND command that runs the benchmark binary
188187
DATAFUSION_DIR directory to use (default $DATAFUSION_DIR)
189188
RESULTS_NAME folder where the benchmark files are stored
190-
PREFER_HASH_JOIN Prefer hash join algorithm (default true)
191189
SIMULATE_LATENCY Serve local files as if they were remote object storage: per-request
192190
latency, per-connection bandwidth and paginated LIST (default false)
193191
DATAFUSION_* Set the given datafusion configuration
@@ -237,7 +235,6 @@ main() {
237235
echo "BENCHMARK: ${BENCHMARK}"
238236
echo "DATA_DIR: ${DATA_DIR}"
239237
echo "CARGO_COMMAND: ${CARGO_COMMAND}"
240-
echo "PREFER_HASH_JOIN: ${PREFER_HASH_JOIN}"
241238
echo "***************************"
242239
case "$BENCHMARK" in
243240
all)
@@ -480,7 +477,6 @@ main() {
480477
echo "DATA_DIR: ${DATA_DIR}"
481478
echo "RESULTS_DIR: ${RESULTS_DIR}"
482479
echo "CARGO_COMMAND: ${CARGO_COMMAND}"
483-
echo "PREFER_HASH_JOIN: ${PREFER_HASH_JOIN}"
484480
echo "SIMULATE_LATENCY: ${SIMULATE_LATENCY}"
485481
echo "***************************"
486482

@@ -846,7 +842,6 @@ run_tpch() {
846842
BENCH_RESULTS_FILE="$(sql_results_file tpch_sf${SCALE_FACTOR}_${FORMAT})" \
847843
BENCH_SIZE="${SCALE_FACTOR}" \
848844
DATA_DIR="${DATA_DIR}" \
849-
PREFER_HASH_JOIN="${PREFER_HASH_JOIN}" \
850845
TPCH_FILE_TYPE="${FORMAT}" \
851846
SIMULATE_LATENCY="${SIMULATE_LATENCY}" \
852847
${QUERY:+BENCH_QUERY="${QUERY}"} \
@@ -1058,7 +1053,6 @@ run_tpch_mem() {
10581053
BENCH_SIZE="${SCALE_FACTOR}" \
10591054
DATA_DIR="${DATA_DIR}" \
10601055
TPCH_FILE_TYPE="mem" \
1061-
PREFER_HASH_JOIN="${PREFER_HASH_JOIN}" \
10621056
SIMULATE_LATENCY="${SIMULATE_LATENCY}" \
10631057
${QUERY:+BENCH_QUERY="${QUERY}"} \
10641058
bash -c "$SQL_CARGO_COMMAND"
@@ -1081,7 +1075,7 @@ run_tpcds() {
10811075
echo "RESULTS_FILE: ${RESULTS_FILE}"
10821076
echo "Running tpcds benchmark..."
10831077

1084-
debug_run $CARGO_COMMAND --bin dfbench -- tpcds --iterations 5 --path "${TPCDS_DIR}" --query_path "../datafusion/core/tests/tpc-ds" --prefer_hash_join "${PREFER_HASH_JOIN}" -o "${RESULTS_FILE}" ${QUERY_ARG} ${LATENCY_ARG}
1078+
debug_run $CARGO_COMMAND --bin dfbench -- tpcds --iterations 5 --path "${TPCDS_DIR}" --query_path "../datafusion/core/tests/tpc-ds" -o "${RESULTS_FILE}" ${QUERY_ARG} ${LATENCY_ARG}
10851079
}
10861080

10871081
# Runs the compile profile benchmark helper
@@ -1298,7 +1292,7 @@ run_imdb() {
12981292
RESULTS_FILE="${RESULTS_DIR}/imdb.json"
12991293
echo "RESULTS_FILE: ${RESULTS_FILE}"
13001294
echo "Running imdb benchmark..."
1301-
debug_run $CARGO_COMMAND --bin imdb -- benchmark datafusion --iterations 5 --path "${IMDB_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" --format parquet -o "${RESULTS_FILE}" ${QUERY_ARG} ${LATENCY_ARG}
1295+
debug_run $CARGO_COMMAND --bin imdb -- benchmark datafusion --iterations 5 --path "${IMDB_DIR}" --format parquet -o "${RESULTS_FILE}" ${QUERY_ARG} ${LATENCY_ARG}
13021296
}
13031297

13041298
data_h2o() {

‎benchmarks/sql_benchmarks/README.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,31 @@ compatibility with benchmark files and direct Criterion runs. Direct
141141
`cargo bench --bench sql` invocations cannot accept custom arguments, so they
142142
still use environment variables.
143143

144+
DataFusion session settings are read through `SessionConfig::from_env()` in both
145+
CLI and direct Criterion runs. Use the standard `DATAFUSION_*` environment variables:
146+
147+
| Setting | Environment variable |
148+
| ------------------------------ | --------------------------------------------------- |
149+
| Parallel partitions | `DATAFUSION_EXECUTION_TARGET_PARTITIONS` |
150+
| Batch size | `DATAFUSION_EXECUTION_BATCH_SIZE` |
151+
| Sort spill reservation (bytes) | `DATAFUSION_EXECUTION_SORT_SPILL_RESERVATION_BYTES` |
152+
| Prefer hash joins | `DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN` |
153+
| Enable Piecewise Merge Join | `DATAFUSION_OPTIMIZER_ENABLE_PIECEWISE_MERGE_JOIN` |
154+
| Hash join buffering capacity | `DATAFUSION_EXECUTION_HASH_JOIN_BUFFERING_CAPACITY` |
155+
| Collect statistics | `DATAFUSION_EXECUTION_COLLECT_STATISTICS` |
156+
157+
Explicit CLI settings override environment settings; otherwise DataFusion's defaults
158+
apply. The old `PARTITIONS`, `BATCH_SIZE`, `SORT_SPILL_RESERVATION_BYTES`,
159+
`PREFER_HASH_JOIN`, and `HASH_JOIN_BUFFERING_CAPACITY` environment variables are
160+
no longer used. For example:
161+
162+
```shell
163+
DATAFUSION_EXECUTION_TARGET_PARTITIONS=2 DATAFUSION_OPTIMIZER_PREFER_HASH_JOIN=false cargo bench --bench sql
164+
```
165+
166+
Memory limits and pool types configure `RuntimeEnv` separately. `MEMORY_LIMIT`
167+
(or `--memory-limit`) takes precedence over `DATAFUSION_RUNTIME_MEMORY_LIMIT`.
168+
144169
The SQL benchmarking tool uses the following environment variables:
145170

146171
| Environment Variable | Description |
@@ -171,8 +196,6 @@ Some benchmarks use custom environment variables as outlined below:
171196
| H2O_FILE_TYPE | Used in the h2o benchmark to specify which file type to query against. The valid options are `csv` and `parquet` | `csv` |
172197
| CLICKBENCH_TYPE | The type of partitioning for the clickbench benchmark. Valid options are `single` and `partitioned` | `single` |
173198
| H2O_BENCH_SIZE | Used in the h2o benchmark. The valid options are `small`, `medium` and `big` | `small` |
174-
| PREFER_HASH_JOIN | Control datafusion's config option `datafusion.optimizer.prefer_hash_join` | true |
175-
| HASH_JOIN_BUFFERING_CAPACITY | Control datafusion's config option `datafusion.execution.hash_join_buffering_capacity` | 0 |
176199
| BENCH_SORTED | Used in the sort_tpch benchmark to indicate whether the lineitem table should be sorted. | false |
177200
| SORTED_BY | Used in the clickbench_sorted benchmark to indicate the column to sort by. | `EventTime` |
178201
| SORTED_ORDER | Used in the clickbench_sorted benchmark to indicate the sort order of the column. | `ASC` |
@@ -422,7 +445,7 @@ QUERY_NUMBER_PADDED=01
422445
The echo directive allows for echoing a string to stdout during the execution of the benchmark and may be useful for
423446
debugging.<br/><br/>Example:<br/>
424447
<blockquote>
425-
echo The value for batch size is ${BATCH_SIZE:-8192}
448+
echo The value for batch size is ${DATAFUSION_EXECUTION_BATCH_SIZE:-8192}
426449
</blockquote>
427450

428451
</td>

‎benchmarks/sql_benchmarks/tpch/benchmarks/q01.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q01
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

‎benchmarks/sql_benchmarks/tpch/benchmarks/q02.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q02
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

‎benchmarks/sql_benchmarks/tpch/benchmarks/q03.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q03
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

‎benchmarks/sql_benchmarks/tpch/benchmarks/q04.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q04
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

‎benchmarks/sql_benchmarks/tpch/benchmarks/q05.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q05
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

‎benchmarks/sql_benchmarks/tpch/benchmarks/q06.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q06
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

‎benchmarks/sql_benchmarks/tpch/benchmarks/q07.benchmark‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@ name Q07
22
group tpch
33
subgroup sf${BENCH_SIZE:-1}
44

5-
init sql_benchmarks/tpch/init/set_config.sql
6-
75
load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
86

97
assert I

0 commit comments

Comments
 (0)