Skip to content

Commit 88fb94d

Browse files
Add sorted TopK TPC-H benchmark target (#23003)
## Which issue does this PR close? This PR adds a first-class benchmark target for an existing `sort-tpch` mode. ## Rationale for this change `dfbench sort-tpch` already supports TopK queries over input declared as sorted with `--sorted --limit`, but `bench.sh` only exposed the unsorted TopK wrapper as `topk_tpch`. That made the sorted TopK path harder to run from the benchmark bot and easier to miss during performance work. Adding `topk_sorted_tpch` gives a named target for the sorted-input TopK case: ```bash ./benchmarks/bench.sh run topk_sorted_tpch ``` The new target uses `--limit 100` so it is the sorted counterpart to the existing `topk_tpch` benchmark. ## What changes are included in this PR? - Adds `topk_sorted_tpch` to the benchmark script help text. - Reuses the existing TPC-H SF1 parquet data setup. - Adds a `run_topk_sorted_tpch` wrapper around `dfbench sort-tpch --sorted --limit 100`. - Writes results to `run_topk_sorted_tpch.json`. - Documents the new benchmark target in `benchmarks/README.md`. ## Are these changes tested? Validated with: ```bash bash -n benchmarks/bench.sh CARGO_COMMAND=echo DATA_DIR=/tmp/df-topk-bench-data RESULTS_NAME=topk_sorted_tpch_smoke ./benchmarks/bench.sh run topk_sorted_tpch git diff --check cargo fmt --all cargo clippy --all-targets --all-features -- -D warnings ``` The smoke run verified that the script dispatches to: ```bash dfbench sort-tpch --iterations 5 --path ... --sorted --limit 100 ``` ## Are there any user-facing changes? No engine or API behavior changes. This only adds a new opt-in benchmark target. --------- Co-authored-by: Gabriel <[email protected]>
1 parent 62b3b62 commit 88fb94d

3 files changed

Lines changed: 35 additions & 5 deletions

File tree

‎benchmarks/README.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -558,7 +558,15 @@ Test performance of end-to-end sort SQL queries. (While the `Sort` benchmark foc
558558

559559
Sort integration benchmark runs whole table sort queries on TPCH `lineitem` table, with different characteristics. For example, different number of sort keys, different sort key cardinality, different number of payload columns, etc.
560560

561-
If the TPCH tables have been converted as sorted on their first column (see [Sorted Conversion](#sorted-conversion)), you can use the `--sorted` flag to indicate that the input data is pre-sorted, allowing DataFusion to leverage that order during query execution.
561+
The `--sorted` flag does not sort or rewrite the input files. It declares that the `lineitem` Parquet input is already sorted ascending by its first column (`l_orderkey`). DataFusion can then leverage that ordering during query execution.
562+
563+
To generate the expected TPC-H SF=1 Parquet input for this benchmark, run:
564+
565+
```bash
566+
./bench.sh data tpch
567+
```
568+
569+
For the `lineitem` table used by `sort-tpch`, this uses `tpchgen-cli` to generate Parquet data that is already ordered by `l_orderkey`. If you use a different input directory, only pass `--sorted` when the `lineitem` files already have that ordering.
562570

563571
Additionally, an optional `--limit` flag is available for the sort benchmark. When specified, this flag appends a `LIMIT n` clause to the SQL query, effectively converting the query into a TopK query. Combining the `--sorted` and `--limit` options enables benchmarking of TopK queries on pre-sorted inputs.
564572

@@ -578,7 +586,7 @@ See [`sort_tpch.rs`](src/sort_tpch.rs) for more details.
578586
cargo run --release --bin dfbench -- sort-tpch -p './datafusion/benchmarks/data/tpch_sf1' -o '/tmp/sort_tpch.json' --query 2
579587
```
580588

581-
3. Run all queries as TopK queries on presorted data:
589+
3. Run all queries as TopK queries on already sorted data:
582590

583591
```bash
584592
cargo run --release --bin dfbench -- sort-tpch --sorted --limit 10 -p './datafusion/benchmarks/data/tpch_sf1' -o '/tmp/sort_tpch.json'
@@ -598,6 +606,14 @@ In addition, topk_tpch is available from the bench.sh script:
598606
./bench.sh run topk_tpch
599607
```
600608

609+
To benchmark TopK queries on TPC-H `lineitem` input ordered by `l_orderkey`, use:
610+
611+
```bash
612+
./bench.sh run topk_sorted_tpch
613+
```
614+
615+
This runs `dfbench sort-tpch --sorted --limit 100` through the benchmark script, using `--sorted` to declare the existing `l_orderkey` ordering.
616+
601617
## IMDB
602618

603619
Run Join Order Benchmark (JOB) on IMDB dataset.

‎benchmarks/bench.sh‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ tpcds: TPCDS inspired benchmark on Scale Factor (SF) 1 (~1GB),
9999
sort_tpch: Benchmark of sorting speed for end-to-end sort queries on TPC-H dataset (SF=1)
100100
sort_tpch10: Benchmark of sorting speed for end-to-end sort queries on TPC-H dataset (SF=10)
101101
topk_tpch: Benchmark of top-k (sorting with limit) queries on TPC-H dataset (SF=1)
102+
topk_sorted_tpch: Benchmark of top-k queries on TPC-H lineitem ordered by l_orderkey (SF=1)
102103
push_down_topk: Benchmark of ORDER BY ... LIMIT over outer joins on TPC-H dataset (SF=1) — exercises pushing TopK through a join
103104
external_aggr: External aggregation benchmark on TPC-H dataset (SF=1)
104105
wide_schema: Small-projection queries on a wide synthetic dataset (1024 cols × 256 files) — measures per-file metadata overhead
@@ -346,7 +347,7 @@ main() {
346347
# same data as for tpch10
347348
data_tpch "10" "parquet"
348349
;;
349-
topk_tpch)
350+
topk_tpch|topk_sorted_tpch)
350351
# same data as for tpch
351352
data_tpch "1" "parquet"
352353
;;
@@ -577,6 +578,9 @@ main() {
577578
topk_tpch)
578579
run_topk_tpch
579580
;;
581+
topk_sorted_tpch)
582+
run_topk_sorted_tpch
583+
;;
580584
push_down_topk)
581585
run_push_down_topk
582586
;;
@@ -1506,6 +1510,16 @@ run_topk_tpch() {
15061510
$CARGO_COMMAND --bin dfbench -- sort-tpch --iterations 5 --path "${TPCH_DIR}" -o "${RESULTS_FILE}" --limit 100 ${QUERY_ARG} ${LATENCY_ARG}
15071511
}
15081512

1513+
# Runs the sorted sort tpch integration benchmark with limit 100 (topk)
1514+
run_topk_sorted_tpch() {
1515+
TPCH_DIR="${DATA_DIR}/tpch_sf1"
1516+
RESULTS_FILE="${RESULTS_DIR}/run_topk_sorted_tpch.json"
1517+
echo "RESULTS_FILE: ${RESULTS_FILE}"
1518+
echo "Running sorted topk tpch benchmark..."
1519+
1520+
$CARGO_COMMAND --bin dfbench -- sort-tpch --iterations 5 --path "${TPCH_DIR}" -o "${RESULTS_FILE}" --sorted --limit 100 ${QUERY_ARG} ${LATENCY_ARG}
1521+
}
1522+
15091523
# Runs the nlj benchmark
15101524
run_nlj() {
15111525
RESULTS_FILE="${RESULTS_DIR}/nlj.json"

‎benchmarks/src/sort_tpch.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,8 @@ pub struct RunOpt {
6464
#[arg(short = 'm', long = "mem-table")]
6565
mem_table: bool,
6666

67-
/// Mark the first column of each table as sorted in ascending order.
68-
/// The tables should have been created with the `--sort` option for this to have any effect.
67+
/// Declare that the first column of the input table is already sorted in ascending order.
68+
/// This flag only attaches ordering metadata; it does not sort the input files.
6969
#[arg(short = 't', long = "sorted")]
7070
sorted: bool,
7171

0 commit comments

Comments
 (0)