Repository navigation
Commit 6e8372f
authored
bench: add ASOF join benchmark with a large unfiltered right input (apache#26483)
## Which issue does this PR close?
- Part of apache#26478.
## Rationale for this change
The existing ASOF join benchmarks don't stress the broadcast right
input. The largest right
inputs that reach the join unfiltered have 1M rows (q02, q05, q06), and
q09/q10 start from
50M right rows, but their key predicates prune the right input to one or
two groups before
it is sorted and broadcast. This adds that workload as a baseline for
apache#26478.
On main, the ASOF join keeps the whole right input in memory that can't
be spilled, so the
query needs at least that much memory to finish. With `--partitions 4`,
Q11 completes with a
600 MB memory pool (peak 543 MB) and fails at 500 MB, where the join's
`AsOfJoinInput`
reservation holds 457.8 MB, which is the entire right input (20M rows x
3 Int64 columns).
## What changes are included in this PR?
- `benchmarks/sql_benchmarks/asof_join/benchmarks/q11.benchmark`: 1M
left rows, 20M right rows,
10K balanced equality groups, integer payloads, both sides covering the
same time range,
inputs unsorted so the required sorts are included.
- An example entry for Q11 in `asof_join.suite`.
## What is the testing strategy for this PR?
This PR only adds a benchmark, so there are no new tests. The benchmark
checks that the plan
contains `AsOfJoinExec`. I ran it locally and it returns 1,000,000 rows
(one per left row):
cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
asof_join --query 11
Local timings (i7-13700KF, WSL2, rough): about 4.0 s with the default 24
partitions,
2.9 s with `--partitions 4`, and 5.6 s with `--partitions 1`.
## Are there any user-facing changes?
No.1 parent 6131a98 commit 6e8372f
2 files changed
Lines changed: 30 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
0 commit comments