Skip to content

Commit 6e8372f

Browse files
bench: add ASOF join benchmark with a large unfiltered right input (apache#26483)
## Which issue does this PR close? - Part of apache#26478. ## Rationale for this change The existing ASOF join benchmarks don't stress the broadcast right input. The largest right inputs that reach the join unfiltered have 1M rows (q02, q05, q06), and q09/q10 start from 50M right rows, but their key predicates prune the right input to one or two groups before it is sorted and broadcast. This adds that workload as a baseline for apache#26478. On main, the ASOF join keeps the whole right input in memory that can't be spilled, so the query needs at least that much memory to finish. With `--partitions 4`, Q11 completes with a 600 MB memory pool (peak 543 MB) and fails at 500 MB, where the join's `AsOfJoinInput` reservation holds 457.8 MB, which is the entire right input (20M rows x 3 Int64 columns). ## What changes are included in this PR? - `benchmarks/sql_benchmarks/asof_join/benchmarks/q11.benchmark`: 1M left rows, 20M right rows, 10K balanced equality groups, integer payloads, both sides covering the same time range, inputs unsorted so the required sorts are included. - An example entry for Q11 in `asof_join.suite`. ## What is the testing strategy for this PR? This PR only adds a benchmark, so there are no new tests. The benchmark checks that the plan contains `AsOfJoinExec`. I ran it locally and it returns 1,000,000 rows (one per left row): cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- asof_join --query 11 Local timings (i7-13700KF, WSL2, rough): about 4.0 s with the default 24 partitions, 2.9 s with `--partitions 4`, and 5.6 s with `--partitions 1`. ## Are there any user-facing changes? No.
1 parent 6131a98 commit 6e8372f

2 files changed

Lines changed: 30 additions & 0 deletions

File tree

‎benchmarks/sql_benchmarks/asof_join/asof_join.suite‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,7 @@ description = "Run the equality-key filter ASOF join query."
3333
[[examples]]
3434
command = "cargo run --release --bin benchmark_runner -- asof_join --query 10"
3535
description = "Run the compound equality-key filter ASOF join query."
36+
37+
[[examples]]
38+
command = "cargo run --release --bin benchmark_runner -- asof_join --query 11"
39+
description = "Run the large unfiltered right-input ASOF join query."
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name Q11
2+
group asof_join
3+
4+
expect_plan AsOfJoinExec
5+
6+
run
7+
-- Left: 1M rows; right: 20M rows; 10K balanced equality groups; integer payloads.
8+
-- Both sides cover the same time range and the right input is unfiltered, so the
9+
-- whole right input is collected and scanned by every left partition.
10+
-- Inputs are not ordered by (key, ts), so required ordering is included.
11+
WITH left_input AS (
12+
SELECT value % 10000 AS key,
13+
(value / 10000) * 20 + 1 AS ts,
14+
value AS payload
15+
FROM range(1000000)
16+
),
17+
right_input AS (
18+
SELECT value % 10000 AS key,
19+
value / 10000 AS ts,
20+
value AS payload
21+
FROM range(20000000)
22+
)
23+
SELECT l.key, l.ts, l.payload, r.payload AS right_payload
24+
FROM left_input l
25+
ASOF JOIN right_input r MATCH_CONDITION (l.ts >= r.ts)
26+
ON l.key = r.key;

0 commit comments

Comments
 (0)