Repository navigation
Commit c922f88
authored
perf: stop redoing per-chunk setup on the hash join probe path (apache#25658)
## Which issue does this PR close?
- Closes apache#12131. The fused index compare that issue asked for landed in
apache#23843; this picks up the per-chunk allocations it left behind and
removes the stale note that still pointed at the old code.
## Rationale for this change
Three pieces of per-chunk work on the hash join probe path repeat work
whose inputs do not change within a probe batch. The multi-column key
comparator was rebuilt for every output chunk, and building it scans the
entire build side for negative zero when a key is a float. The
integer-map path cloned both index buffers per chunk. Every output batch
deep-cloned the output schema, metadata included, to wrap it in a new
`Arc`.
## What changes are included in this PR?
- The key comparator is built once per probe batch, cached on the
stream, and cleared when the next batch arrives or the stream stops.
- The integer-map path takes and returns its index buffers instead of
cloning them.
- `build_batch_from_indices` and `build_batch_empty_build_side` accept a
`SchemaRef` and clone the `Arc`.
Two-column join on `(l_orderkey, CAST(l_quantity AS DOUBLE))` over TPC-H
SF1 with a 60K row build side and 17M output rows, median of five: 150
ms before, 108 ms after; `HashJoinExec` `elapsed_compute` 147 ms to 104
ms. Low-fanout float joins, the `hj` benchmark queries 23 to 25, and
TPC-H q7 and q9 stay within run-to-run noise.
## What is the testing strategy for this PR?
New tests in `datafusion/physical-plan/src/joins/hash_join`:
`join_inner_multi_key_float_zero_and_nulls_across_chunks`,
`join_inner_multi_key_across_probe_batches` (fails if the per-batch
comparator reset is removed), and `join_inner_array_map_across_chunks`.
Existing join tests and the `join` sqllogictest files pass unchanged.
## Are there any user-facing changes?
No.1 parent a138ce4 commit c922f88
4 files changed
Lines changed: 450 additions & 25 deletions
0 commit comments