Skip to content

Commit c922f88

Browse files
authored
perf: stop redoing per-chunk setup on the hash join probe path (apache#25658)
## Which issue does this PR close? - Closes apache#12131. The fused index compare that issue asked for landed in apache#23843; this picks up the per-chunk allocations it left behind and removes the stale note that still pointed at the old code. ## Rationale for this change Three pieces of per-chunk work on the hash join probe path repeat work whose inputs do not change within a probe batch. The multi-column key comparator was rebuilt for every output chunk, and building it scans the entire build side for negative zero when a key is a float. The integer-map path cloned both index buffers per chunk. Every output batch deep-cloned the output schema, metadata included, to wrap it in a new `Arc`. ## What changes are included in this PR? - The key comparator is built once per probe batch, cached on the stream, and cleared when the next batch arrives or the stream stops. - The integer-map path takes and returns its index buffers instead of cloning them. - `build_batch_from_indices` and `build_batch_empty_build_side` accept a `SchemaRef` and clone the `Arc`. Two-column join on `(l_orderkey, CAST(l_quantity AS DOUBLE))` over TPC-H SF1 with a 60K row build side and 17M output rows, median of five: 150 ms before, 108 ms after; `HashJoinExec` `elapsed_compute` 147 ms to 104 ms. Low-fanout float joins, the `hj` benchmark queries 23 to 25, and TPC-H q7 and q9 stay within run-to-run noise. ## What is the testing strategy for this PR? New tests in `datafusion/physical-plan/src/joins/hash_join`: `join_inner_multi_key_float_zero_and_nulls_across_chunks`, `join_inner_multi_key_across_probe_batches` (fails if the per-batch comparator reset is removed), and `join_inner_array_map_across_chunks`. Existing join tests and the `join` sqllogictest files pass unchanged. ## Are there any user-facing changes? No.
1 parent a138ce4 commit c922f88

4 files changed

Lines changed: 450 additions & 25 deletions

File tree

0 commit comments

Comments
 (0)