Repository navigation
eliminate double copy in Top-K, hash join, and sort-merge join using BatchCoalescer::push_batch_interleaved #25515
Description
Several DataFusion operators share the same two-copy pattern when producing output:
- Copy 1 : interleave_record_batch (or build_batch_from_indices) gathers selected rows from multiple source batches into a new fully-materialized RecordBatch.
- Copy 2 : BatchCoalescer::push_batch copies those rows into the coalescer's internal column builders.
Prerequisite: apache/arrow-rs#11141
Affected call sites
┌──────────────────┬────────────────────┬───────────────────────────────────────┬────────┐
│ Operator │ File │ Current pattern │ Copies │
├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤
│ Top-K │ topk/mod.rs:1016 │ interleave_record_batch → push_batch │ 2 │
├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤
│ Top-K rank │ topk/mod.rs:1875 │ same │ 2 │
├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤
│ Hash join │ hash_join/… │ build_batch_from_indices → push_batch │ 2 │
├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤
│ Nested loop join │ nested_loop_join/… │ interleave_record_batch → push_batch │ 2 │
├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤
│ Sort merge join │ sort_merge_join/… │ materialization step → push_batch │ 2 │
└──────────────────┴────────────────────┴───────────────────────────────────────┴────────┘
Activity