You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
BatchCoalescer already has push_batch_with_indices(batch, indices: &dyn Array) for the single-batch take case. There is no equivalent for the multi-batch case.
Callers that need to gather rows from multiple source batches must today call interleave_record_batch (copy 1) and then push_batch the result (copy 2). The intermediate batch is immediately discarded.
Proposed API
/// Semantically equivalent to `interleave_record_batch(batches, indices)`/// followed by `push_batch`, without the intermediate RecordBatch allocation.pubfnpush_batch_interleaved(&mutself,batches:&[RecordBatch],indices:&[(usize,usize)],// (batch_idx, row_idx)) -> Result<(),ArrowError>
Top-K; TopKHeap::emit_with_state calls interleave_record_batch to reconstruct k surviving rows from the RecordBatchStore, then immediately pushes the result to a BatchCoalescer.
Hash join / nested loop join; build_batch_from_indices or interleave_record_batch to materialize matched row pairs (copy 1), then push to the coalescer (copy 2).
Sort merge join; join rows go through a materialization step before reaching the coalescer.
Activity
Rich-T-kid commentedon Sep 20, 2026
this is actually somewhat similar to #8957
Rich-T-kid commentedon Oct 12, 2026
take
Implement `BatchCoalescer::push_batch_interleaved` (apache#11519)