Repository navigation
Optimize BatchCoalescer::push_batch_with_indices #8957
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We currently have BatchCoalescer:: push_batch_with_filter which can speed up filtering of multiple batches while targeting a batch_size, without extra copies / allocations (see #8951).
It would be nice to speed up BatchCoalescer:: push_batch_with_indices to cover the case of take + concat as well (joins / repartitions ...)
Describe the solution you'd like
This is added, but just uses take followed by the coalesce kernel.
We can speed it up / avoid a copy by handling take internally.
Describe alternatives you've considered
Additional context
Activity
feat: impl BatchCoalescer::push_batch_with_indices
feat: impl BatchCoalescer::push_batch_with_indices (#8991)
[-]Support `BatchCoalescer::push_batch_with_indices`[/-][+]Optimize `BatchCoalescer::push_batch_with_indices`[/+]Rich-T-kid commentedon Aug 25, 2026
take
perf(coalesce): route push_batch_with_indices through zero-copy fast …
perf(coalesce): route push_batch_with_indices through zero-copy fast …
perf(coalesce): zero-copy push_batch_with_indices fast path
perf(coalesce): zero-copy push_batch_with_indices fast path
perf(coalesce): zero-copy push_batch_with_indices fast path
perf(coalesce): zero-copy push_batch_with_indices fast path
perf(coalesce): zero-copy push_batch_with_indices fast path
perf(coalesce): zero-copy push_batch_with_indices fast path
perf(coalesce): directly copy sparse indexed rows
perf(coalesce): directly copy sparse indexed rows
perf(coalesce): directly copy sparse indexed rows
perf(coalesce): directly copy sparse indexed rows
perf(coalesce): directly copy sparse UInt32 selections (apache#11174)