Skip to content

BatchCoalescer: add push_batch_interleaved #11141

Closed

Description

@Rich-T-kid

BatchCoalescer already has push_batch_with_indices(batch, indices: &dyn Array) for the single-batch take case. There is no equivalent for the multi-batch case.

Callers that need to gather rows from multiple source batches must today call interleave_record_batch (copy 1) and then push_batch the result (copy 2). The intermediate batch is immediately discarded.

Proposed API

/// Semantically equivalent to `interleave_record_batch(batches, indices)`
/// followed by `push_batch`, without the intermediate RecordBatch allocation.
pub fn push_batch_interleaved(
    &mut self,
    batches: &[RecordBatch],
    indices: &[(usize, usize)], // (batch_idx, row_idx)
) -> Result<(), ArrowError>

Known callers in DataFusion that would benefit (tracked in apache/datafusion#25515):

  • Top-K; TopKHeap::emit_with_state calls interleave_record_batch to reconstruct k surviving rows from the RecordBatchStore, then immediately pushes the result to a BatchCoalescer.
  • Hash join / nested loop join; build_batch_from_indices or interleave_record_batch to materialize matched row pairs (copy 1), then push to the coalescer (copy 2).
  • Sort merge join; join rows go through a materialization step before reaching the coalescer.

Activity

  1. Rich-T-kid commented on Sep 20, 2026

    @Rich-T-kid
    ContributorAuthor

    this is actually somewhat similar to #8957

  2. Rich-T-kid commented on Oct 12, 2026

    @Rich-T-kid
    ContributorAuthor

    take

  3. added a commit that references this issue on Oct 20, 2026
    1357f25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions