[Bug] Sort merge retains consumed source batches across retract-only keys with pending output #1386

Search before asking

• I searched in the issues and found nothing similar.

Searched issues and PRs for batch_buffer, retained batch, sort merge memory, and merge memory. #911 concerns broader resource governance; no directly matching issue was found for this Reader retention path as of November 12, 2026.

Please describe the bug

Source inspection of main f91fdb8ab034887547a5ee2f26c1979e30250558 identifies a batch-retention path in sort_merge_stream:

• A consumed input batch is added to batch_buffer.
• A surviving key adds an entry to output_indices.
• MergeResult::Omit adds no output.
• The buffer is compacted when output is empty, or after a full output batch has been materialized.

When output is nonempty but below the configured batch size, a subsequent long sequence of different keys whose winners are retracts does not meet either cleanup condition. Consumed source batches can remain retained until the pending output is finally emitted or the input ends.

A focused regression can exercise the merge reader with multiple input streams, one early surviving key, then many distinct keys whose final state is deleted. Keep the total number of surviving keys below batch_size and split inputs across many small RecordBatches. The output should contain the surviving row, while retention of unrelated consumed batches should not grow with the number of deleted keys.

This report records a static code-path finding. No new runtime reproduction, allocation count, or RSS measurement has been run, and no OOM frequency is claimed.

Solution

Reclaim consumed batches independently of filling a complete output batch, while preserving batches still referenced by cursors and pending output. Partial-output materialization or reference-aware reclamation are possible directions; the final algorithm needs review.

Use deterministic retained-buffer or ownership observations in a regression rather than an RSS-only threshold. Cover output order, row values, all-retract input, and pending surviving rows.

One key with arbitrarily many versions is a separate merge-state boundary and is outside this narrowly scoped report. This issue does not claim to establish a bound on every allocation in the Reader.

Reference: sort_merge output and cleanup conditions.

Are you willing to submit a PR?

• I'm willing to submit a PR!