You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a67d64a
Browse filesBrowse the repository at this point in the historyBrowse files
fix: charge semi/anti sort-merge join key groups by their sliced size (apache#25933)
## Which issue does this PR close?
- Part of apache#25923 (the memory accounting half; the per-row filter cost is
separate).
## Rationale for this change
A filtered LeftSemi/LeftAnti sort-merge join under a tight memory pool
spills once per matched key group, even when each group holds only a few
rows. Each buffered inner key group is charged its parent batch's full
size, so a group of at most 7 rows is charged 131,264 bytes (a whole
8192-row batch). On the benchmark from the issue, with a
`GreedyMemoryPool` of 64 KB over 20K keys, every one of the 18,128
matched groups spilled to its own file. In a real plan the join shares
the pool with the sorts in the same task, so this can start well before
the data needs to spill.
## What changes are included in this PR?
`buffer_inner_key_group` in `bitwise_stream.rs` now reserves
`get_sliced_size()` for each buffered slice instead of
`get_array_memory_size()`, the same measure sort uses. Only the
reservation (and so the spill decision and `peak_mem_used`) was
inflated; `spilled_bytes` counts serialized bytes and is unaffected.
A group that reaches the end of its inner batch is still charged the
whole parent batch: once the cursor moves on, that slice is the only
thing keeping the parent alive. View arrays keep their parent's data
buffers in the measure, so they can still be overcounted. Both are noted
at the call site.
## What is the testing strategy for this PR?
`bitwise_small_key_groups_charged_by_sliced_size` joins inputs with 2
and 3 row key groups under a pool of half a batch, for LeftSemi,
LeftAnti, RightSemi and RightAnti. It checks that the join spills at
most once per inner batch (for the group reaching each batch's end; 2,
2, 1 and 1 spills) and that the results match an unbounded run. On main
it spills 683 times.
`bitwise_key_group_at_inner_batch_end_charged_full_parent` uses unique
keys, so the last row of every inner batch is such a group, and checks
that `peak_mem_used` covers a whole inner batch.
Five existing spill tests used a 100-byte pool to force a spill. A
one-row slice is now charged about 12 bytes, so their pool is 1 byte;
their assertions are unchanged.
With the first version of this change on 55.1.0, the issue's 64 KB pool
case went from 18,128 spills to none, and the join ran as fast as with
an unbounded pool (11.9 ms both, where the 64 KB pool previously took
1,088 ms against 9.3 ms unbounded). Charging the batch-end group in full
adds at most one spill per inner batch on top of that, since that pool
is smaller than one batch.
## Are there any user-facing changes?
No API changes. Filtered semi and anti sort-merge joins spill less under
memory pressure.
0 commit comments