Repository navigation
Commit 18cd886
fix: reserve the concatenated build batch and computed join keys in HashJoinExec (apache#25496)
## Which issue does this PR close?
- Closes apache#25269.
## Rationale for this change
A hash join reserves every build-side batch as it arrives, then copies
all of them
into a single batch without reserving the copy, while the input batches
stay alive
until the build finishes. Near the memory limit the process therefore
holds about
twice the build side while the memory pool sees it once, so instead of a
`ResourcesExhausted` error the process can be killed by the OS. It also
means
nothing that reacts to pool pressure can rely on what a hash join
reports.
Measured on a 1.25 GB build side (TPC-H SF1 `lineitem`, 16 columns, as
the build
side): the pool was charged ~1.25 GB while peak RSS was ~2.46 GB.
## What changes are included in this PR?
- `concat_build_batches`: reserves the copy before `concat_batches`,
drops the input
batches, then trims the reservation to what the concatenated batch
retains
(`get_record_batch_memory_size`), growing instead if the estimate was
low.
`build_mem_used` follows every grow and shrink.
- Nothing extra is reserved when there is nothing to copy: concatenating
a single
batch is zero-copy, and for `Utf8View`/`BinaryView` columns only the
views are
copied while the data buffers stay shared with the inputs.
- `try_create_array_map` becomes `array_map_key_range`: it only decides
whether the
perfect hash join applies and reserves the `ArrayMap`. Both the hash map
and the
`ArrayMap` paths now hand the batches to the helper by value, so the
inputs are
gone before the visited bitmap and the null-aware scope maps are
allocated.
- Join key arrays that do not share the buffers of the build batch (any
on-expression
that is not a plain column) are kept for the whole join and are now
reserved too.
This uses a new public `RecordBatchMemoryCounter::count_array`.
This does not reduce memory usage: the build still peaks at about twice
the build
side, it is now counted. A memory-limited query that only fit because
the copy was
not counted can now fail with `ResourcesExhausted`.
## What is the testing strategy for this PR?
- `join_build_concat_is_reserved` (CollectLeft and Partitioned, hash map
and
`ArrayMap`): fails with `ResourcesExhausted` at map + 1.5x inputs,
passes at
map + 3x inputs. It fails on `main` in all four variants.
- `join_build_key_arrays_are_reserved` (hash map and `ArrayMap`): with a
single build
batch and an `a1 + 1` join key, the join fails at inputs + map + half
the key array
and passes with room for it; a plain column key passes at the lower
limit. It fails
when the reservation is removed.
- Unit tests of the helper: output equals `concat_batches` in both
orders, the copy
is reserved and the reservation then matches the retained batch, a
single batch
is not reserved twice, view data buffers are not reserved twice.
- Existing `single_partition_join_overallocation` /
`partitioned_join_overallocation`
are unchanged.
## Are there any user-facing changes?
One public addition: `RecordBatchMemoryCounter::count_array`. Hash joins
running close
to their memory limit may now report `ResourcesExhausted` where they
previously exceeded the limit unnoticed.1 parent 2bb3667 commit 18cd886
2 files changed
Lines changed: 523 additions & 54 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
196 | 204 | | |
197 | 205 | | |
198 | 206 | | |
| |||
581 | 589 | | |
582 | 590 | | |
583 | 591 | | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
584 | 604 | | |
585 | 605 | | |
586 | 606 | | |
| |||
0 commit comments