Repository navigation
Commit d1fc4b3
authored
Account for transformer-retained batches in symmetric hash join (apache#24364)
## Which issue does this PR close?
* Part of apache#23393
## Rationale for this change
`SymmetricHashJoinStream` did not include the `RecordBatch` retained by
its `BatchTransformer` in the stream's memory reservation. As a result,
the symmetric hash join could under-account memory while either
`NoopBatchTransformer` or `BatchSplitter` held an output batch.
Transformer-held batches can also share Arrow buffers or nested array
objects with other batches retained by the stream, so independently
summing batch memory would double-count shared allocations.
This change makes the stream reservation reflect the memory retained at
that boundary exactly once. Corrected accounting can increase
reservation pressure and may cause a bounded memory pool to report
resource exhaustion earlier, without changing join results.
## What changes are included in this PR?
* Adds `RecordBatchMemoryCounter::count_batch_with_array_overhead` to
count unique Arrow buffers and recursively reachable array-object
allocations across a sequence of batches.
* Deduplicates shared nested array objects for struct, list/list-view,
fixed-size-list, map, union, dictionary, and run-end encoded arrays.
* Adds `BatchTransformer::count_memory` and implements it for both
`NoopBatchTransformer` and `BatchSplitter`.
* Updates `SymmetricHashJoinStream::size()` to account for its input
buffers and transformer-retained batch through one shared
`RecordBatchMemoryCounter`, avoiding duplicate accounting of shared
allocations.
* Updates the stream reservation when a transformer retains or releases
a batch.
* Releases the stream reservation and resets the stream memory metric
when the stream reaches its completed state.
* Leaves the existing `OneSideHashJoiner` accounting formulas otherwise
unchanged.
## Are these changes tested?
Yes. The patch adds regression coverage for:
*
`test_record_batch_memory_counter_array_overhead_shared_across_batches`
*
`test_record_batch_memory_counter_deduplicates_shared_nested_array_overhead`
* `test_record_batch_memory_counter_deduplicates_shared_list_child`
* `test_record_batch_memory_counter_deduplicates_shared_map_children`
* `test_record_batch_memory_counter_deduplicates_shared_union_child`
*
`test_record_batch_memory_counter_deduplicates_shared_dictionary_child`
*
`test_record_batch_memory_counter_deduplicates_shared_run_end_encoded_child`
* `stream_accounts_for_transformer_batches_once`
* `symmetric_hash_join_releases_reservation_when_complete`
* `stream_deduplicates_nested_transformer_batches`
* `transformer_reservation_exhausts_pool`
* `symmetric_hash_join_updates_reservation_while_transforming_output`,
for both `NoopBatchTransformer` and `BatchSplitter`
* `symmetric_hash_join_transformer_retention_exhausts_bounded_pool`, for
both `NoopBatchTransformer` and `BatchSplitter`
The patch shown does not establish that the full `cargo test -p
datafusion-physical-plan`, clippy, repository lint, or extended test
suites have been run.
## Are there any user-facing changes?
There are no intended changes to join results or query semantics.
Memory accounting for symmetric hash joins is more accurate. In
bounded-memory configurations, a join that previously under-accounted
transformer-retained output may now reach the configured memory limit
and report resource exhaustion earlier.
No public API breaking change is introduced by this patch.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.1 parent 3cabfa9 commit d1fc4b3
3 files changed
Lines changed: 833 additions & 32 deletions
0 commit comments