Skip to content

Commit d1fc4b3

Browse files
authored
Account for transformer-retained batches in symmetric hash join (apache#24364)
## Which issue does this PR close? * Part of apache#23393 ## Rationale for this change `SymmetricHashJoinStream` did not include the `RecordBatch` retained by its `BatchTransformer` in the stream's memory reservation. As a result, the symmetric hash join could under-account memory while either `NoopBatchTransformer` or `BatchSplitter` held an output batch. Transformer-held batches can also share Arrow buffers or nested array objects with other batches retained by the stream, so independently summing batch memory would double-count shared allocations. This change makes the stream reservation reflect the memory retained at that boundary exactly once. Corrected accounting can increase reservation pressure and may cause a bounded memory pool to report resource exhaustion earlier, without changing join results. ## What changes are included in this PR? * Adds `RecordBatchMemoryCounter::count_batch_with_array_overhead` to count unique Arrow buffers and recursively reachable array-object allocations across a sequence of batches. * Deduplicates shared nested array objects for struct, list/list-view, fixed-size-list, map, union, dictionary, and run-end encoded arrays. * Adds `BatchTransformer::count_memory` and implements it for both `NoopBatchTransformer` and `BatchSplitter`. * Updates `SymmetricHashJoinStream::size()` to account for its input buffers and transformer-retained batch through one shared `RecordBatchMemoryCounter`, avoiding duplicate accounting of shared allocations. * Updates the stream reservation when a transformer retains or releases a batch. * Releases the stream reservation and resets the stream memory metric when the stream reaches its completed state. * Leaves the existing `OneSideHashJoiner` accounting formulas otherwise unchanged. ## Are these changes tested? Yes. The patch adds regression coverage for: * `test_record_batch_memory_counter_array_overhead_shared_across_batches` * `test_record_batch_memory_counter_deduplicates_shared_nested_array_overhead` * `test_record_batch_memory_counter_deduplicates_shared_list_child` * `test_record_batch_memory_counter_deduplicates_shared_map_children` * `test_record_batch_memory_counter_deduplicates_shared_union_child` * `test_record_batch_memory_counter_deduplicates_shared_dictionary_child` * `test_record_batch_memory_counter_deduplicates_shared_run_end_encoded_child` * `stream_accounts_for_transformer_batches_once` * `symmetric_hash_join_releases_reservation_when_complete` * `stream_deduplicates_nested_transformer_batches` * `transformer_reservation_exhausts_pool` * `symmetric_hash_join_updates_reservation_while_transforming_output`, for both `NoopBatchTransformer` and `BatchSplitter` * `symmetric_hash_join_transformer_retention_exhausts_bounded_pool`, for both `NoopBatchTransformer` and `BatchSplitter` The patch shown does not establish that the full `cargo test -p datafusion-physical-plan`, clippy, repository lint, or extended test suites have been run. ## Are there any user-facing changes? There are no intended changes to join results or query semantics. Memory accounting for symmetric hash joins is more accurate. In bounded-memory configurations, a join that previously under-accounted transformer-retained output may now reach the configured memory limit and report resource exhaustion earlier. No public API breaking change is introduced by this patch. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 3cabfa9 commit d1fc4b3

3 files changed

Lines changed: 833 additions & 32 deletions

File tree

0 commit comments

Comments
 (0)