Skip to content

Commit 18cd886

Browse files
jayzhan211Omega359
authored andcommitted
fix: reserve the concatenated build batch and computed join keys in HashJoinExec (apache#25496)
## Which issue does this PR close? - Closes apache#25269. ## Rationale for this change A hash join reserves every build-side batch as it arrives, then copies all of them into a single batch without reserving the copy, while the input batches stay alive until the build finishes. Near the memory limit the process therefore holds about twice the build side while the memory pool sees it once, so instead of a `ResourcesExhausted` error the process can be killed by the OS. It also means nothing that reacts to pool pressure can rely on what a hash join reports. Measured on a 1.25 GB build side (TPC-H SF1 `lineitem`, 16 columns, as the build side): the pool was charged ~1.25 GB while peak RSS was ~2.46 GB. ## What changes are included in this PR? - `concat_build_batches`: reserves the copy before `concat_batches`, drops the input batches, then trims the reservation to what the concatenated batch retains (`get_record_batch_memory_size`), growing instead if the estimate was low. `build_mem_used` follows every grow and shrink. - Nothing extra is reserved when there is nothing to copy: concatenating a single batch is zero-copy, and for `Utf8View`/`BinaryView` columns only the views are copied while the data buffers stay shared with the inputs. - `try_create_array_map` becomes `array_map_key_range`: it only decides whether the perfect hash join applies and reserves the `ArrayMap`. Both the hash map and the `ArrayMap` paths now hand the batches to the helper by value, so the inputs are gone before the visited bitmap and the null-aware scope maps are allocated. - Join key arrays that do not share the buffers of the build batch (any on-expression that is not a plain column) are kept for the whole join and are now reserved too. This uses a new public `RecordBatchMemoryCounter::count_array`. This does not reduce memory usage: the build still peaks at about twice the build side, it is now counted. A memory-limited query that only fit because the copy was not counted can now fail with `ResourcesExhausted`. ## What is the testing strategy for this PR? - `join_build_concat_is_reserved` (CollectLeft and Partitioned, hash map and `ArrayMap`): fails with `ResourcesExhausted` at map + 1.5x inputs, passes at map + 3x inputs. It fails on `main` in all four variants. - `join_build_key_arrays_are_reserved` (hash map and `ArrayMap`): with a single build batch and an `a1 + 1` join key, the join fails at inputs + map + half the key array and passes with room for it; a plain column key passes at the lower limit. It fails when the reservation is removed. - Unit tests of the helper: output equals `concat_batches` in both orders, the copy is reserved and the reservation then matches the retained batch, a single batch is not reserved twice, view data buffers are not reserved twice. - Existing `single_partition_join_overallocation` / `partitioned_join_overallocation` are unchanged. ## Are there any user-facing changes? One public addition: `RecordBatchMemoryCounter::count_array`. Hash joins running close to their memory limit may now report `ResourcesExhausted` where they previously exceeded the limit unnoticed.
1 parent 2bb3667 commit 18cd886

2 files changed

Lines changed: 523 additions & 54 deletions

File tree

‎datafusion/common/src/utils/memory.rs‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,14 @@ impl RecordBatchMemoryCounter {
193193
self.memory_usage - previous_memory_usage
194194
}
195195

196+
/// Count `array`, returning the memory used by its buffers that have not
197+
/// been counted before.
198+
pub fn count_array(&mut self, array: &dyn Array) -> usize {
199+
let previous_memory_usage = self.memory_usage;
200+
self.count_array_memory_size(array);
201+
self.memory_usage - previous_memory_usage
202+
}
203+
196204
/// Counts unique buffers and Array objects retained by `batch`.
197205
///
198206
/// This is useful for accounting a sequence of batches at an operator
@@ -581,6 +589,18 @@ mod record_batch_tests {
581589
assert_eq!(counter.memory_usage(), array_data_memory_size(array));
582590
}
583591

592+
#[test]
593+
fn test_count_array_counts_shared_buffers_once() {
594+
let array = Int32Array::from(vec![1, 2, 3, 4, 5]);
595+
let size = array_data_memory_size(&array);
596+
597+
let mut counter = RecordBatchMemoryCounter::new();
598+
assert_eq!(counter.count_array(&array), size);
599+
// A slice shares the buffer that is already counted
600+
assert_eq!(counter.count_array(&array.slice(1, 2)), 0);
601+
assert_eq!(counter.memory_usage(), size);
602+
}
603+
584604
#[test]
585605
fn test_get_record_batch_memory_size() {
586606
let schema = Arc::new(Schema::new(vec![

0 commit comments

Comments
 (0)