Skip to content

Commit 39d5064

Browse files
authored
chore: refactor MaterializingSortMergeJoinStream into generators and simplify code to be textbook like as possible (#23976)
## Which issue does this PR close? Part of: - #23974 ## Rationale for this change Simplify the code and make code similar to textbook so it is easier to read ## What changes are included in this PR? Changed to async generators + simplified to be similar to textbook Most of the code was written by Claude Fable 5 since it was just too large to comprehend in head ## Are these changes tested? existing tests + a little more tests to verify we are not counting the childs in the join time ## Are there any user-facing changes? The join_time now includes the time to read from the async spill stream between pending which is arguable more correct since this time is part of the operator, although long waits between pending calls will be counted in the op `join_time` while the alternative is not counting the read from file and decoding...
1 parent 1ae6b87 commit 39d5064

3 files changed

Lines changed: 720 additions & 624 deletions

File tree

‎datafusion/physical-plan/src/joins/sort_merge_join/exec.rs‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -81,8 +81,7 @@ use datafusion_physical_expr_common::sort_expr::{LexOrdering, OrderingRequiremen
8181
/// on the output batch size of the execution plan. There is no spilling support for streamed input.
8282
/// The comparisons are performed from values of join keys in streamed input with the values of
8383
/// join keys in buffered input. One row in streamed record batch could be matched with multiple rows in
84-
/// buffered input batches. The streamed input is managed through the states in `StreamedState`
85-
/// and streamed input batches are represented by `StreamedBatch`.
84+
/// buffered input batches. Streamed input batches are represented by `StreamedBatch`.
8685
///
8786
/// Buffered input is buffered for all record batches having the same value of join key.
8887
/// If the memory limit increases beyond the specified value and spilling is enabled,
@@ -92,8 +91,7 @@ use datafusion_physical_expr_common::sort_expr::{LexOrdering, OrderingRequiremen
9291
/// memory/disk depends on the number of rows of buffered input having the same value
9392
/// of join key as that of streamed input rows currently present in memory. Due to pre-sorted inputs,
9493
/// the algorithm understands when it is not needed anymore, and releases the buffered batches
95-
/// from memory/disk. The buffered input is managed through the states in `BufferedState`
96-
/// and buffered input batches are represented by `BufferedBatch`.
94+
/// from memory/disk. Buffered input batches are represented by `BufferedBatch`.
9795
///
9896
/// Depending on the type of join, left or right input may be selected as streamed or buffered
9997
/// respectively. For example, in a left-outer join, the left execution plan will be selected as
@@ -546,7 +544,7 @@ impl ExecutionPlan for SortMergeJoinExec {
546544
context.runtime_env(),
547545
)
548546
} else {
549-
Ok(Box::pin(MaterializingSortMergeJoinStream::try_new(
547+
MaterializingSortMergeJoinStream::try_new(
550548
Arc::clone(&self.schema),
551549
self.sort_options.clone(),
552550
self.null_equality,
@@ -561,7 +559,7 @@ impl ExecutionPlan for SortMergeJoinExec {
561559
reservation,
562560
spill_manager,
563561
context.runtime_env(),
564-
)?))
562+
)
565563
}
566564
}
567565

0 commit comments

Comments
 (0)