Skip to content

Commit 58325af

Browse files
committed
perf: filter sort-merge inner join pairs before materializing them
An Inner sort-merge join with a filter materialized every column of every pair that matched on the join keys, and only then evaluated the filter and dropped the pairs that failed it. When a key matches many rows on each side and the filter keeps few of the pairs, most of the join's time went into copying rows that were then dropped. An Inner join with a filter now evaluates it before materializing: - A run of at least 256 buffered rows is filtered as it is paired with a streamed row. The filter reads the run as a slice of its buffered batch and the streamed row's values repeated, and only the pairs that pass are recorded, in chunks marked as filtered. - The pairs of shorter runs are filtered at freeze on just the columns the filter reads, and the output columns are materialized for the pairs that pass. - When earlier freezes kept most of their pairs and the filter reads most of the output columns, the freeze materializes the batch whole and filters it, as before. Outer joins keep the deferred filtering they need for null-joined rows, and joins without a filter keep the original pairing loop.
1 parent 9d916ab commit 58325af

2 files changed

Lines changed: 920 additions & 77 deletions

File tree

0 commit comments

Comments
 (0)