Repository navigation
Commit 58325af
committed
perf: filter sort-merge inner join pairs before materializing them
An Inner sort-merge join with a filter materialized every column of every
pair that matched on the join keys, and only then evaluated the filter and
dropped the pairs that failed it. When a key matches many rows on each side
and the filter keeps few of the pairs, most of the join's time went into
copying rows that were then dropped.
An Inner join with a filter now evaluates it before materializing:
- A run of at least 256 buffered rows is filtered as it is paired with a
streamed row. The filter reads the run as a slice of its buffered batch
and the streamed row's values repeated, and only the pairs that pass are
recorded, in chunks marked as filtered.
- The pairs of shorter runs are filtered at freeze on just the columns the
filter reads, and the output columns are materialized for the pairs that
pass.
- When earlier freezes kept most of their pairs and the filter reads most
of the output columns, the freeze materializes the batch whole and
filters it, as before.
Outer joins keep the deferred filtering they need for null-joined rows,
and joins without a filter keep the original pairing loop.1 parent 9d916ab commit 58325af
2 files changed
Lines changed: 920 additions & 77 deletions
0 commit comments