You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 201099b
Browse filesBrowse the repository at this point in the historyBrowse files
authored
fix: evaluate sort-merge join filters that reference no columns (apache#26010)
## Which issue does this PR close?
- Closesapache#26002.
## Rationale for this change
`SortMergeJoinExec` only evaluated its join filter when the filter read
at least one column. A filter that reads none (a bound parameter such as
`$1 > 5`, or a volatile function such as `random() > 2`) was skipped, so
every key match was kept. For `FULL JOIN` that returns matched pairs the
filter rejects, where hash join null-pads both sides.
## What changes are included in this PR?
- `freeze_streamed_matched` (`sort_merge_join/materializing_stream.rs`)
now evaluates the filter whenever the join has one. The filter batch is
built with `RecordBatchOptions::with_row_count(total_matched_rows)`,
because a batch with no columns cannot infer its row count.
- Two queries in `sort_merge_join.slt`: `FULL JOIN ... AND random() > 2`
(every row null-padded) and `AND random() < 2` (all key matches kept).
- `bitwise_stream.rs` (semi/anti/mark joins) built its filter batch the
same way and failed with `must either specify a row count or at least
one column` for a column-free filter. It now passes the row count too,
with `join_semi_anti_mark_column_free_filter` covering all six join
types. SQL cannot reach that path because the planner pushes such a
predicate into an input; a `SortMergeJoinExec` built directly can.
## What is the testing strategy for this PR?
`cargo test -p datafusion-sqllogictest --test sqllogictests --
sort_merge_join`: all 4 files pass. With the `materializing_stream.rs`
change reverted, `sort_merge_join.slt` fails; with the
`bitwise_stream.rs` change reverted, the new unit test fails with the
error above. `cargo test -p datafusion-physical-plan --lib
sort_merge_join`: 250 passed. `cargo fmt` and `cargo clippy -p
datafusion-physical-plan --all-targets -- -D warnings` pass.
## Are there any user-facing changes?
`FULL JOIN` (and other join types run by sort-merge join) with a
column-free `ON` condition now returns the same rows as hash join. No
API change.
AI assistance: written with Claude Code; I reproduced the bug, reviewed
the diff and ran the tests above.
Co-authored-by: mrhard9090 <[email protected]>
Co-authored-by: Claude Sonnet 5.5 <[email protected]>
Co-authored-by: Jay Zhan <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
0 commit comments