Repository navigation
Commit 93a29e0
## Which issue does this PR close?
- Part of #26868 (the 55.3.0 patch release tracking issue).
- Backport of #26027 to `branch-55`. #26027 fixed #26001.
## Rationale for this change
With a sort-merge join (`datafusion.optimizer.prefer_hash_join =
false`), a `LEFT`, `RIGHT` or `FULL` join with an extra join filter
fails with `Arrow error: Invalid argument error: Column '...' is
declared as non-nullable but contains null values` when the NULL-padded
side has `NOT NULL` columns referenced by the filter. Hash join returns
the correct result for the same queries. This is a failed query, which
makes it eligible for backport as a correctness fix.
When a streamed row has no match but the buffered side still holds a
batch, the NULL-padded row is materialized in the same chunk as matched
rows, so its NULL buffered-side columns reach filter evaluation, whose
intermediate schema declares them non-nullable. #26027 fixed it on
`main` (labelled `v56.0.0`); it is on `branch-56` but not on
`branch-55`, so it is in neither 55.1.0 nor 55.2.0. It was requested for
55.3.0 on #26868
(#26868 (comment)).
## What changes are included in this PR?
- A `git cherry-pick -x` of 8248a57
(#26027, by @CuteChuanChuan).
-
`datafusion/physical-plan/src/joins/sort_merge_join/materializing_stream.rs`:
`MaterializingSortMergeJoinStream::try_new` builds the filter schema
once with the buffered-side fields marked nullable, and filter
evaluation uses it. It is only built when the join needs deferred
filtering (outer join with a filter); inner joins keep the original
schema. The filter result of a NULL-padded row does not affect the
output, so this does not change results.
- `datafusion/sqllogictest/test_files/sort_merge_join.slt`: the `LEFT`,
`FULL` and `RIGHT` cases from the issue, and a `LEFT` case whose filter
divides by a buffered-side column.
- One conflict, in `materializing_stream.rs`: on this branch
`needs_deferred_filtering` takes `&Option<JoinFilter>` (on `main` it
takes `Option<&JoinFilter>`), so the new local passes `&filter`. The
rest is as upstream.
## Are these changes tested?
Yes. On this branch (Rust 1.95.0) I ran:
- `cargo test -p datafusion-sqllogictest --test sqllogictests -- join`:
all 16 matching files pass, including `sort_merge_join.slt` with the new
cases.
- With the change in `materializing_stream.rs` reverted and the new slt
cases kept, the four new queries fail with the Arrow error above
(`sort_merge_join.slt` lines 990, 997, 1004 and 1013).
- `cargo test -p datafusion-physical-plan --lib`: 1700 passed.
- `cargo fmt --all -- --check` and `cargo clippy -p
datafusion-physical-plan --all-targets --all-features -- -D warnings`
pass.
## Are there any user-facing changes?
No. Bug fix only; no public API change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Yu-Chuan Hung <[email protected]>
Co-authored-by: Claude Sonnet 5.5 <[email protected]>
1 parent 23c9f41 commit 93a29e0
2 files changed
Lines changed: 92 additions & 4 deletions
File tree
- datafusion
- physical-plan/src/joins/sort_merge_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| |||
295 | 295 | | |
296 | 296 | | |
297 | 297 | | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
298 | 302 | | |
299 | 303 | | |
300 | 304 | | |
| |||
553 | 557 | | |
554 | 558 | | |
555 | 559 | | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
556 | 584 | | |
557 | 585 | | |
558 | 586 | | |
| |||
568 | 596 | | |
569 | 597 | | |
570 | 598 | | |
571 | | - | |
| 599 | + | |
| 600 | + | |
572 | 601 | | |
573 | 602 | | |
574 | 603 | | |
| |||
1560 | 1589 | | |
1561 | 1590 | | |
1562 | 1591 | | |
| 1592 | + | |
1563 | 1593 | | |
1564 | | - | |
| 1594 | + | |
1565 | 1595 | | |
1566 | 1596 | | |
1567 | 1597 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
971 | 971 | | |
972 | 972 | | |
973 | 973 | | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
974 | 1032 | | |
975 | 1033 | | |
976 | 1034 | | |
0 commit comments