Repository navigation
fix: avoid sort-based joins for nested float join keys with negative zeros - #26146
entity-0-0-2 wants to merge 1 commit into
Conversation
…zeros SortMergeJoinExec and PiecewiseMergeJoinExec normalize -0.0 to +0.0 in their merge comparators, but the sorts feeding them use Arrow's IEEE total order where -0.0 < +0.0. For flat keys the two orders agree. For nested keys with float leaves the comparison continues past the zero element and the orders diverge, so the merge stream sees mis-sorted input and silently drops matched rows. The divergence opened when normalize_float_zero learned to recurse into nested children in apache#24827. The planner now routes joins whose keys are nested with float leaves to HashJoinExec instead of SortMergeJoinExec, and to NestedLoopJoinExec instead of PiecewiseMergeJoinExec. Key types where both orders provably agree keep the sort-based fast paths. Fixes apache#26115
github-actions
Bot
added
core
|
I am commenting to flag this user and PR as likely created by a bot. AgentScan shows 46 PRs across 3 days with 40/46 fork branches followed by upstream PRs within 82s, and a maintainer on an unrelated project flagged the same account. If you are not a bot, please reply and I'd be happy to give you a review. |
alamb
added
the
Prolific Author
|
Thanks for the contribution! I am closing this under DataFusion's AI Dump policy. This has the hallmarks of AI code that was not reviewed by the author: an account flagged on this thread as automated with no response, and a fix that routes around the bug in the physical planner rather than addressing the comparator mismatch it describes. If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review. |
Sort merge join silently drops rows when the join keys contain nested float columns with negative zeros: the issue's query loses row (2, 11) with no error.
The comparator built in datafusion/physical-plan/src/joins/utils.rs (JoinKeyComparator::build) normalizes negative zeros on both sides of the key, but since d42cd85 (#24827) that normalization recurses into nested children while the sort feeding the merge keeps Arrow's IEEE total order, where -0.0 sorts before +0.0. For nested keys the two orders disagree and the merge stream skips matches.
The physical planner now routes joins with nested float keys to hash join (or nested loop join for range conditions), so sort-based joins only see keys where normalization and sort order agree. A sqllogictest covering the issue's query fails on main with the missing row and passes with the change.
Fixes #26115