Repository navigation
Commit 23c9f41
## Which issue does this PR close?
- Part of #26868 (the 55.3.0 patch release tracking issue).
- Backport of #25134 to `branch-55`. #25134 fixed #25133.
## Rationale for this change
On 55.2.0, a mark, semi or anti sort-merge join on a floating-point key
fails when a key group ends at the end of an input batch: a Float32 or
Float64 key that is NaN panics with ``called `Option::unwrap()` on a
`None` value``, and a Float16 key returns `Unsupported data type in sort
merge join comparator: Float16`. The main merge scan handles the same
keys. These are failed queries, which makes this eligible for backport
as a correctness fix.
#25134 fixed it on `main` (labelled `v56.0.0`); it is on `branch-56` but
not on `branch-55`. It was requested for 55.3.0 on #26868
(#26868 (comment)).
## What changes are included in this PR?
- A `git cherry-pick -x` of 2abf69d
(#25134, by @jeffw13). It applied without conflicts and this PR changes
nothing else.
-
`datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs`:
`keys_match` uses the existing `JoinKeyComparator` for floating-point
keys, matching the merge scan's equality (including signed zero),
instead of the scalar comparator whose `partial_cmp` panics on NaN.
Integer-only comparisons keep the scalar path.
- `datafusion/physical-plan/src/joins/sort_merge_join/tests.rs`:
`join_float_key_batch_boundaries`, 144 cases (Float16/32/64, left/right
mark/semi/anti, both null-equality modes, ascending/descending, one- and
two-row input batches).
## Are these changes tested?
Yes, by the 144 cases that #25134 added. On this branch (Rust 1.95.0) I
ran:
- `cargo test -p datafusion-physical-plan --lib`: 1844 passed.
- With the change in `bitwise_stream.rs` reverted and the tests kept,
`cargo test -p datafusion-physical-plan --lib sort_merge_join` fails
exactly the 144 new cases (96 NaN panics and 48 Float16 errors).
- `cargo fmt --all -- --check` and `cargo clippy -p
datafusion-physical-plan --all-targets --all-features -- -D warnings`
pass.
## Are there any user-facing changes?
Affected sort-merge joins complete instead of failing at floating-point
batch boundaries. No public API change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Jeff Wang <[email protected]>
Co-authored-by: Claude Sonnet 5.5 <[email protected]>
1 parent 4f4d73f commit 23c9f41
2 files changed
Lines changed: 129 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1178 | 1178 | | |
1179 | 1179 | | |
1180 | 1180 | | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
1181 | 1196 | | |
1182 | 1197 | | |
1183 | 1198 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4029 | 4029 | | |
4030 | 4030 | | |
4031 | 4031 | | |
| 4032 | + | |
| 4033 | + | |
| 4034 | + | |
| 4035 | + | |
| 4036 | + | |
| 4037 | + | |
| 4038 | + | |
| 4039 | + | |
| 4040 | + | |
| 4041 | + | |
| 4042 | + | |
| 4043 | + | |
| 4044 | + | |
| 4045 | + | |
| 4046 | + | |
| 4047 | + | |
| 4048 | + | |
| 4049 | + | |
| 4050 | + | |
| 4051 | + | |
| 4052 | + | |
| 4053 | + | |
| 4054 | + | |
| 4055 | + | |
| 4056 | + | |
| 4057 | + | |
| 4058 | + | |
| 4059 | + | |
| 4060 | + | |
| 4061 | + | |
| 4062 | + | |
| 4063 | + | |
| 4064 | + | |
| 4065 | + | |
| 4066 | + | |
| 4067 | + | |
| 4068 | + | |
| 4069 | + | |
| 4070 | + | |
| 4071 | + | |
| 4072 | + | |
| 4073 | + | |
| 4074 | + | |
| 4075 | + | |
| 4076 | + | |
| 4077 | + | |
| 4078 | + | |
| 4079 | + | |
| 4080 | + | |
| 4081 | + | |
| 4082 | + | |
| 4083 | + | |
| 4084 | + | |
| 4085 | + | |
| 4086 | + | |
| 4087 | + | |
| 4088 | + | |
| 4089 | + | |
| 4090 | + | |
| 4091 | + | |
| 4092 | + | |
| 4093 | + | |
| 4094 | + | |
| 4095 | + | |
| 4096 | + | |
| 4097 | + | |
| 4098 | + | |
| 4099 | + | |
| 4100 | + | |
| 4101 | + | |
| 4102 | + | |
| 4103 | + | |
| 4104 | + | |
| 4105 | + | |
| 4106 | + | |
| 4107 | + | |
| 4108 | + | |
| 4109 | + | |
| 4110 | + | |
| 4111 | + | |
| 4112 | + | |
| 4113 | + | |
| 4114 | + | |
| 4115 | + | |
| 4116 | + | |
| 4117 | + | |
| 4118 | + | |
| 4119 | + | |
| 4120 | + | |
| 4121 | + | |
| 4122 | + | |
| 4123 | + | |
| 4124 | + | |
| 4125 | + | |
| 4126 | + | |
| 4127 | + | |
| 4128 | + | |
| 4129 | + | |
| 4130 | + | |
| 4131 | + | |
| 4132 | + | |
| 4133 | + | |
| 4134 | + | |
| 4135 | + | |
| 4136 | + | |
| 4137 | + | |
| 4138 | + | |
| 4139 | + | |
| 4140 | + | |
| 4141 | + | |
| 4142 | + | |
| 4143 | + | |
| 4144 | + | |
| 4145 | + | |
4032 | 4146 | | |
4033 | 4147 | | |
4034 | 4148 | | |
| |||
0 commit comments