Repository navigation
Commit af752e5
authored
fix(smj): compare all key types across batch boundaries in semi/anti/mark joins (apache#26830)
## Which issue does this PR close?
Closes apache#26003.
## Rationale for this change
`EXISTS`, `IN` and `NOT EXISTS` queries planned as sort-merge joins fail
with "Unsupported data type in sort merge join comparator" on
timezone-aware timestamp, `Dictionary`, `Decimal256` and
`Time32`/`Time64` keys, once a key group runs to the end of a batch.
Inner sort-merge join and hash join already work on these keys.
## What changes are included in this PR?
The bitwise stream compares keys with `JoinKeyComparator` everywhere
except in `keys_match`, the check for whether a key group continues into
the next batch. That function still called `compare_join_arrays` for
non-float keys, and `compare_join_arrays` only knows a fixed list of
types. apache#25134 moved float keys over to `JoinKeyComparator`; this moves
the rest.
`keys_match` now builds a one-row comparator each time a key group
reaches the end of a batch. The scan already rebuilds its comparators on
every new batch, so I don't expect a measurable cost.
One edge case changes: `compare_join_arrays` treated `DataType::Null`
keys as always equal, while `JoinKeyComparator` treats them as unequal
under `NullEqualsNothing`. As far as I can tell this can't change any
output, because such keys never compare equal in the scan either, so the
code paths that call `keys_match` don't run for them.
`compare_join_arrays` has no callers left, but it's public in
`joins::utils`, so I left it alone. Happy to deprecate it here or in a
follow-up if you prefer.
## What is the testing strategy for this PR?
- `sort_merge_join.slt`: the reproducer from the issue, plus `batch_size
= 2` queries for `Dictionary`, `Decimal256` and `Time64` keys.
- `join_key_type_batch_boundaries` in `sort_merge_join/tests.rs`: the
same matrix as `join_float_key_batch_boundaries` (join type, null
equality, sort direction, 1- and 2-row batches) for the five key types
above. I moved the body of the float test into a shared helper so the
two don't duplicate it. Like apache#25134, it's a physical-plan test, so
batches split at exact rows regardless of what the optimizer does.
Without the fix, all of the new cases fail with the error from the
issue.
The change also covers other types `compare_join_arrays` didn't handle
(e.g. `Duration`, `Interval`, `Decimal32`/`Decimal64`, nested types),
but I haven't added tests for those.
## Are there any user-facing changes?
Those queries now return results instead of an error. No API changes.1 parent d1684ab commit af752e5
3 files changed
Lines changed: 184 additions & 42 deletions
File tree
- datafusion
- physical-plan/src/joins/sort_merge_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
113 | | - | |
| 113 | + | |
114 | 114 | | |
115 | 115 | | |
116 | 116 | | |
| |||
1215 | 1215 | | |
1216 | 1216 | | |
1217 | 1217 | | |
1218 | | - | |
1219 | | - | |
1220 | | - | |
1221 | | - | |
1222 | | - | |
1223 | | - | |
1224 | | - | |
1225 | | - | |
1226 | | - | |
1227 | | - | |
1228 | | - | |
1229 | | - | |
1230 | | - | |
1231 | | - | |
1232 | | - | |
1233 | | - | |
1234 | | - | |
1235 | | - | |
1236 | | - | |
1237 | | - | |
1238 | | - | |
1239 | | - | |
1240 | | - | |
1241 | | - | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
1242 | 1227 | | |
1243 | 1228 | | |
1244 | 1229 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5268 | 5268 | | |
5269 | 5269 | | |
5270 | 5270 | | |
5271 | | - | |
5272 | | - | |
| 5271 | + | |
| 5272 | + | |
| 5273 | + | |
| 5274 | + | |
| 5275 | + | |
| 5276 | + | |
| 5277 | + | |
| 5278 | + | |
| 5279 | + | |
| 5280 | + | |
| 5281 | + | |
| 5282 | + | |
| 5283 | + | |
| 5284 | + | |
| 5285 | + | |
| 5286 | + | |
| 5287 | + | |
| 5288 | + | |
| 5289 | + | |
| 5290 | + | |
| 5291 | + | |
| 5292 | + | |
| 5293 | + | |
| 5294 | + | |
| 5295 | + | |
| 5296 | + | |
| 5297 | + | |
| 5298 | + | |
| 5299 | + | |
| 5300 | + | |
| 5301 | + | |
| 5302 | + | |
| 5303 | + | |
| 5304 | + | |
| 5305 | + | |
| 5306 | + | |
| 5307 | + | |
| 5308 | + | |
| 5309 | + | |
| 5310 | + | |
| 5311 | + | |
| 5312 | + | |
| 5313 | + | |
| 5314 | + | |
| 5315 | + | |
| 5316 | + | |
| 5317 | + | |
| 5318 | + | |
| 5319 | + | |
| 5320 | + | |
5273 | 5321 | | |
5274 | | - | |
5275 | | - | |
5276 | | - | |
| 5322 | + | |
| 5323 | + | |
| 5324 | + | |
5277 | 5325 | | |
5278 | | - | |
5279 | | - | |
5280 | | - | |
5281 | | - | |
5282 | | - | |
| 5326 | + | |
| 5327 | + | |
| 5328 | + | |
| 5329 | + | |
| 5330 | + | |
| 5331 | + | |
| 5332 | + | |
| 5333 | + | |
| 5334 | + | |
| 5335 | + | |
| 5336 | + | |
| 5337 | + | |
| 5338 | + | |
| 5339 | + | |
| 5340 | + | |
| 5341 | + | |
| 5342 | + | |
| 5343 | + | |
| 5344 | + | |
| 5345 | + | |
| 5346 | + | |
| 5347 | + | |
| 5348 | + | |
| 5349 | + | |
| 5350 | + | |
| 5351 | + | |
| 5352 | + | |
| 5353 | + | |
| 5354 | + | |
| 5355 | + | |
| 5356 | + | |
| 5357 | + | |
| 5358 | + | |
| 5359 | + | |
| 5360 | + | |
| 5361 | + | |
5283 | 5362 | | |
5284 | 5363 | | |
5285 | | - | |
5286 | 5364 | | |
| 5365 | + | |
| 5366 | + | |
| 5367 | + | |
| 5368 | + | |
| 5369 | + | |
| 5370 | + | |
| 5371 | + | |
5287 | 5372 | | |
5288 | 5373 | | |
5289 | 5374 | | |
5290 | 5375 | | |
5291 | 5376 | | |
5292 | 5377 | | |
5293 | | - | |
5294 | | - | |
5295 | | - | |
5296 | | - | |
| 5378 | + | |
5297 | 5379 | | |
5298 | 5380 | | |
5299 | 5381 | | |
| |||
5309 | 5391 | | |
5310 | 5392 | | |
5311 | 5393 | | |
5312 | | - | |
5313 | | - | |
| 5394 | + | |
| 5395 | + | |
5314 | 5396 | | |
5315 | 5397 | | |
5316 | 5398 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1054 | 1054 | | |
1055 | 1055 | | |
1056 | 1056 | | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
| 1131 | + | |
1057 | 1132 | | |
1058 | 1133 | | |
1059 | 1134 | | |
0 commit comments