Repository navigation
Commit de9d1f9
authored
fix: do not prune ORDER BY keys using nullable UNIQUE constraints (apache#23918)
## Which issue does this PR close?
- Closes apache#23818.
## Rationale for this change
Please check the issue for details, but the main idea is that a nullable
`UNIQUE` column permits multiple NULL rows, so it does not determine
later sort keys across them and they must not be pruned.
## What changes are included in this PR?
- `FunctionalDependence` gets a `null_equality` field that records the
NULL semantics under which the dependency holds. The default is
`NullEqualsNothing`.
- Sort-key pruning only uses a dependency that holds when NULLs are
treated as equal (for example a GROUP BY key), has a non-nullable
determinant, or whose source columns are all `NOT NULL`.
- A primary key that is downgraded on the NULL-padded side of an outer
join keeps `NullEqualsNull`, because padded rows are NULL in every
column on that side.
- The dependency for the whole GROUP BY key is no longer skipped because
of a narrower dependency that doesn't hold across NULLs.
## Are these changes tested?
Yes. Regression tests are added in `functional_dependencies.slt` for
nullable `UNIQUE`, `NOT NULL UNIQUE`, LEFT JOIN with a primary key, and
GROUP BY over a nullable `UNIQUE` column.
## Are there any user-facing changes?
Yes: incorrect results are fixed. There is also an API change: the
public `FunctionalDependence` struct has a new `null_equality` field, so
code that builds it with a struct literal must switch to
`FunctionalDependence::new(..)`. This is documented in the 56.0.0
upgrade guide.1 parent 622354f commit de9d1f9
4 files changed
Lines changed: 151 additions & 16 deletions
File tree
- datafusion
- common/src
- expr/src/logical_plan
- sqllogictest/test_files
- docs/source/library-user-guide/upgrading
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
147 | 154 | | |
148 | 155 | | |
149 | 156 | | |
| |||
168 | 175 | | |
169 | 176 | | |
170 | 177 | | |
| 178 | + | |
| 179 | + | |
171 | 180 | | |
172 | 181 | | |
173 | 182 | | |
| |||
177 | 186 | | |
178 | 187 | | |
179 | 188 | | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
180 | 208 | | |
181 | 209 | | |
182 | 210 | | |
| |||
301 | 329 | | |
302 | 330 | | |
303 | 331 | | |
| 332 | + | |
304 | 333 | | |
305 | 334 | | |
306 | 335 | | |
| |||
321 | 350 | | |
322 | 351 | | |
323 | 352 | | |
324 | | - | |
| 353 | + | |
| 354 | + | |
325 | 355 | | |
326 | 356 | | |
327 | 357 | | |
| |||
386 | 416 | | |
387 | 417 | | |
388 | 418 | | |
389 | | - | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
390 | 425 | | |
391 | 426 | | |
392 | 427 | | |
| |||
432 | 467 | | |
433 | 468 | | |
434 | 469 | | |
| 470 | + | |
435 | 471 | | |
436 | 472 | | |
437 | 473 | | |
| |||
470 | 506 | | |
471 | 507 | | |
472 | 508 | | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
473 | 517 | | |
474 | 518 | | |
475 | 519 | | |
476 | 520 | | |
477 | 521 | | |
478 | 522 | | |
479 | | - | |
| 523 | + | |
| 524 | + | |
480 | 525 | | |
481 | 526 | | |
482 | 527 | | |
| |||
497 | 542 | | |
498 | 543 | | |
499 | 544 | | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
500 | 548 | | |
501 | 549 | | |
502 | 550 | | |
503 | 551 | | |
504 | | - | |
505 | 552 | | |
506 | | - | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
507 | 556 | | |
508 | 557 | | |
509 | 558 | | |
| |||
617 | 666 | | |
618 | 667 | | |
619 | 668 | | |
620 | | - | |
621 | | - | |
622 | | - | |
| 669 | + | |
| 670 | + | |
623 | 671 | | |
624 | 672 | | |
625 | 673 | | |
626 | 674 | | |
627 | 675 | | |
628 | 676 | | |
| 677 | + | |
629 | 678 | | |
630 | 679 | | |
631 | 680 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3235 | 3235 | | |
3236 | 3236 | | |
3237 | 3237 | | |
3238 | | - | |
| 3238 | + | |
| 3239 | + | |
3239 | 3240 | | |
3240 | 3241 | | |
3241 | 3242 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | 139 | | |
143 | 140 | | |
144 | 141 | | |
145 | 142 | | |
146 | | - | |
147 | 143 | | |
| 144 | + | |
148 | 145 | | |
149 | 146 | | |
150 | 147 | | |
151 | 148 | | |
152 | 149 | | |
153 | | - | |
| 150 | + | |
154 | 151 | | |
155 | 152 | | |
156 | | - | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
157 | 168 | | |
158 | 169 | | |
159 | 170 | | |
| |||
163 | 174 | | |
164 | 175 | | |
165 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
166 | 228 | | |
167 | 229 | | |
168 | 230 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
668 | 668 | | |
669 | 669 | | |
670 | 670 | | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
671 | 694 | | |
672 | 695 | | |
673 | 696 | | |
| |||
0 commit comments