Repository navigation
Commit fb08efa
refactor: derive null-aware joins from the
The rule decided null-awareness with three special cases: a flag that
`PullUpCorrelatedExpr` set when it dropped a duplicated correlation and
cleared again for a list of plan nodes, a switch to a weaker column test
when the null-aware `LeftAnti` executor could not take the join, and an
empty key list that selected that weaker test at the constant `IN` call.
Replace them with one question. `IN` can only be UNKNOWN when its value
or the subquery output can be NULL inside the scope of an outer row. A
correlated conjunct such as `y = x` is never TRUE for a NULL, so a key it
compares cannot be NULL in scope. `PullUpCorrelatedExpr` now records every
correlated conjunct before it drops the duplicated one, and `JoinKeys`
asks the two sides of the `IN` predicate for their nullability against
those conjuncts. A correlation key that is NULL only empties the scope,
so it never makes the join null-aware.
The executor limits are then applied in one place. A `NOT IN` whose value
can be NULL and whose join has more than one key becomes a null-aware
`LeftMark` join, which takes any number of keys, plus a filter on the
mark. A `NOT IN` with a residual filter becomes the mark joins that
materialize its three-valued result, the same plan as a projected `IN`.
Both shapes failed to plan or gave wrong results before. The constant
value projection now applies to correlated subqueries too, since the
mark join accepts the correlation as a second key.
The guard for a grouping set above the dropped filter is gone with the
flag. That query is wrong on `main` for `EXISTS` as well, because the
pull up moves the filter above the aggregate; that is
apache#25519.
Co-Authored-By: Claude Fable 5.1 <[email protected]>IN value in scope1 parent 7bb69a5 commit fb08efa
5 files changed
Lines changed: 470 additions & 350 deletions
File tree
- datafusion
- optimizer/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
38 | | - | |
| 37 | + | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
78 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
79 | 80 | | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
92 | 87 | | |
93 | 88 | | |
94 | 89 | | |
| |||
110 | 105 | | |
111 | 106 | | |
112 | 107 | | |
113 | | - | |
| 108 | + | |
114 | 109 | | |
115 | 110 | | |
116 | 111 | | |
| |||
189 | 184 | | |
190 | 185 | | |
191 | 186 | | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | 187 | | |
199 | 188 | | |
200 | 189 | | |
| |||
206 | 195 | | |
207 | 196 | | |
208 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
209 | 203 | | |
210 | 204 | | |
211 | | - | |
212 | 205 | | |
213 | | - | |
214 | 206 | | |
215 | 207 | | |
216 | 208 | | |
| |||
504 | 496 | | |
505 | 497 | | |
506 | 498 | | |
507 | | - | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
512 | | - | |
513 | | - | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
518 | | - | |
519 | | - | |
520 | | - | |
521 | | - | |
522 | | - | |
523 | | - | |
524 | | - | |
525 | | - | |
526 | 499 | | |
527 | 500 | | |
528 | 501 | | |
| |||
0 commit comments