Repository navigation
Commit 46bbf0d
authored
fix: preserve null semantics in expression simplification (apache#24258)
## Which issue does this PR close?
- Closes apache#24246.
## Rationale for this change
Set algebra over `IN` / `NOT IN` expressions can erase SQL
three-valued-logic semantics when either the tested expression or a list
item is nullable. The `array_has` to `IN` rewrite has a similar
mismatch: an array containing NULL does not behave like an `IN` list
containing NULL.
Preserving those NULL branches is required in general expression
contexts. In a filter, however, both FALSE and NULL reject the row, so
the optimizer can still produce the strongest equivalent plan there.
## What changes are included in this PR?
- Keep `IN` / `NOT IN` set operations unchanged when list items may be
NULL, and preserve the NULL branch when an otherwise safe set operation
becomes a constant for a nullable tested expression.
- Avoid rewriting `array_has` to `IN` when a nullable array element
would change the result, while retaining the safe non-volatile
same-expression case.
- Extend `EliminateFilter` to simplify positive `AND` / `OR` trees
according to filter semantics. This restores plans such as
`EmptyRelation` for disjoint `IN` predicates while retaining correct
NULL behavior in projections. The simplification deliberately does not
cross `NOT` or arbitrary expressions.
- Add unit, runtime, and SQL logic regression coverage for NULL results
and the resulting optimized plans.
## Are these changes tested?
Yes. The following checks pass:
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./dev/rust_lint.sh`
- `cargo test --profile=ci --test sqllogictests`
- `prlimit --nofile=65536:65536 -- env RUST_BACKTRACE=1 cargo test
--profile ci --exclude datafusion-examples --exclude
datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests
--bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
## Are there any user-facing changes?
Yes. Expressions involving nullable inputs retain their SQL
three-valued-logic results, while equivalent filter predicates continue
to optimize away rejected rows. There are no public API changes.1 parent cc29ea1 commit 46bbf0d
8 files changed
Lines changed: 625 additions & 80 deletions
File tree
- datafusion
- functions-nested/src
- optimizer/src
- simplify_expressions
- sqllogictest/test_files
- array
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
131 | | - | |
| 131 | + | |
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
| |||
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
153 | | - | |
| 153 | + | |
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
157 | | - | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
158 | 166 | | |
159 | 167 | | |
160 | 168 | | |
| |||
167 | 175 | | |
168 | 176 | | |
169 | 177 | | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
176 | 199 | | |
177 | 200 | | |
178 | 201 | | |
| |||
1197 | 1220 | | |
1198 | 1221 | | |
1199 | 1222 | | |
1200 | | - | |
| 1223 | + | |
1201 | 1224 | | |
1202 | 1225 | | |
1203 | | - | |
| 1226 | + | |
1204 | 1227 | | |
1205 | 1228 | | |
1206 | 1229 | | |
| |||
1263 | 1286 | | |
1264 | 1287 | | |
1265 | 1288 | | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
1266 | 1361 | | |
1267 | 1362 | | |
1268 | 1363 | | |
| |||
0 commit comments