Repository navigation
Commit 836b4dd
authored
fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias (#26551)
## Which issue does this PR close?
- Closes #26504.
## Rationale for this change
When a correlated subquery contains an aliased derived table
(`SubqueryAlias`) with a correlated predicate inside it and a `Filter`
directly above the `SubqueryAlias` whose predicates are all pulled up as
join filters (for example `WHERE x.d > 0`), `PullUpCorrelatedExpr::f_up`
eliminates the `Filter` and rebuilds `new_plan` equal to
`*plan_filter.input` (`SubqueryAlias`).
Previously, `self.correlated_subquery_cols_map.insert(new_plan.clone(),
correlated_subquery_cols)` overwrote the entry that `SubqueryAlias` had
already recorded for `new_plan`. Because `collect_local_correlated_cols`
treats `SubqueryAlias` as a leaf node and does not recurse into its
input, any `Projection` or `Aggregate` above the eliminated `Filter`
lost the correlated columns from the derived table and failed with
`Schema error: No field named __correlated_sq_1.c`.
## What changes are included in this PR?
- Use
`.entry(new_plan.clone()).or_default().extend(correlated_subquery_cols)`
in `PullUpCorrelatedExpr::f_up` when handling `LogicalPlan::Filter` so
existing correlated columns recorded on `new_plan` are preserved when
the `Filter` is removed.
- Add optimizer unit tests in `decorrelate_predicate_subquery` and
`scalar_subquery_to_join` and sqllogictest coverage in `subquery.slt`.
## What is the testing strategy for this PR?
Covered by unit tests in `datafusion-optimizer`
(`exists_subquery_filter_above_correlated_derived_table` and
`correlated_scalar_subquery_filter_above_correlated_derived_table`) and
sqllogictest cases in `subquery.slt`.
## Are there any user-facing changes?
Correlated `EXISTS`, `IN`, and scalar subqueries that filter an aliased
derived table on an outer column now plan and execute instead of failing
with a schema error.1 parent 702803d commit 836b4dd
4 files changed
Lines changed: 102 additions & 2 deletions
File tree
- datafusion
- optimizer/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
494 | 494 | | |
495 | 495 | | |
496 | 496 | | |
497 | | - | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
498 | 500 | | |
499 | 501 | | |
500 | 502 | | |
| |||
506 | 508 | | |
507 | 509 | | |
508 | 510 | | |
509 | | - | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
510 | 514 | | |
511 | 515 | | |
512 | 516 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3507 | 3507 | | |
3508 | 3508 | | |
3509 | 3509 | | |
| 3510 | + | |
| 3511 | + | |
| 3512 | + | |
| 3513 | + | |
| 3514 | + | |
| 3515 | + | |
| 3516 | + | |
| 3517 | + | |
| 3518 | + | |
| 3519 | + | |
| 3520 | + | |
| 3521 | + | |
| 3522 | + | |
| 3523 | + | |
| 3524 | + | |
| 3525 | + | |
| 3526 | + | |
| 3527 | + | |
| 3528 | + | |
| 3529 | + | |
| 3530 | + | |
| 3531 | + | |
| 3532 | + | |
| 3533 | + | |
| 3534 | + | |
| 3535 | + | |
| 3536 | + | |
| 3537 | + | |
| 3538 | + | |
| 3539 | + | |
| 3540 | + | |
| 3541 | + | |
| 3542 | + | |
| 3543 | + | |
3510 | 3544 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1276 | 1276 | | |
1277 | 1277 | | |
1278 | 1278 | | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
1279 | 1321 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4195 | 4195 | | |
4196 | 4196 | | |
4197 | 4197 | | |
| 4198 | + | |
| 4199 | + | |
| 4200 | + | |
| 4201 | + | |
| 4202 | + | |
| 4203 | + | |
| 4204 | + | |
| 4205 | + | |
| 4206 | + | |
| 4207 | + | |
| 4208 | + | |
| 4209 | + | |
| 4210 | + | |
| 4211 | + | |
| 4212 | + | |
| 4213 | + | |
| 4214 | + | |
| 4215 | + | |
| 4216 | + | |
| 4217 | + | |
0 commit comments