Repository navigation
Commit 1e77af8
authored
fix: don't duplicate volatile expressions when pushing projection into file scan (apache#23395)
## Which issue does this PR close?
- Closes apache#23220.
## Rationale for this change
The reported bug showed that when volatile (non-deterministic)
expressions are referenced multiple times in an outer function the query
results can be incorrect.
To produce correct results, a volatile expression (e.g. `random()`,
`uuid()`) aliased once in a subquery and referenced multiple times must
be evaluated once and reused. Since 52.0.0 this evaluation pattern for
volatile functions has been broken; the physical projection-pushdown
rule merges the outer projection into the file `DataSourceExec`,
inlining the aliased volatile expression at each reference site. Instead
of being evaluated once, the volatile expression is evaluated N times,
so the references diverge:
```sql
SELECT s.r AS x, s.r AS y FROM (SELECT random() AS r FROM t) AS s; -- x != y on >= 52.0.0
```
This was correct in 51.0.0 and regressed in 52.0.0/53.0.0. It reproduces
on file scans (Parquet/CSV) but not in-memory tables, and was surfaced
downstream in Ibis. Worth noting, apache#10337 appears to report the same
class of bug but a different cause (I'll try to look into that issue
soon as well).
## What changes are included in this PR?
`FileScanConfig::try_swapping_with_projection` now declines to merge a
projection into the file source when the merge would inline a volatile
expression that the incoming projection references. The check reuses the
existing `is_volatile()` utility from `datafusion_physical_expr_common`
— the same volatility gate the physical `ProjectionPushdown` and
`FilterPushdown` rules already apply — so file-source projection merging
is now consistent with them. Deterministic expressions still merge
freely.
The guard blocks when a volatile inner expression is referenced at least
once (not only more than once), because a single outer expression can
itself duplicate the value (e.g.`r + r`).
## Are these changes tested?
Yes:
- Unit tests in `file_scan_config.rs`:
- volatile referenced ≥1× → blocked;
- deterministic computed and column-only → allowed;
- unreferenced volatile → allowed;
- single-expression self-reference (`r + r`) → blocked;
- volatile nested in arithmetic → blocked;
- empty → allowed.
- A regression test in `projection_pushdown.slt` asserting `x = y` for
the aliased-`random()` pattern over a Parquet scan.
- Full `datafusion-sqllogictest` suite passes.
## Are there any user-facing changes?
No API changes! Query results for the affected pattern are corrected (a
volatile expression aliased in a subquery is no longer duplicated by
projection pushdown into a file scan). Physical plans for such queries
now retain a `ProjectionExec` above the scan rather than inlining the
volatile expression into the `DataSourceExec` projection.1 parent f4a700d commit 1e77af8
2 files changed
Lines changed: 260 additions & 1 deletion
File tree
- datafusion
- datasource/src/file_scan_config
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
625 | 625 | | |
626 | 626 | | |
627 | 627 | | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
628 | 673 | | |
629 | 674 | | |
630 | 675 | | |
| |||
911 | 956 | | |
912 | 957 | | |
913 | 958 | | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
914 | 968 | | |
915 | 969 | | |
916 | 970 | | |
| |||
3314 | 3368 | | |
3315 | 3369 | | |
3316 | 3370 | | |
| 3371 | + | |
| 3372 | + | |
| 3373 | + | |
| 3374 | + | |
| 3375 | + | |
| 3376 | + | |
| 3377 | + | |
| 3378 | + | |
| 3379 | + | |
| 3380 | + | |
| 3381 | + | |
| 3382 | + | |
| 3383 | + | |
| 3384 | + | |
| 3385 | + | |
| 3386 | + | |
| 3387 | + | |
| 3388 | + | |
| 3389 | + | |
| 3390 | + | |
| 3391 | + | |
| 3392 | + | |
| 3393 | + | |
| 3394 | + | |
| 3395 | + | |
| 3396 | + | |
| 3397 | + | |
| 3398 | + | |
| 3399 | + | |
| 3400 | + | |
| 3401 | + | |
| 3402 | + | |
| 3403 | + | |
| 3404 | + | |
| 3405 | + | |
| 3406 | + | |
| 3407 | + | |
| 3408 | + | |
| 3409 | + | |
| 3410 | + | |
| 3411 | + | |
| 3412 | + | |
| 3413 | + | |
| 3414 | + | |
| 3415 | + | |
| 3416 | + | |
| 3417 | + | |
| 3418 | + | |
| 3419 | + | |
| 3420 | + | |
| 3421 | + | |
| 3422 | + | |
| 3423 | + | |
| 3424 | + | |
| 3425 | + | |
| 3426 | + | |
| 3427 | + | |
| 3428 | + | |
| 3429 | + | |
| 3430 | + | |
| 3431 | + | |
| 3432 | + | |
| 3433 | + | |
| 3434 | + | |
| 3435 | + | |
| 3436 | + | |
| 3437 | + | |
| 3438 | + | |
| 3439 | + | |
| 3440 | + | |
| 3441 | + | |
| 3442 | + | |
| 3443 | + | |
| 3444 | + | |
| 3445 | + | |
| 3446 | + | |
| 3447 | + | |
| 3448 | + | |
| 3449 | + | |
| 3450 | + | |
| 3451 | + | |
| 3452 | + | |
| 3453 | + | |
| 3454 | + | |
| 3455 | + | |
| 3456 | + | |
| 3457 | + | |
| 3458 | + | |
| 3459 | + | |
| 3460 | + | |
| 3461 | + | |
| 3462 | + | |
| 3463 | + | |
| 3464 | + | |
| 3465 | + | |
| 3466 | + | |
| 3467 | + | |
| 3468 | + | |
| 3469 | + | |
| 3470 | + | |
| 3471 | + | |
| 3472 | + | |
| 3473 | + | |
| 3474 | + | |
| 3475 | + | |
| 3476 | + | |
| 3477 | + | |
| 3478 | + | |
| 3479 | + | |
| 3480 | + | |
| 3481 | + | |
| 3482 | + | |
| 3483 | + | |
| 3484 | + | |
| 3485 | + | |
| 3486 | + | |
| 3487 | + | |
| 3488 | + | |
| 3489 | + | |
| 3490 | + | |
| 3491 | + | |
| 3492 | + | |
| 3493 | + | |
| 3494 | + | |
| 3495 | + | |
| 3496 | + | |
| 3497 | + | |
| 3498 | + | |
| 3499 | + | |
| 3500 | + | |
| 3501 | + | |
| 3502 | + | |
| 3503 | + | |
| 3504 | + | |
| 3505 | + | |
| 3506 | + | |
| 3507 | + | |
| 3508 | + | |
| 3509 | + | |
| 3510 | + | |
| 3511 | + | |
| 3512 | + | |
| 3513 | + | |
| 3514 | + | |
| 3515 | + | |
| 3516 | + | |
| 3517 | + | |
| 3518 | + | |
| 3519 | + | |
| 3520 | + | |
| 3521 | + | |
| 3522 | + | |
| 3523 | + | |
| 3524 | + | |
| 3525 | + | |
| 3526 | + | |
| 3527 | + | |
| 3528 | + | |
| 3529 | + | |
| 3530 | + | |
| 3531 | + | |
| 3532 | + | |
| 3533 | + | |
| 3534 | + | |
| 3535 | + | |
| 3536 | + | |
| 3537 | + | |
| 3538 | + | |
| 3539 | + | |
| 3540 | + | |
| 3541 | + | |
| 3542 | + | |
| 3543 | + | |
| 3544 | + | |
| 3545 | + | |
3317 | 3546 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2077 | 2077 | | |
2078 | 2078 | | |
2079 | 2079 | | |
| 2080 | + | |
| 2081 | + | |
| 2082 | + | |
| 2083 | + | |
| 2084 | + | |
| 2085 | + | |
| 2086 | + | |
| 2087 | + | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
| 2095 | + | |
| 2096 | + | |
| 2097 | + | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
| 2102 | + | |
| 2103 | + | |
| 2104 | + | |
| 2105 | + | |
| 2106 | + | |
| 2107 | + | |
| 2108 | + | |
| 2109 | + | |
0 commit comments