Repository navigation
Commit d5eb246
fix: evaluate struct-returning UDFs once across repeated field accesses (apache#23691)
## Which issue does this PR close?
- Closes apache#23655.
## Rationale for this change
`SELECT f(c)['a'], f(c)['b']`, where `f` is a struct-returning UDF,
evaluates `f(c)` twice in the physical `ProjectionExec` — once per
`get_field` — although the two calls are identical.
`CommonSubexprEliminate` already hoists the shared `f(c)` into a
`__common_expr_N` column so it runs once.
[`PushDownLeafProjections`](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/extract_leaf_expressions.rs)
then undoes that, one pass at a time:
- After CSE, each field access reads its base from the hoist column, so
`get_field`'s placement flips to `MoveTowardsLeafNodes` and the pass
extracts it.
- Merging the extraction into the hoist projection inlines that column's
definition back into every access (via `build_projection_replace_map`),
re-duplicating the `KeepInPlace` UDF.
- `optimize_projections` drops the now-unused hoist column, and the two
passes oscillate pass after pass — settling on a double evaluation.
This is the instability the `is_ignored` note in
`common_subexpr_eliminate` already warns about.
## What changes are included in this PR?
Adds a guard `merge_would_duplicate_kept_expr` and calls it from
`split_and_push_projection`, which now bails out — leaving the
projection untouched — when merging the extractions into the input
projection would inline a `KeepInPlace` definition that more than one
extraction pair references. This mirrors the existing
`optimize_projections` merge guard (apache#8296): a compute-once expression
referenced more than once is kept in place instead of duplicated.
With the guard, CSE's hoist survives, `optimize_projections` keeps it,
and the pipeline converges to a single evaluation:
Before:
```text
Projection: get_field(f(c), 'a'), get_field(f(c), 'b') -- f(c) twice
TableScan
```
After:
```text
Projection: get_field(__common_expr_1, 'a'), get_field(__common_expr_1, 'b')
Projection: f(c) AS __common_expr_1 -- f(c) once
TableScan
```
## Are these changes tested?
Yes.
- `test_struct_returning_udf_evaluated_once` (in
`extract_leaf_expressions.rs`) runs the four interacting rules — CSE →
extraction → pushdown → projection pruning — and asserts no field access
wraps a fresh UDF call, plus an `insta` snapshot of the single-eval
plan. It runs with the default pass budget, so any residual oscillation
would surface as a double evaluation.
- `get_field_like` — a small test UDF added to `test/udfs.rs` —
reproduces `GetFieldFunc`'s conditional placement
(`MoveTowardsLeafNodes` only when the base is pushable and the keys are
literals). This lets the optimizer crate model struct-field access
without depending on `datafusion-functions`, which it intentionally does
not.
`cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D
warnings` are clean.
## Are there any user-facing changes?
Yes, but only in the sense that a redundantly-evaluated path becomes
single-evaluation. No API or query-result changes: affected plans
evaluate a struct-returning UDF once instead of once per field access, a
saving that scales with the number of distinct field accesses on the
same UDF.
---------
Co-authored-by: tohuya6 <[email protected]>1 parent b163e82 commit d5eb246
2 files changed
Lines changed: 233 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
837 | 837 | | |
838 | 838 | | |
839 | 839 | | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
840 | 900 | | |
841 | 901 | | |
842 | 902 | | |
| |||
990 | 1050 | | |
991 | 1051 | | |
992 | 1052 | | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
993 | 1061 | | |
994 | 1062 | | |
995 | 1063 | | |
| |||
1346 | 1414 | | |
1347 | 1415 | | |
1348 | 1416 | | |
| 1417 | + | |
1349 | 1418 | | |
1350 | | - | |
| 1419 | + | |
1351 | 1420 | | |
1352 | 1421 | | |
1353 | 1422 | | |
| |||
3431 | 3500 | | |
3432 | 3501 | | |
3433 | 3502 | | |
| 3503 | + | |
| 3504 | + | |
| 3505 | + | |
| 3506 | + | |
| 3507 | + | |
| 3508 | + | |
| 3509 | + | |
| 3510 | + | |
| 3511 | + | |
| 3512 | + | |
| 3513 | + | |
| 3514 | + | |
| 3515 | + | |
| 3516 | + | |
| 3517 | + | |
| 3518 | + | |
| 3519 | + | |
| 3520 | + | |
| 3521 | + | |
| 3522 | + | |
| 3523 | + | |
| 3524 | + | |
| 3525 | + | |
| 3526 | + | |
| 3527 | + | |
| 3528 | + | |
| 3529 | + | |
| 3530 | + | |
| 3531 | + | |
| 3532 | + | |
| 3533 | + | |
| 3534 | + | |
| 3535 | + | |
| 3536 | + | |
| 3537 | + | |
| 3538 | + | |
| 3539 | + | |
| 3540 | + | |
| 3541 | + | |
| 3542 | + | |
| 3543 | + | |
| 3544 | + | |
| 3545 | + | |
| 3546 | + | |
| 3547 | + | |
| 3548 | + | |
| 3549 | + | |
| 3550 | + | |
| 3551 | + | |
| 3552 | + | |
| 3553 | + | |
| 3554 | + | |
| 3555 | + | |
| 3556 | + | |
| 3557 | + | |
| 3558 | + | |
| 3559 | + | |
| 3560 | + | |
| 3561 | + | |
| 3562 | + | |
| 3563 | + | |
| 3564 | + | |
| 3565 | + | |
| 3566 | + | |
| 3567 | + | |
| 3568 | + | |
| 3569 | + | |
| 3570 | + | |
| 3571 | + | |
| 3572 | + | |
| 3573 | + | |
| 3574 | + | |
| 3575 | + | |
| 3576 | + | |
| 3577 | + | |
| 3578 | + | |
| 3579 | + | |
| 3580 | + | |
| 3581 | + | |
| 3582 | + | |
| 3583 | + | |
| 3584 | + | |
| 3585 | + | |
| 3586 | + | |
| 3587 | + | |
| 3588 | + | |
| 3589 | + | |
| 3590 | + | |
| 3591 | + | |
| 3592 | + | |
| 3593 | + | |
| 3594 | + | |
| 3595 | + | |
| 3596 | + | |
| 3597 | + | |
| 3598 | + | |
| 3599 | + | |
| 3600 | + | |
| 3601 | + | |
3434 | 3602 | | |
3435 | 3603 | | |
3436 | 3604 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
0 commit comments