Repository navigation
Commit 5e55eac
committed
fix: evaluate struct-returning UDFs once across repeated field accesses
## Which issue does this PR close?
Closes apache#23655.
## Rationale for this change
`SELECT f(c)['a'], f(c)['b']`, where `f` is a struct-returning UDF, evaluates `f(c)` twice in the physical `ProjectionExec` — once per `get_field` — although the two calls are identical.
`CommonSubexprEliminate` already hoists the shared `f(c)` into a `__common_expr_N` column so it runs once. `PushDownLeafProjections` then undoes that, one pass at a time:
- After CSE, each field access reads its base from the hoist column, so `get_field`'s placement flips to `MoveTowardsLeafNodes` and the pass extracts it.
- Merging the extraction into the hoist projection inlines that column's definition back into every access (via `build_projection_replace_map`), re-duplicating the `KeepInPlace` UDF.
- `optimize_projections` drops the now-unused hoist column, and the two passes oscillate pass after pass — settling on a double evaluation.
This is the instability the `is_ignored` note in `common_subexpr_eliminate` already warns about.
## What changes are included in this PR?
`split_and_push_projection` now bails out — leaving the projection untouched — when merging the extractions into the input projection would inline a `KeepInPlace` definition that more than one extraction pair references. This mirrors the existing `optimize_projections` merge guard (apache#8296): a compute-once expression referenced more than once is kept in place instead of duplicated.
With the guard, CSE's hoist survives, `optimize_projections` keeps it, and the pipeline converges to a single evaluation:
Before:
```text
Projection: get_field(f(c), 'a'), get_field(f(c), 'b') -- f(c) twice
TableScan
```
After:
```text
Projection: get_field(__common_expr_1, 'a'), get_field(__common_expr_1, 'b')
Projection: f(c) AS __common_expr_1 -- f(c) once
TableScan
```
## Are these changes tested?
Yes.
- `test_struct_returning_udf_evaluated_once` (in `extract_leaf_expressions.rs`) runs the four interacting rules — CSE → extraction → pushdown → projection pruning — and asserts no field access wraps a fresh UDF call, plus an `insta` snapshot of the single-eval plan. It runs with the default pass budget, so any residual oscillation would surface as a double evaluation.
- `get_field_like` — a small test UDF added to `test/udfs.rs` — reproduces `GetFieldFunc`'s conditional placement (`MoveTowardsLeafNodes` only when the base is pushable and the keys are literals). This lets the optimizer crate model struct-field access without depending on `datafusion-functions`, which it intentionally does not.
`cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D warnings` are clean.
## Are there any user-facing changes?
No API changes. Affected plans evaluate a struct-returning UDF once instead of once per field access; query results are unchanged. The saving scales with the number of distinct field accesses on the same UDF.
## Follow-ups
- An end-to-end `.slt` test would need a struct-returning UDF registered in the sqllogictest harness: the built-in `named_struct` / `struct` constructors are folded into their fields by the field-over-struct-literal rewrite before this interaction can occur, so only an opaque (UDF) struct source reproduces it. Covered at the optimizer layer here instead.1 parent 67947b6 commit 5e55eac
2 files changed
Lines changed: 173 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
757 | 757 | | |
758 | 758 | | |
759 | 759 | | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
760 | 805 | | |
761 | 806 | | |
762 | 807 | | |
| |||
899 | 944 | | |
900 | 945 | | |
901 | 946 | | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
902 | 955 | | |
903 | 956 | | |
904 | 957 | | |
| |||
1246 | 1299 | | |
1247 | 1300 | | |
1248 | 1301 | | |
| 1302 | + | |
1249 | 1303 | | |
1250 | | - | |
| 1304 | + | |
1251 | 1305 | | |
1252 | 1306 | | |
1253 | 1307 | | |
| |||
3163 | 3217 | | |
3164 | 3218 | | |
3165 | 3219 | | |
| 3220 | + | |
| 3221 | + | |
| 3222 | + | |
| 3223 | + | |
| 3224 | + | |
| 3225 | + | |
| 3226 | + | |
| 3227 | + | |
| 3228 | + | |
| 3229 | + | |
| 3230 | + | |
| 3231 | + | |
| 3232 | + | |
| 3233 | + | |
| 3234 | + | |
| 3235 | + | |
| 3236 | + | |
| 3237 | + | |
| 3238 | + | |
| 3239 | + | |
| 3240 | + | |
| 3241 | + | |
| 3242 | + | |
| 3243 | + | |
| 3244 | + | |
| 3245 | + | |
| 3246 | + | |
| 3247 | + | |
| 3248 | + | |
| 3249 | + | |
| 3250 | + | |
| 3251 | + | |
| 3252 | + | |
| 3253 | + | |
| 3254 | + | |
| 3255 | + | |
| 3256 | + | |
| 3257 | + | |
| 3258 | + | |
| 3259 | + | |
| 3260 | + | |
| 3261 | + | |
| 3262 | + | |
| 3263 | + | |
| 3264 | + | |
| 3265 | + | |
| 3266 | + | |
| 3267 | + | |
| 3268 | + | |
| 3269 | + | |
| 3270 | + | |
| 3271 | + | |
| 3272 | + | |
3166 | 3273 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
0 commit comments