Repository navigation
Commit 27e81e9
authored
fix: do not duplicate volatile expressions when extracting leaf expressions (apache#24720)
## Which issue does this PR close?
- Closes apache#24678.
## Rationale for this change
A column alias denotes one value per row, and `WHERE p` may only return
rows for which `p` held for that row. Today the leaf-expression
extraction passes break both:
```sql
SELECT s, s['a'] AS field
FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 3));
```
| `s['a']` | `field` |
|---|---|
| 0.5159112071865757 | 0.4514238291986653 |
| 0.0029104680074608646 | 0.28979983332288195 |
| 0.48542729227457915 | 0.04499392663566881 |
`field` is defined as `s['a']` but differs from it on every row, because
the plan is
```
Projection: named_struct(Utf8("a"), random()) AS s, random() AS field
```
The `Filter` form is worse — it returns rows that fail their own
predicate:
```sql
SELECT bool_and(s['a'] > 0.5)
FROM (SELECT s FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 1000))
WHERE s['a'] > 0.5);
-- false; the predicate tested a different draw than the one in the returned `s`
```
Setting `datafusion.optimizer.enable_leaf_expression_pushdown = false`
returns the correct answer in both cases, so the rewrite alone changes
the meaning of the query.
**Root cause.** `build_extraction_projection_impl` merges an extraction
into the input projection by resolving column references through
`build_projection_replace_map`, i.e. by inlining each referenced
column's *defining* expression. Inlining a volatile definition produces
a second, independent evaluation. There was no volatility check in the
file.
This is the same invariant
`FileScanConfig::try_swapping_with_projection` already enforces for the
physical projection-pushdown path via `would_duplicate_costly_exprs`
(apache#23220) — the logical extraction path was missing it.
## What changes are included in this PR?
- `volatile_output_columns()` — a projection's output columns whose
definition is volatile.
- `would_duplicate_volatile()` — true when an extraction references one
of them.
- The guard is applied at the three places that can merge into an input
projection: `extract_from_plan` (pass 1:
Filter/Sort/Limit/Aggregate/Join), `split_and_push_projection` (pass 2),
and `try_push_into_inputs` (multi-input/Union routing). Each already had
a "leave the plan alone" return path.
The guard is targeted rather than blanket: for `ORDER BY s['a']` the
extraction still happens, stacked above the volatile projection instead
of merged into it, so the optimization is kept and the result is
correct.
### Relationship to apache#23691
@fornwall wondered on the issue whether apache#23691 already covers this. I
checked out that branch and ran both shapes against it: `SELECT s,
s['a']` is incidentally fixed there, but `WHERE s['a'] > 0.5` still
duplicates `random()` and still returns rows failing the predicate.
apache#23691 guards `KeepInPlace` *compute cost* in
`split_and_push_projection` only; volatility is a separate concern (one
duplication is already wrong, regardless of cost or placement) and pass
1 is a different code path. The two changes look independent to me and I
believe they compose, but I'd appreciate a second opinion on that from
whoever reviews apache#23691.
## Are these changes tested?
Yes.
- `datafusion/sqllogictest/test_files/projection_pushdown.slt` — a new
section beside the existing apache#23220 volatile section: two `EXPLAIN`s
pinning `random()` to a single occurrence, and two deterministic
`bool_and(...)` correctness queries. All four fail on `main` and pass
here.
- Two rule-level snapshot tests in `extract_leaf_expressions.rs`
covering the pass-2 projection merge and the pass-1 `Filter` extraction,
using a new test-only `PlacementTestUDF::with_volatility()`.
`cargo test -p datafusion-optimizer` (796), `cargo test -p datafusion
--lib --tests` (2049) and the full 504-file sqllogictest suite all pass;
`cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D
warnings` are clean.
## Are there any user-facing changes?
Queries that were silently returning wrong results now return correct
ones. No API change. In the affected shapes the extraction is skipped,
which can cost a small amount of column pruning — only when the
referenced column is defined by a volatile expression.
---
<sub>Per the ASF generative-tooling policy and DataFusion's AI-assisted
contribution guidance: this patch was prepared with AI assistance. The
core idea is the one described above — the merge path inlines a
referenced column's defining expression, which duplicates a volatile
definition — and the open question about how this composes with apache#23691
is flagged deliberately rather than glossed over.</sub>1 parent 17115f3 commit 27e81e9
3 files changed
Lines changed: 240 additions & 4 deletions
File tree
- datafusion
- optimizer/src
- test
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
60 | 112 | | |
61 | 113 | | |
62 | 114 | | |
| |||
195 | 247 | | |
196 | 248 | | |
197 | 249 | | |
198 | | - | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
199 | 262 | | |
200 | 263 | | |
201 | 264 | | |
| |||
917 | 980 | | |
918 | 981 | | |
919 | 982 | | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
920 | 993 | | |
921 | 994 | | |
922 | 995 | | |
| |||
1215 | 1288 | | |
1216 | 1289 | | |
1217 | 1290 | | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
1218 | 1300 | | |
1219 | 1301 | | |
1220 | 1302 | | |
| |||
1270 | 1352 | | |
1271 | 1353 | | |
1272 | 1354 | | |
1273 | | - | |
| 1355 | + | |
1274 | 1356 | | |
1275 | 1357 | | |
1276 | 1358 | | |
| |||
1283 | 1365 | | |
1284 | 1366 | | |
1285 | 1367 | | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
1286 | 1381 | | |
1287 | 1382 | | |
1288 | 1383 | | |
| |||
1960 | 2055 | | |
1961 | 2056 | | |
1962 | 2057 | | |
| 2058 | + | |
| 2059 | + | |
| 2060 | + | |
| 2061 | + | |
| 2062 | + | |
| 2063 | + | |
| 2064 | + | |
| 2065 | + | |
| 2066 | + | |
| 2067 | + | |
| 2068 | + | |
| 2069 | + | |
| 2070 | + | |
| 2071 | + | |
| 2072 | + | |
| 2073 | + | |
| 2074 | + | |
| 2075 | + | |
| 2076 | + | |
| 2077 | + | |
| 2078 | + | |
| 2079 | + | |
| 2080 | + | |
| 2081 | + | |
| 2082 | + | |
| 2083 | + | |
| 2084 | + | |
| 2085 | + | |
| 2086 | + | |
| 2087 | + | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
| 2095 | + | |
| 2096 | + | |
| 2097 | + | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
| 2102 | + | |
| 2103 | + | |
| 2104 | + | |
| 2105 | + | |
| 2106 | + | |
| 2107 | + | |
| 2108 | + | |
| 2109 | + | |
| 2110 | + | |
| 2111 | + | |
| 2112 | + | |
| 2113 | + | |
| 2114 | + | |
1963 | 2115 | | |
1964 | 2116 | | |
1965 | 2117 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
67 | 74 | | |
68 | 75 | | |
69 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2144 | 2144 | | |
2145 | 2145 | | |
2146 | 2146 | | |
| 2147 | + | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
| 2163 | + | |
| 2164 | + | |
| 2165 | + | |
| 2166 | + | |
| 2167 | + | |
| 2168 | + | |
| 2169 | + | |
| 2170 | + | |
| 2171 | + | |
| 2172 | + | |
| 2173 | + | |
| 2174 | + | |
| 2175 | + | |
| 2176 | + | |
| 2177 | + | |
| 2178 | + | |
| 2179 | + | |
| 2180 | + | |
| 2181 | + | |
| 2182 | + | |
| 2183 | + | |
| 2184 | + | |
| 2185 | + | |
| 2186 | + | |
| 2187 | + | |
| 2188 | + | |
| 2189 | + | |
| 2190 | + | |
| 2191 | + | |
| 2192 | + | |
| 2193 | + | |
| 2194 | + | |
| 2195 | + | |
| 2196 | + | |
| 2197 | + | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
| 2206 | + | |
| 2207 | + | |
| 2208 | + | |
| 2209 | + | |
| 2210 | + | |
| 2211 | + | |
| 2212 | + | |
| 2213 | + | |
| 2214 | + | |
| 2215 | + | |
| 2216 | + | |
| 2217 | + | |
| 2218 | + | |
| 2219 | + | |
| 2220 | + | |
| 2221 | + | |
| 2222 | + | |
| 2223 | + | |
2147 | 2224 | | |
2148 | 2225 | | |
2149 | 2226 | | |
| |||
0 commit comments