Repository navigation
Commit 6cf7cb5
authored
fix: preserve lambda bindings in nested-loop join projection pushdown (apache#26399)
## Which issue does this PR close?
- Closes apache#26396.
## Rationale for this change
A nested-loop join whose lambda captures a column from the other input
can fail during physical optimization with `unable to unwrap lambda`,
even though the query executes correctly with physical optimization
disabled.
These queries should work with the default optimizer while retaining
projection pushdown for expressions that can be evaluated on one join
input.
## What changes are included in this PR?
- Keep each lambda and its body in the enclosing high-order function's
scope when extracting join-filter expressions. Ordinary sibling
expressions, other function arguments, and complete high-order
expressions that depend on only one input remain eligible for pushdown.
- Remap captured columns and rebase lambda parameter indices when moving
an expression to an input projection, extending the intermediate filter
schema, and pruning unused filter columns. Nested lambdas retain their
relative parameter offsets and bindings.
- Add `nested_loop_join_lambda.slt` with result and physical-plan
regressions.
This change preserves lambda bodies within their enclosing function; it
does not introduce hoisting of individual computations from inside a
lambda body.
## What is the testing strategy for this PR?
The new SQL test covers the original reproducer with either join-input
order, mixed predicates, nested lambdas, parameter shadowing, empty/NULL
arrays, and extraction of complete high-order functions onto either
input. Different input and filter schema widths exercise parameter
rebasing. Three `EXPLAIN` checks verify that ordinary expressions and
complete high-order functions are still pushed down.
## Are there any user-facing changes?
Nested-loop joins with lambda predicates now plan and execute with the
default physical optimizer. There are no public API or configuration
changes.1 parent fe612e2 commit 6cf7cb5
2 files changed
Lines changed: 275 additions & 41 deletions
File tree
- datafusion
- physical-optimizer/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | | - | |
| 33 | + | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
| 133 | + | |
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
| |||
219 | 219 | | |
220 | 220 | | |
221 | 221 | | |
222 | | - | |
| 222 | + | |
223 | 223 | | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
224 | 230 | | |
225 | 231 | | |
226 | 232 | | |
| |||
240 | 246 | | |
241 | 247 | | |
242 | 248 | | |
243 | | - | |
| 249 | + | |
244 | 250 | | |
245 | 251 | | |
246 | 252 | | |
247 | 253 | | |
248 | 254 | | |
249 | 255 | | |
250 | | - | |
251 | | - | |
| 256 | + | |
252 | 257 | | |
253 | 258 | | |
254 | 259 | | |
| |||
265 | 270 | | |
266 | 271 | | |
267 | 272 | | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
283 | 285 | | |
284 | | - | |
285 | | - | |
| 286 | + | |
| 287 | + | |
286 | 288 | | |
287 | 289 | | |
288 | | - | |
| 290 | + | |
289 | 291 | | |
290 | 292 | | |
291 | 293 | | |
| |||
300 | 302 | | |
301 | 303 | | |
302 | 304 | | |
| 305 | + | |
303 | 306 | | |
304 | 307 | | |
305 | 308 | | |
| |||
327 | 330 | | |
328 | 331 | | |
329 | 332 | | |
| 333 | + | |
330 | 334 | | |
331 | 335 | | |
332 | 336 | | |
| |||
339 | 343 | | |
340 | 344 | | |
341 | 345 | | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
342 | 351 | | |
343 | 352 | | |
344 | 353 | | |
| |||
378 | 387 | | |
379 | 388 | | |
380 | 389 | | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
397 | 405 | | |
398 | 406 | | |
399 | 407 | | |
| |||
420 | 428 | | |
421 | 429 | | |
422 | 430 | | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
423 | 465 | | |
424 | 466 | | |
425 | 467 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
0 commit comments