You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 79de5e9
Browse filesBrowse the repository at this point in the historyBrowse files
fix(lambda): only push referenced params into the merged batch (apache#24162) (#166)
* fix(lambda): only push referenced params into the merged batch (apache#24162)
## Which issue does this PR close?
basically this PR apache#22853 + a
few more tests
## Rationale for this change
The current lambdas in DF only take a single parameter `(v -> ...)`, so
nobody had noticed that `LambdaExpr` mishandles lambdas with more than
one parameter. The bug surfaced while working on `transform_values`
(apache#22689), which needs `(k, v) -> expr ` two parameters, one of which is
very often unused (e.g. `(k, v) -> v * 2`, k never referenced).
The bug is that when a higher order function with more than 1 param
evaluates a lambda, it fills each parameter into a slot based on its
declared position — for example for `(k, v) -> v` `k` always goes into
slot 0, `v` always into slot 1. `LambdaExpr` separately scans the body
and renumbers whatever it finds referenced into a dense `0..n` range, to
avoid carrying around columns nothing uses (like `v` in this case). That
renumbering is fine for outer captures, but applying it to the lambda's
own parameters is wrong, because it changes where the body looks for a
value without changing where the evaluator put it.
### Example:
in `(k, v) -> v` `v` is declared second (slot 1), but since it's the
only parameter the body references, the renumbering logic reassigns it
to slot 0. The evaluator, unaware of this, writes `k`'s values into slot
0 and `v`'s into slot 1. So the body ends up reading slot 0 expecting
`v` — and gets `k` instead. So the results end up being incorrect.
## What changes are included in this PR?
- `LambdaExpr` now computes `used_params`: which is the subset of its
own declared parameters that are actually referenced in the body.
- `LambdaArgument::new` takes `used_params` and only pushes the
referenced parameters in the body into the merged batch, in original
declaration order — so the body's indices always line up with what's
actually built.
- `HigherOrderFunctionExpr::evaluate` forwards `lambda.used_params()` to
`LambdaArgument::new`
## Are these changes tested?
yes, added two new tests one for the unused-parameter case and
nested-lambda for the shadowing case.
## Are there any user-facing changes?
The only public api change is on `LambdaArgument::new ` which now
requires a new argument: `used_params: &HashSet<String>`, however
LambdaArgument::new is very unlikely to be called outside datafusion,
see
[this](apache#22853 (comment))
comment
(cherry picked from commit 4e6acfe)
* Adjust to API change
0 commit comments