Repository navigation
Cherry-pick apache/datafusion#27086 - #198
Merged
LiaCastaneda merged 1 commit intoNov 5, 2026
Merged
Conversation
…7086) ## Which issue does this PR close? - Closes apache#25017. ## Rationale for this change A field access on a lambda parameter over a list of structs fails once the plan goes through Substrait: ```sql CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s; SELECT array_transform(s, x -> get_field(x, 'b')) FROM t; ``` The query runs fine on its own, but converting its plan to Substrait and consuming it back fails: ``` Error during planning: Field b not found in struct ``` This happens because Substrait types carry no struct field names; only relation schemas do. The plan declares the lambda parameter with a bare struct type, so the consumer names its fields `c0, c1`, and `get_field(x, 'b')` looks for a field that doesn't exist. The list `s` itself keeps its real names, as they come from the table schema. ## What changes are included in this PR? When the consumer builds a higher-order function, it now resolves the lambda parameters from the function's arguments (`Expr::resolve_lambda_variables`), as the `TypeCoercion` analyzer already does later. The parameter gets the list's element type, field names included, before anything checks the body. A function that uses a parameter of an outer lambda can't be resolved on its own, because only the outer function knows that parameter's type. For example, the inner call in `array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i))` uses `x`, so it is resolved together with the outer call. ## What is the testing strategy for this PR? `roundtrip_lambda_parameter_struct_field_names` in `roundtrip_logical_plan.rs` round trips a struct field access on: - a parameter over a column - a parameter over a computed list - the inner parameter of nested lambdas - an outer parameter read inside an inner lambda Without the change, the test fails with the error above. The existing lambda tests, including the dictionary and string view ones, still pass. ## Are there any user-facing changes? Substrait plans that access struct fields of lambda parameters now load. No API changes. (cherry picked from commit 238ae32)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## branch-55 #198 +/- ##
==========================================
Coverage 81.20% 81.20%
==========================================
Files 1111 1111
Lines 387932 387956 +24
Branches 387932 387956 +24
==========================================
+ Hits 315022 315045 +23
+ Misses 54333 54332 -1
- Partials 18577 18579 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ahmed-mez
approved these changes
Nov 5, 2026
LiaCastaneda
deleted the
lia.castaneda/cherry-pick/apache-pr-27086-20261105
branch
November 5, 2026 14:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-picks apache#27086