Repository navigation
fix: keep struct field names on Substrait lambda parameters - #27086
Merged
gabotechs merged 3 commits intoNov 4, 2026
Merged
Conversation
LiaCastaneda
marked this pull request as ready for review
November 4, 2026 12:27
gabotechs
approved these changes
Nov 4, 2026
gabotechs
left a comment
Contributor
There was a problem hiding this comment.
👍 Looks good, thanks @LiaCastaneda!
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 4, 2026
LiaCastaneda
added a commit
to DataDog/datafusion
that referenced
this pull request
Nov 5, 2026
…7086) (#198) ## Which issue does this PR close? - Closes apache#25017. ## Rationale for this change A field access on a lambda parameter over a list of structs fails once the plan goes through Substrait: ```sql CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s; SELECT array_transform(s, x -> get_field(x, 'b')) FROM t; ``` The query runs fine on its own, but converting its plan to Substrait and consuming it back fails: ``` Error during planning: Field b not found in struct ``` This happens because Substrait types carry no struct field names; only relation schemas do. The plan declares the lambda parameter with a bare struct type, so the consumer names its fields `c0, c1`, and `get_field(x, 'b')` looks for a field that doesn't exist. The list `s` itself keeps its real names, as they come from the table schema. ## What changes are included in this PR? When the consumer builds a higher-order function, it now resolves the lambda parameters from the function's arguments (`Expr::resolve_lambda_variables`), as the `TypeCoercion` analyzer already does later. The parameter gets the list's element type, field names included, before anything checks the body. A function that uses a parameter of an outer lambda can't be resolved on its own, because only the outer function knows that parameter's type. For example, the inner call in `array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i))` uses `x`, so it is resolved together with the outer call. ## What is the testing strategy for this PR? `roundtrip_lambda_parameter_struct_field_names` in `roundtrip_logical_plan.rs` round trips a struct field access on: - a parameter over a column - a parameter over a computed list - the inner parameter of nested lambdas - an outer parameter read inside an inner lambda Without the change, the test fails with the error above. The existing lambda tests, including the dictionary and string view ones, still pass. ## Are there any user-facing changes? Substrait plans that access struct fields of lambda parameters now load. No API changes. (cherry picked from commit 238ae32)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
A field access on a lambda parameter over a list of structs fails once the plan goes through Substrait:
The query runs fine on its own, but converting its plan to Substrait and consuming it back fails:
This happens because Substrait types carry no struct field names; only relation schemas do. The plan declares the lambda parameter with a bare struct type, so the consumer names its fields
c0, c1, andget_field(x, 'b')looks for a field that doesn't exist. The listsitself keeps its real names, as they come from the table schema.What changes are included in this PR?
When the consumer builds a higher-order function, it now resolves the lambda parameters from the function's arguments (
Expr::resolve_lambda_variables), as theTypeCoercionanalyzer already does later. The parameter gets the list's element type, field names included, before anything checks the body.A function that uses a parameter of an outer lambda can't be resolved on its own, because only the outer function knows that parameter's type. For example, the inner call in
array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i))usesx, so it is resolved together with the outer call.What is the testing strategy for this PR?
roundtrip_lambda_parameter_struct_field_namesinroundtrip_logical_plan.rsround trips a struct field access on:Without the change, the test fails with the error above. The existing lambda tests, including the dictionary and string view ones, still pass.
Are there any user-facing changes?
Substrait plans that access struct fields of lambda parameters now load. No API changes.