Skip to content

fix: keep struct field names on Substrait lambda parameters - #27086

Merged
gabotechs merged 3 commits into
apache:mainfrom
LiaCastaneda:substrait-lambda-parameter-types
Nov 4, 2026
Merged

gabotechs merged 3 commits into
apache:mainfrom
LiaCastaneda:substrait-lambda-parameter-types

Conversation

@LiaCastaneda

@LiaCastaneda LiaCastaneda commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A field access on a lambda parameter over a list of structs fails once the plan goes through Substrait:

CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s;
SELECT array_transform(s, x -> get_field(x, 'b')) FROM t;

The query runs fine on its own, but converting its plan to Substrait and consuming it back fails:

Error during planning: Field b not found in struct

This happens because Substrait types carry no struct field names; only relation schemas do. The plan declares the lambda parameter with a bare struct type, so the consumer names its fields c0, c1, and get_field(x, 'b') looks for a field that doesn't exist. The list s itself keeps its real names, as they come from the table schema.

What changes are included in this PR?

When the consumer builds a higher-order function, it now resolves the lambda parameters from the function's arguments (Expr::resolve_lambda_variables), as the TypeCoercion analyzer already does later. The parameter gets the list's element type, field names included, before anything checks the body.

A function that uses a parameter of an outer lambda can't be resolved on its own, because only the outer function knows that parameter's type. For example, the inner call in array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i)) uses x, so it is resolved together with the outer call.

What is the testing strategy for this PR?

roundtrip_lambda_parameter_struct_field_names in roundtrip_logical_plan.rs round trips a struct field access on:

  • a parameter over a column
  • a parameter over a computed list
  • the inner parameter of nested lambdas
  • an outer parameter read inside an inner lambda

Without the change, the test fails with the error above. The existing lambda tests, including the dictionary and string view ones, still pass.

Are there any user-facing changes?

Substrait plans that access struct fields of lambda parameters now load. No API changes.

@github-actions github-actions Bot added the substrait Changes to the substrait crate label Nov 4, 2026
@LiaCastaneda
LiaCastaneda marked this pull request as ready for review November 4, 2026 12:27

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good, thanks @LiaCastaneda!

@gabotechs
gabotechs added this pull request to the merge queue Nov 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 4, 2026
@gabotechs
gabotechs added this pull request to the merge queue Nov 4, 2026
Merged via the queue into apache:main with commit 238ae32 Nov 4, 2026
44 checks passed
LiaCastaneda added a commit to DataDog/datafusion that referenced this pull request Nov 5, 2026
…7086) (#198)

## Which issue does this PR close?

- Closes apache#25017.

## Rationale for this change

A field access on a lambda parameter over a list of structs fails once
the plan goes through Substrait:

```sql
CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s;
SELECT array_transform(s, x -> get_field(x, 'b')) FROM t;
```

The query runs fine on its own, but converting its plan to Substrait and
consuming it back fails:

```
Error during planning: Field b not found in struct
```

This happens because Substrait types carry no struct field names; only
relation schemas do. The plan declares the lambda parameter with a bare
struct type, so the consumer names its fields `c0, c1`, and
`get_field(x, 'b')` looks for a field that doesn't exist. The list `s`
itself keeps its real names, as they come from the table schema.

## What changes are included in this PR?

When the consumer builds a higher-order function, it now resolves the
lambda parameters from the function's arguments
(`Expr::resolve_lambda_variables`), as the `TypeCoercion` analyzer
already does later. The parameter gets the list's element type, field
names included, before anything checks the body.

A function that uses a parameter of an outer lambda can't be resolved on
its own, because only the outer function knows that parameter's type.
For example, the inner call in `array_transform(s, x ->
array_transform([1, 2], i -> get_field(x, 'a') + i))` uses `x`, so it is
resolved together with the outer call.

## What is the testing strategy for this PR?

`roundtrip_lambda_parameter_struct_field_names` in
`roundtrip_logical_plan.rs` round trips a struct field access on:

- a parameter over a column
- a parameter over a computed list
- the inner parameter of nested lambdas
- an outer parameter read inside an inner lambda

Without the change, the test fails with the error above. The existing
lambda tests, including the dictionary and string view ones, still pass.

## Are there any user-facing changes?

Substrait plans that access struct fields of lambda parameters now load.
No API changes.

(cherry picked from commit 238ae32)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait round trip fails for struct field access on lambda parameters

2 participants