Skip to content

Cherry-pick apache/datafusion#27086 - #198

Merged
LiaCastaneda merged 1 commit into
branch-55from
lia.castaneda/cherry-pick/apache-pr-27086-20261105
Nov 5, 2026
Merged

LiaCastaneda merged 1 commit into
branch-55from
lia.castaneda/cherry-pick/apache-pr-27086-20261105

Conversation

@LiaCastaneda

Copy link
Copy Markdown

cherry-picks apache#27086

…7086)

## Which issue does this PR close?

- Closes apache#25017.

## Rationale for this change

A field access on a lambda parameter over a list of structs fails once
the plan goes through Substrait:

```sql
CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s;
SELECT array_transform(s, x -> get_field(x, 'b')) FROM t;
```

The query runs fine on its own, but converting its plan to Substrait and
consuming it back fails:

```
Error during planning: Field b not found in struct
```

This happens because Substrait types carry no struct field names; only
relation schemas do. The plan declares the lambda parameter with a bare
struct type, so the consumer names its fields `c0, c1`, and
`get_field(x, 'b')` looks for a field that doesn't exist. The list `s`
itself keeps its real names, as they come from the table schema.

## What changes are included in this PR?

When the consumer builds a higher-order function, it now resolves the
lambda parameters from the function's arguments
(`Expr::resolve_lambda_variables`), as the `TypeCoercion` analyzer
already does later. The parameter gets the list's element type, field
names included, before anything checks the body.

A function that uses a parameter of an outer lambda can't be resolved on
its own, because only the outer function knows that parameter's type.
For example, the inner call in `array_transform(s, x ->
array_transform([1, 2], i -> get_field(x, 'a') + i))` uses `x`, so it is
resolved together with the outer call.

## What is the testing strategy for this PR?

`roundtrip_lambda_parameter_struct_field_names` in
`roundtrip_logical_plan.rs` round trips a struct field access on:

- a parameter over a column
- a parameter over a computed list
- the inner parameter of nested lambdas
- an outer parameter read inside an inner lambda

Without the change, the test fails with the error above. The existing
lambda tests, including the dictionary and string view ones, still pass.

## Are there any user-facing changes?

Substrait plans that access struct fields of lambda parameters now load.
No API changes.

(cherry picked from commit 238ae32)
@codecov-commenter

codecov-commenter commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.61538% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.20%. Comparing base (9eda67e) to head (0f925a2).

Files with missing lines Patch % Lines
.../src/logical_plan/consumer/expr/scalar_function.rs 84.61% 0 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           branch-55     #198   +/-   ##
==========================================
  Coverage      81.20%   81.20%           
==========================================
  Files           1111     1111           
  Lines         387932   387956   +24     
  Branches      387932   387956   +24     
==========================================
+ Hits          315022   315045   +23     
+ Misses         54333    54332    -1     
- Partials       18577    18579    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@LiaCastaneda
LiaCastaneda merged commit 4d7f444 into branch-55 Nov 5, 2026
65 of 67 checks passed
@LiaCastaneda
LiaCastaneda deleted the lia.castaneda/cherry-pick/apache-pr-27086-20261105 branch November 5, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants