Repository navigation
Commit 238ae32
authored
fix: keep struct field names on Substrait lambda parameters (apache#27086)
## Which issue does this PR close?
- Closes apache#25017.
## Rationale for this change
A field access on a lambda parameter over a list of structs fails once
the plan goes through Substrait:
```sql
CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s;
SELECT array_transform(s, x -> get_field(x, 'b')) FROM t;
```
The query runs fine on its own, but converting its plan to Substrait and
consuming it back fails:
```
Error during planning: Field b not found in struct
```
This happens because Substrait types carry no struct field names; only
relation schemas do. The plan declares the lambda parameter with a bare
struct type, so the consumer names its fields `c0, c1`, and
`get_field(x, 'b')` looks for a field that doesn't exist. The list `s`
itself keeps its real names, as they come from the table schema.
## What changes are included in this PR?
When the consumer builds a higher-order function, it now resolves the
lambda parameters from the function's arguments
(`Expr::resolve_lambda_variables`), as the `TypeCoercion` analyzer
already does later. The parameter gets the list's element type, field
names included, before anything checks the body.
A function that uses a parameter of an outer lambda can't be resolved on
its own, because only the outer function knows that parameter's type.
For example, the inner call in `array_transform(s, x ->
array_transform([1, 2], i -> get_field(x, 'a') + i))` uses `x`, so it is
resolved together with the outer call.
## What is the testing strategy for this PR?
`roundtrip_lambda_parameter_struct_field_names` in
`roundtrip_logical_plan.rs` round trips a struct field access on:
- a parameter over a column
- a parameter over a computed list
- the inner parameter of nested lambdas
- an outer parameter read inside an inner lambda
Without the change, the test fails with the error above. The existing
lambda tests, including the dictionary and string view ones, still pass.
## Are there any user-facing changes?
Substrait plans that access struct fields of lambda parameters now load.
No API changes.1 parent c120619 commit 238ae32
2 files changed
Lines changed: 103 additions & 2 deletions
File tree
- datafusion/substrait
- src/logical_plan/consumer/expr
- tests/cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
59 | | - | |
| 60 | + | |
60 | 61 | | |
61 | 62 | | |
62 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
63 | 78 | | |
64 | 79 | | |
65 | 80 | | |
| |||
81 | 96 | | |
82 | 97 | | |
83 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
84 | 140 | | |
85 | 141 | | |
86 | 142 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2653 | 2653 | | |
2654 | 2654 | | |
2655 | 2655 | | |
| 2656 | + | |
| 2657 | + | |
| 2658 | + | |
| 2659 | + | |
| 2660 | + | |
| 2661 | + | |
| 2662 | + | |
| 2663 | + | |
| 2664 | + | |
| 2665 | + | |
| 2666 | + | |
| 2667 | + | |
| 2668 | + | |
| 2669 | + | |
| 2670 | + | |
| 2671 | + | |
| 2672 | + | |
| 2673 | + | |
| 2674 | + | |
| 2675 | + | |
| 2676 | + | |
| 2677 | + | |
| 2678 | + | |
| 2679 | + | |
| 2680 | + | |
| 2681 | + | |
| 2682 | + | |
| 2683 | + | |
| 2684 | + | |
| 2685 | + | |
| 2686 | + | |
| 2687 | + | |
| 2688 | + | |
| 2689 | + | |
| 2690 | + | |
| 2691 | + | |
| 2692 | + | |
| 2693 | + | |
| 2694 | + | |
| 2695 | + | |
| 2696 | + | |
| 2697 | + | |
| 2698 | + | |
| 2699 | + | |
| 2700 | + | |
2656 | 2701 | | |
2657 | 2702 | | |
2658 | 2703 | | |
| |||
0 commit comments