Skip to content

Commit 238ae32

Browse files
authored
fix: keep struct field names on Substrait lambda parameters (#27086)
## Which issue does this PR close? - Closes #25017. ## Rationale for this change A field access on a lambda parameter over a list of structs fails once the plan goes through Substrait: ```sql CREATE TABLE t AS SELECT [named_struct('a', 1, 'b', 'x')] AS s; SELECT array_transform(s, x -> get_field(x, 'b')) FROM t; ``` The query runs fine on its own, but converting its plan to Substrait and consuming it back fails: ``` Error during planning: Field b not found in struct ``` This happens because Substrait types carry no struct field names; only relation schemas do. The plan declares the lambda parameter with a bare struct type, so the consumer names its fields `c0, c1`, and `get_field(x, 'b')` looks for a field that doesn't exist. The list `s` itself keeps its real names, as they come from the table schema. ## What changes are included in this PR? When the consumer builds a higher-order function, it now resolves the lambda parameters from the function's arguments (`Expr::resolve_lambda_variables`), as the `TypeCoercion` analyzer already does later. The parameter gets the list's element type, field names included, before anything checks the body. A function that uses a parameter of an outer lambda can't be resolved on its own, because only the outer function knows that parameter's type. For example, the inner call in `array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i))` uses `x`, so it is resolved together with the outer call. ## What is the testing strategy for this PR? `roundtrip_lambda_parameter_struct_field_names` in `roundtrip_logical_plan.rs` round trips a struct field access on: - a parameter over a column - a parameter over a computed list - the inner parameter of nested lambdas - an outer parameter read inside an inner lambda Without the change, the test fails with the error above. The existing lambda tests, including the dictionary and string view ones, still pass. ## Are there any user-facing changes? Substrait plans that access struct fields of lambda parameters now load. No API changes.
1 parent c120619 commit 238ae32

2 files changed

Lines changed: 103 additions & 2 deletions

File tree

‎datafusion/substrait/src/logical_plan/consumer/expr/scalar_function.rs‎

Lines changed: 58 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
use crate::logical_plan::consumer::{SubstraitConsumer, from_substrait_func_args};
66
use datafusion::common::Result;
7+
use datafusion::common::tree_node::{TreeNode, TreeNodeRecursion};
78
use datafusion::common::{
89
DFSchema, DataFusionError, ScalarValue, not_impl_err, plan_err, substrait_err,
910
};
@@ -56,10 +57,24 @@ pub async fn from_scalar_function(
5657
// try to first match the requested function into registered higher-order functions, then udfs, built-in ops
5758
// and finally built-in expressions
5859
if let Ok(func) = higher_order_func {
59-
Ok(Expr::HigherOrderFunction(expr::HigherOrderFunction::new(
60+
let expr = Expr::HigherOrderFunction(expr::HigherOrderFunction::new(
6061
func.to_owned(),
6162
args,
62-
)))
63+
));
64+
// Substrait types carry no struct field names, so a lambda parameter over a list
65+
// of structs is declared as `{c0, c1, ...}`, and a field access in the body such
66+
// as `get_field(x, 'b')` would not find `b`. Give the parameters the fields the
67+
// function derives from its arguments instead, which keep the list's field names.
68+
//
69+
// A function that uses a parameter of an outer lambda can't be resolved on its
70+
// own, as only the outer function knows that parameter's type. In
71+
// `array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i))`
72+
// the inner call uses `x`, so it is resolved when the outer call is.
73+
if uses_outer_lambda_parameters(&expr, &mut Vec::new())? {
74+
Ok(expr)
75+
} else {
76+
Ok(expr.resolve_lambda_variables(input_schema)?.data)
77+
}
6378
} else if let Ok(func) = udf_func {
6479
Ok(Expr::ScalarFunction(expr::ScalarFunction::new_udf(
6580
func.to_owned(),
@@ -81,6 +96,47 @@ pub async fn from_scalar_function(
8196
}
8297
}
8398

99+
/// Checks whether `expr` uses a lambda parameter that comes from outside of it.
100+
///
101+
/// In `array_transform(s, x -> array_transform([1, 2], i -> get_field(x, 'a') + i))`,
102+
/// the inner `array_transform([1, 2], i -> get_field(x, 'a') + i)` uses its own
103+
/// parameter `i`, but also `x`, which belongs to the outer lambda, so this returns true
104+
/// for it. For the whole expression it returns false: `x` and `i` both belong to
105+
/// lambdas inside it.
106+
///
107+
/// `own_params` collects the parameters of the lambdas we are inside of while walking
108+
/// `expr`.
109+
fn uses_outer_lambda_parameters(
110+
expr: &Expr,
111+
own_params: &mut Vec<String>,
112+
) -> Result<bool> {
113+
match expr {
114+
// a parameter that wasn't collected on the way down comes from outside `expr`
115+
Expr::LambdaVariable(var) => Ok(!own_params.contains(&var.name)),
116+
Expr::Lambda(lambda) => {
117+
// a lambda's parameters can be used in its body, and only there
118+
let before = own_params.len();
119+
own_params.extend(lambda.params.iter().cloned());
120+
let uses = uses_outer_lambda_parameters(&lambda.body, own_params);
121+
own_params.truncate(before);
122+
uses
123+
}
124+
// anything else: check its children, stopping at the first one that uses one
125+
_ => {
126+
let mut uses = false;
127+
expr.apply_children(|child| {
128+
uses = uses_outer_lambda_parameters(child, own_params)?;
129+
Ok(if uses {
130+
TreeNodeRecursion::Stop
131+
} else {
132+
TreeNodeRecursion::Continue
133+
})
134+
})?;
135+
Ok(uses)
136+
}
137+
}
138+
}
139+
84140
pub fn substrait_fun_name(name: &str) -> &str {
85141
(match name.rsplit_once(':') {
86142
// Since 0.32.0, Substrait requires the function names to be in a compound format

‎datafusion/substrait/tests/cases/roundtrip_logical_plan.rs‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2653,6 +2653,51 @@ async fn roundtrip_array_transform_higher_order_function() -> Result<()> {
26532653
Ok(())
26542654
}
26552655

2656+
// Substrait lambda parameter types carry no struct field names, so a parameter over a
2657+
// list of structs must take its field names from the list, not from its declared type.
2658+
#[tokio::test]
2659+
async fn roundtrip_lambda_parameter_struct_field_names() -> Result<()> {
2660+
let ctx = higher_order_function_ctx().await?;
2661+
ctx.sql(
2662+
"CREATE TABLE structs AS SELECT \
2663+
[named_struct('a', 1, 'b', 'x'), named_struct('a', 2, 'b', 'y')] AS s, \
2664+
[[named_struct('a', 3, 'b', 'z')]] AS nested",
2665+
)
2666+
.await?
2667+
.collect()
2668+
.await?;
2669+
2670+
// field of a parameter over a column
2671+
roundtrip_with_ctx(
2672+
"SELECT array_transform(s, p0 -> get_field(p0, 'b')) FROM structs",
2673+
ctx.clone(),
2674+
)
2675+
.await?;
2676+
2677+
// field of a parameter over a computed list
2678+
roundtrip_with_ctx(
2679+
"SELECT array_transform(array_concat(s, s), p0 -> get_field(p0, 'a')) FROM structs",
2680+
ctx.clone(),
2681+
)
2682+
.await?;
2683+
2684+
// field of the inner parameter, where the inner list is the outer parameter
2685+
roundtrip_with_ctx(
2686+
"SELECT array_transform(nested, p0 -> array_transform(p0, p1 -> get_field(p1, 'b'))) FROM structs",
2687+
ctx.clone(),
2688+
)
2689+
.await?;
2690+
2691+
// field of the outer parameter, read inside an inner lambda
2692+
roundtrip_with_ctx(
2693+
"SELECT array_transform(s, p0 -> array_transform([1, 2], p1 -> get_field(p0, 'a') + p1)) FROM structs",
2694+
ctx.clone(),
2695+
)
2696+
.await?;
2697+
2698+
Ok(())
2699+
}
2700+
26562701
pub(crate) async fn higher_order_function_ctx() -> Result<SessionContext> {
26572702
let ctx = create_context_with_dialect(Some(Dialect::Databricks)).await?;
26582703

0 commit comments

Comments
 (0)