Skip to content

Commit 7b24735

Browse files
authored
Add resolve_lambda_variables helper to Expr and LogicalPlan (apache#22101)
## Which issue does this PR close? Part of apache#21172 ## Rationale for this change This has been pulled out from main lambda PR to reduce it's size. Currently, the `LambdaVariable` field is an `Option<FieldRef>` but must be set manually for `expr_api` users (see below, passing `None` would result in planning/execution errors). This is similar to `Expr::Placeholder` field and its helper `Expr::infer_placeholder_types` https://github.com/apache/datafusion/blob/7f2f78d48b6d3d6aee2ce2fd29910bb4c11b1012/datafusion/functions-nested/src/array_transform.rs#L328-L341 Now it can be omitted and later resolved by the new helpers: https://github.com/apache/datafusion/blob/7c266c10e806cf87edfe5a2eccccccf639fdb2fe/datafusion/functions-nested/src/array_transform.rs#L326-L330 ## What changes are included in this PR? This PR adds the method `resolve_lambda_variables` to `Expr` and `LogicalPlan` to automatically resolve all variables of the given expr or logical plan. Nothing changes for sql planning ## Are these changes tested? There's a unit test which should cover all edge cases. Is not trivial to test via sqllogictests because sql planning produces already resolved lambda variables. ## Are there any user-facing changes? New methods added only
1 parent b426232 commit 7b24735

7 files changed

Lines changed: 569 additions & 45 deletions

File tree

‎datafusion/common/src/lib.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,7 @@ pub type HashMap<K, V, S = DefaultHashBuilder> = hashbrown::HashMap<K, V, S>;
115115
pub type HashSet<T, S = DefaultHashBuilder> = hashbrown::HashSet<T, S>;
116116
pub mod hash_map {
117117
pub use hashbrown::hash_map::Entry;
118+
pub use hashbrown::hash_map::EntryRef;
118119
}
119120
pub mod hash_set {
120121
pub use hashbrown::hash_set::Entry;

‎datafusion/expr/src/expr.rs‎

Lines changed: 37 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ use std::sync::Arc;
2626

2727
use crate::expr_fn::binary_expr;
2828
use crate::function::WindowFunctionSimplification;
29-
use crate::higher_order_function::HigherOrderUDF;
29+
use crate::higher_order_function::{HigherOrderUDF, resolve_lambda_variables};
3030
use crate::logical_plan::Subquery;
3131
use crate::type_coercion::functions::value_fields_with_higher_order_udf;
3232
use crate::{AggregateUDF, LambdaParametersProgress, ValueOrLambda, Volatility};
@@ -453,7 +453,12 @@ impl HigherOrderFunction {
453453
}
454454

455455
/// Invokes the inner function [`HigherOrderUDF::lambda_parameters`]
456-
/// using the arguments of this invocation
456+
/// using the arguments of this invocation. This expression lambda
457+
/// variables must be already resolved either by coming from the
458+
/// default sql planner or by calling [Expr::resolve_lambda_variables]
459+
/// or [LogicalPlan::resolve_lambda_variables]
460+
///
461+
/// [LogicalPlan::resolve_lambda_variables]: crate::LogicalPlan::resolve_lambda_variables
457462
pub fn lambda_parameters(
458463
&self,
459464
schema: &dyn ExprSchema,
@@ -496,9 +501,20 @@ impl PartialEq for HigherOrderFunction {
496501
}
497502

498503
/// A named reference to a lambda parameter which includes it's own [`FieldRef`],
499-
/// which is used to implement [`ExprSchemable`], for example. Note the field must
500-
/// be set in order to create a physical lambda variable. A helper to automatically
501-
/// set them will be added in the future
504+
/// which is used to implement [`ExprSchemable`], for example. It is an option only to make
505+
/// easier for `expr_api` users to construct lambda variables, but any expression
506+
/// tree or [`LogicalPlan`] containing unresolved variables must be resolved before
507+
/// usage with either [`Expr::resolve_lambda_variables`] or
508+
/// [`LogicalPlan::resolve_lambda_variables`]. The default SQL planner produces
509+
/// already resolved variables and no further resolving is required.
510+
///
511+
/// After resolving, if any argument from the lambda function which this
512+
/// variables originates from have it's field changed (type, nullability,
513+
/// metadata, etc), the resolved variable may became outdated and must be
514+
/// resolved again.
515+
///
516+
/// [`LogicalPlan`]: crate::LogicalPlan
517+
/// [`LogicalPlan::resolve_lambda_variables`]: crate::LogicalPlan::resolve_lambda_variables
502518
#[derive(Clone, PartialEq, PartialOrd, Eq, Debug, Hash)]
503519
pub struct LambdaVariable {
504520
pub name: String,
@@ -507,7 +523,12 @@ pub struct LambdaVariable {
507523
}
508524

509525
impl LambdaVariable {
510-
/// Create a lambda variable from a name and a Field.
526+
/// Create a lambda variable from a name and an optional field.
527+
/// If the field is none, the expression tree or LogicalPlan which
528+
/// owns this variable must be resolved before usage with either
529+
/// [`Expr::resolve_lambda_variables`] or [`LogicalPlan::resolve_lambda_variables`].
530+
///
531+
/// [`LogicalPlan::resolve_lambda_variables`]: crate::LogicalPlan::resolve_lambda_variables
511532
pub fn new(name: String, field: Option<FieldRef>) -> Self {
512533
Self {
513534
name,
@@ -2229,6 +2250,16 @@ impl Expr {
22292250
None
22302251
}
22312252
}
2253+
2254+
/// Return a `Expr` with all [`LambdaVariable`] resolved only if all of them
2255+
/// are contained in the subtree of the [`HigherOrderFunction`] it originates from,
2256+
/// otherwise returns an error
2257+
pub fn resolve_lambda_variables(
2258+
self,
2259+
schema: &DFSchema,
2260+
) -> Result<Transformed<Expr>> {
2261+
resolve_lambda_variables(self, schema, &mut HashMap::new())
2262+
}
22322263
}
22332264

22342265
impl Normalizeable for Expr {

‎datafusion/expr/src/expr_fn.rs‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -728,10 +728,14 @@ pub fn lambda(params: impl IntoIterator<Item = impl Into<String>>, body: Expr) -
728728
))
729729
}
730730

731-
/// Create a lambda variable expression
732-
// todo: make this pub when support for optional field lands
733-
#[expect(unused)]
734-
fn lambda_var(name: impl Into<String>) -> Expr {
731+
/// Create an unresolved lambda variable expression
732+
///
733+
/// The expression tree or [`LogicalPlan`] which
734+
/// owns this variable must be resolved before usage with either
735+
/// [`Expr::resolve_lambda_variables`] or [`LogicalPlan::resolve_lambda_variables`].
736+
///
737+
/// [LogicalPlan::resolve_lambda_variables]: crate::LogicalPlan::resolve_lambda_variables
738+
pub fn lambda_var(name: impl Into<String>) -> Expr {
735739
Expr::LambdaVariable(LambdaVariable::new(name.into(), None))
736740
}
737741

0 commit comments

Comments
 (0)