Skip to content

Commit 0ef844e

Browse files
authored
refactor: move lambda variable scope into Physical Planning Context (#23989)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #23700 . ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This is a clean up following #23649 ## What changes are included in this PR? - Moves the lambda-variable scope state out of `ExecutionProps` and into `PhysicalPlanningContext`, following the same pattern as the scalar-subquery state in #23649. - Puts `PhysicalPlanningContext::indexes` behind an `Arc` so the new per-lambda-body clone doesn't deep-copy the subquery index map. `PhysicalPlanningContext::new`'s signature is unchanged. ## Are these changes tested? - New `lambda_variables_shadow_outer_scope` unit test in `physical_planning_context.rs` covers `with_qualified_lambda_variables` / `lambda_variable_qualifier` directly, including an inner lambda shadowing an outer parameter name. ## Are there any user-facing changes? this is a breaking change for lib users.
1 parent c8f89cf commit 0ef844e

4 files changed

Lines changed: 90 additions & 45 deletions

File tree

‎datafusion/expr/src/execution_props.rs‎

Lines changed: 1 addition & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
use crate::var_provider::{VarProvider, VarType};
1919
use chrono::{DateTime, Utc};
2020
use datafusion_common::HashMap;
21-
use datafusion_common::TableReference;
2221
use datafusion_common::alias::AliasGenerator;
2322
use datafusion_common::config::ConfigOptions;
2423
use std::sync::Arc;
@@ -60,10 +59,6 @@ pub struct ExecutionProps {
6059
pub config_options: Option<Arc<ConfigOptions>>,
6160
/// Providers for scalar variables
6261
pub var_providers: Option<HashMap<VarType, Arc<dyn VarProvider + Send + Sync>>>,
63-
/// Maps each lambda variable name to its lambda qualifier generated
64-
/// during physical planning. Populated by the physical planner for
65-
/// each lambda before calling `create_physical_expr`.
66-
pub lambda_variable_qualifier: HashMap<String, TableReference>,
6762
}
6863

6964
impl Default for ExecutionProps {
@@ -80,7 +75,6 @@ impl ExecutionProps {
8075
alias_generator: Arc::new(AliasGenerator::new()),
8176
config_options: None,
8277
var_providers: None,
83-
lambda_variable_qualifier: HashMap::new(),
8478
}
8579
}
8680

@@ -139,22 +133,6 @@ impl ExecutionProps {
139133
pub fn config_options(&self) -> Option<&Arc<ConfigOptions>> {
140134
self.config_options.as_ref()
141135
}
142-
143-
/// Adds a mapping for each variable to the given qualifier. Existing
144-
/// variables with conflicting names get's shadowed
145-
pub fn with_qualified_lambda_variables(
146-
mut self,
147-
qualifier: &TableReference,
148-
variables: &[String],
149-
) -> Self {
150-
for var in variables {
151-
self.lambda_variable_qualifier
152-
.entry_ref(var)
153-
.insert(qualifier.clone());
154-
}
155-
156-
self
157-
}
158136
}
159137

160138
#[cfg(test)]
@@ -165,7 +143,7 @@ mod test {
165143
fn debug() {
166144
let props = ExecutionProps::new();
167145
assert_eq!(
168-
"ExecutionProps { query_execution_start_time: None, alias_generator: AliasGenerator { next_id: 1 }, config_options: None, var_providers: None, lambda_variable_qualifier: {} }",
146+
"ExecutionProps { query_execution_start_time: None, alias_generator: AliasGenerator { next_id: 1 }, config_options: None, var_providers: None }",
169147
format!("{props:?}")
170148
);
171149
}

‎datafusion/expr/src/physical_planning_context.rs‎

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,32 +19,42 @@ use std::fmt;
1919
use std::hash::{Hash, Hasher};
2020
use std::sync::{Arc, Mutex};
2121

22-
use datafusion_common::{HashMap, Result, ScalarValue, internal_err};
22+
use datafusion_common::{HashMap, Result, ScalarValue, TableReference, internal_err};
2323

2424
/// Context used while converting a logical plan subtree into a physical plan.
2525
///
2626
/// Unlike [`ExecutionProps`](crate::execution_props::ExecutionProps), which
2727
/// applies to the overall planning and execution of a query, this context can
28-
/// differ between recursively planned subtrees. It currently carries the state
29-
/// needed to create physical expressions for [`Expr::ScalarSubquery`] nodes
30-
/// that read from a shared
31-
/// [`ScalarSubqueryResults`] container.
28+
/// differ between recursively planned subtrees. It currently carries:
29+
///
30+
/// * the state needed to create physical expressions for
31+
/// [`Expr::ScalarSubquery`] nodes that read from a shared
32+
/// [`ScalarSubqueryResults`] container, and
33+
/// * the qualifiers assigned to the [`Expr::LambdaVariable`]s that are in scope.
3234
///
3335
/// The physical planner builds this context from the set of uncorrelated scalar
3436
/// subqueries it has scheduled for a subtree. It is then passed explicitly
3537
/// through `create_physical_expr` so that function can find the slot index for
36-
/// each [`Subquery`].
38+
/// each [`Subquery`]. While planning the body of a lambda,
39+
/// `create_physical_expr` extends the context with the lambda's parameters via
40+
/// [`Self::with_qualified_lambda_variables`].
3741
///
3842
/// An empty [`PhysicalPlanningContext`] (the [`Default`]) is what every
3943
/// non-physical-planner caller passes; if such a caller encounters a scalar
4044
/// subquery, `create_physical_expr` returns a `not_impl_err`.
4145
///
4246
/// [`Expr::ScalarSubquery`]: crate::Expr::ScalarSubquery
47+
/// [`Expr::LambdaVariable`]: crate::Expr::LambdaVariable
4348
/// [`Subquery`]: crate::logical_plan::Subquery
4449
#[derive(Clone, Debug, Default)]
4550
pub struct PhysicalPlanningContext {
46-
indexes: HashMap<crate::logical_plan::Subquery, SubqueryIndex>,
51+
/// Behind an `Arc` because the context is cloned for each lambda body that
52+
/// is planned, and the indexes are the same for the whole subtree.
53+
indexes: Arc<HashMap<crate::logical_plan::Subquery, SubqueryIndex>>,
4754
results: ScalarSubqueryResults,
55+
/// Maps each lambda variable name in scope to the qualifier generated for
56+
/// its lambda during physical planning.
57+
lambda_variable_qualifier: HashMap<String, TableReference>,
4858
}
4959

5060
impl PhysicalPlanningContext {
@@ -55,7 +65,11 @@ impl PhysicalPlanningContext {
5565
indexes: HashMap<crate::logical_plan::Subquery, SubqueryIndex>,
5666
results: ScalarSubqueryResults,
5767
) -> Self {
58-
Self { indexes, results }
68+
Self {
69+
indexes: Arc::new(indexes),
70+
results,
71+
lambda_variable_qualifier: HashMap::new(),
72+
}
5973
}
6074

6175
/// Returns the slot index assigned to `subquery`, if any.
@@ -70,6 +84,27 @@ impl PhysicalPlanningContext {
7084
pub fn results(&self) -> &ScalarSubqueryResults {
7185
&self.results
7286
}
87+
88+
/// Adds a mapping for each variable to the given qualifier. Existing
89+
/// variables with conflicting names are shadowed.
90+
pub fn with_qualified_lambda_variables(
91+
mut self,
92+
qualifier: &TableReference,
93+
variables: &[String],
94+
) -> Self {
95+
for var in variables {
96+
self.lambda_variable_qualifier
97+
.entry_ref(var)
98+
.insert(qualifier.clone());
99+
}
100+
101+
self
102+
}
103+
104+
/// Returns the qualifier of the lambda variable `name`, if it is in scope.
105+
pub fn lambda_variable_qualifier(&self, name: &str) -> Option<&TableReference> {
106+
self.lambda_variable_qualifier.get(name)
107+
}
73108
}
74109

75110
/// Index of a scalar subquery within a [`ScalarSubqueryResults`] container.
@@ -192,6 +227,20 @@ mod tests {
192227
Ok(())
193228
}
194229

230+
#[test]
231+
fn lambda_variables_shadow_outer_scope() {
232+
let outer = TableReference::bare("lambda_1");
233+
let inner = TableReference::bare("lambda_2");
234+
235+
let ctx = PhysicalPlanningContext::default()
236+
.with_qualified_lambda_variables(&outer, &["x".to_string(), "y".to_string()])
237+
.with_qualified_lambda_variables(&inner, &["y".to_string()]);
238+
239+
assert_eq!(ctx.lambda_variable_qualifier("x"), Some(&outer));
240+
assert_eq!(ctx.lambda_variable_qualifier("y"), Some(&inner));
241+
assert_eq!(ctx.lambda_variable_qualifier("z"), None);
242+
}
243+
195244
#[test]
196245
fn scalar_subquery_results_clear() -> Result<()> {
197246
let results = ScalarSubqueryResults::new(1);

‎datafusion/physical-expr/src/planner.rs‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -121,9 +121,11 @@ use datafusion_expr::{
121121
/// to qualified or unqualified fields by name.
122122
/// * `execution_props` - Per-execution properties such as the query start time.
123123
/// * `planning_ctx` - The [`PhysicalPlanningContext`] used to resolve
124-
/// `Expr::ScalarSubquery` nodes. The physical planner threads the subquery
125-
/// index map and shared results container from its `ScalarSubqueryExec`
126-
/// construction into calls to `create_physical_expr`. Callers creating
124+
/// `Expr::ScalarSubquery` and `Expr::LambdaVariable` nodes. The physical
125+
/// planner threads the subquery index map and shared results container from
126+
/// its `ScalarSubqueryExec` construction into calls to
127+
/// `create_physical_expr`; the lambda variable qualifiers are added by this
128+
/// function itself as it descends into lambda bodies. Callers creating
127129
/// physical expressions outside of physical planning should pass
128130
/// `&PhysicalPlanningContext::default()`; converting a scalar subquery then returns a
129131
/// planning error.
@@ -612,15 +614,15 @@ pub fn create_physical_expr(
612614
input_dfschema.metadata().clone(),
613615
)?;
614616

615-
let execution_props = execution_props
617+
let planning_ctx = planning_ctx
616618
.clone()
617619
.with_qualified_lambda_variables(&qualifier, &lambda.params);
618620

619621
create_physical_expr(
620622
arg,
621623
&lambda_schema,
622-
&execution_props,
623-
planning_ctx,
624+
execution_props,
625+
&planning_ctx,
624626
)
625627
}
626628
_ => create_physical_expr(
@@ -657,12 +659,14 @@ pub fn create_physical_expr(
657659
plan_datafusion_err!("unresolved LambdaVariable {name}")
658660
})?;
659661

660-
let qualifier = execution_props
661-
.lambda_variable_qualifier
662-
.get(name)
663-
.ok_or_else(|| {
664-
plan_datafusion_err!("qualifier for lambda variable {name} not found")
665-
})?;
662+
let qualifier =
663+
planning_ctx
664+
.lambda_variable_qualifier(name)
665+
.ok_or_else(|| {
666+
plan_datafusion_err!(
667+
"qualifier for lambda variable {name} not found"
668+
)
669+
})?;
666670

667671
let index = input_dfschema
668672
.index_of_column_by_name(Some(qualifier), name)

‎docs/source/library-user-guide/upgrading/55.0.0.md‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -740,14 +740,19 @@ unit is truncated -- so `time(s) + interval '1 nanosecond'` is a no-op.
740740

741741
See [PR #23279](https://github.com/apache/datafusion/pull/23279) for details.
742742

743-
### Scalar-subquery state moved to an explicit `PhysicalPlanningContext`
743+
### Physical-planning state moved to an explicit `PhysicalPlanningContext`
744744

745745
The `subquery_indexes` and `subquery_results` public fields on
746746
`datafusion_expr::execution_props::ExecutionProps` have been removed. They were
747747
added in `54.0.0` as the channel through which the physical planner passed
748748
uncorrelated scalar-subquery state to functions that create physical
749749
`Arc<dyn PhysicalExpr>` values from logical `Expr` values.
750750

751+
The `lambda_variable_qualifier` public field and the
752+
`with_qualified_lambda_variables` method on `ExecutionProps` have been removed
753+
for the same reason: they carried the qualifiers of the lambda variables in
754+
scope while `create_physical_expr` descended into a lambda body.
755+
751756
That state is now carried by a dedicated
752757
`datafusion_expr::physical_planning_context::PhysicalPlanningContext` passed explicitly
753758
through functions and planner traits. Unlike `ExecutionProps`, which applies
@@ -790,6 +795,14 @@ Convenience methods such as `SessionContext::create_physical_expr` and
790795
parameter and forward it.
791796
- Code that read or wrote `execution_props.subquery_indexes` /
792797
`execution_props.subquery_results`: build a `PhysicalPlanningContext` instead.
798+
- Code that read `execution_props.lambda_variable_qualifier` or called
799+
`ExecutionProps::with_qualified_lambda_variables`: remove that usage. Callers
800+
that only plan a `HigherOrderFunction` are not affected --
801+
`create_physical_expr` populates the lambda qualifiers itself as it descends
802+
into lambda bodies. Code that needs to read or extend the lambda
803+
scope should use the equivalents on `PhysicalPlanningContext`:
804+
`PhysicalPlanningContext::lambda_variable_qualifier` and
805+
`PhysicalPlanningContext::with_qualified_lambda_variables`.
793806

794807
**Migration guide:**
795808

@@ -833,7 +846,8 @@ async fn plan_extension(
833846
}
834847
```
835848

836-
See [PR #23649](https://github.com/apache/datafusion/pull/23649) for details.
849+
See [PR #23649](https://github.com/apache/datafusion/pull/23649) and
850+
[PR #23989](https://github.com/apache/datafusion/pull/23989) for details.
837851

838852
### Catalog, planner, and optimizer contracts moved to `datafusion-session`
839853

0 commit comments

Comments
 (0)