Skip to content

Commit 5e55eac

Browse files
committed
fix: evaluate struct-returning UDFs once across repeated field accesses
## Which issue does this PR close? Closes apache#23655. ## Rationale for this change `SELECT f(c)['a'], f(c)['b']`, where `f` is a struct-returning UDF, evaluates `f(c)` twice in the physical `ProjectionExec` — once per `get_field` — although the two calls are identical. `CommonSubexprEliminate` already hoists the shared `f(c)` into a `__common_expr_N` column so it runs once. `PushDownLeafProjections` then undoes that, one pass at a time: - After CSE, each field access reads its base from the hoist column, so `get_field`'s placement flips to `MoveTowardsLeafNodes` and the pass extracts it. - Merging the extraction into the hoist projection inlines that column's definition back into every access (via `build_projection_replace_map`), re-duplicating the `KeepInPlace` UDF. - `optimize_projections` drops the now-unused hoist column, and the two passes oscillate pass after pass — settling on a double evaluation. This is the instability the `is_ignored` note in `common_subexpr_eliminate` already warns about. ## What changes are included in this PR? `split_and_push_projection` now bails out — leaving the projection untouched — when merging the extractions into the input projection would inline a `KeepInPlace` definition that more than one extraction pair references. This mirrors the existing `optimize_projections` merge guard (apache#8296): a compute-once expression referenced more than once is kept in place instead of duplicated. With the guard, CSE's hoist survives, `optimize_projections` keeps it, and the pipeline converges to a single evaluation: Before: ```text Projection: get_field(f(c), 'a'), get_field(f(c), 'b') -- f(c) twice TableScan ``` After: ```text Projection: get_field(__common_expr_1, 'a'), get_field(__common_expr_1, 'b') Projection: f(c) AS __common_expr_1 -- f(c) once TableScan ``` ## Are these changes tested? Yes. - `test_struct_returning_udf_evaluated_once` (in `extract_leaf_expressions.rs`) runs the four interacting rules — CSE → extraction → pushdown → projection pruning — and asserts no field access wraps a fresh UDF call, plus an `insta` snapshot of the single-eval plan. It runs with the default pass budget, so any residual oscillation would surface as a double evaluation. - `get_field_like` — a small test UDF added to `test/udfs.rs` — reproduces `GetFieldFunc`'s conditional placement (`MoveTowardsLeafNodes` only when the base is pushable and the keys are literals). This lets the optimizer crate model struct-field access without depending on `datafusion-functions`, which it intentionally does not. `cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D warnings` are clean. ## Are there any user-facing changes? No API changes. Affected plans evaluate a struct-returning UDF once instead of once per field access; query results are unchanged. The saving scales with the number of distinct field accesses on the same UDF. ## Follow-ups - An end-to-end `.slt` test would need a struct-returning UDF registered in the sqllogictest harness: the built-in `named_struct` / `struct` constructors are folded into their fields by the field-over-struct-literal rewrite before this interaction can occur, so only an opaque (UDF) struct source reproduces it. Covered at the optimizer layer here instead.
1 parent 67947b6 commit 5e55eac

2 files changed

Lines changed: 173 additions & 3 deletions

File tree

‎datafusion/optimizer/src/extract_leaf_expressions.rs‎

Lines changed: 108 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -757,6 +757,51 @@ fn try_push_input(
757757
split_and_push_projection(proj, alias_generator)
758758
}
759759

760+
/// Returns `true` if merging the extracted `pairs` into `child` would inline a
761+
/// `KeepInPlace` definition into more than one extracted expression — e.g. a
762+
/// struct-returning UDF `f(c)` behind both `f(c)['a']` and `f(c)['b']`.
763+
///
764+
/// `CommonSubexprEliminate` hoists such an expression into a shared column so it
765+
/// runs once; [`build_extraction_projection_impl`] then resolves that column
766+
/// back to its definition and re-duplicates it. Mirrors the
767+
/// `optimize_projections` merge guard (#8296): a compute-once expression used
768+
/// more than once stays in place.
769+
fn merge_would_duplicate_kept_expr(pairs: &[(Expr, String)], child: &Projection) -> bool {
770+
// Columns whose `KeepInPlace` definition must stay put; inlining one would
771+
// duplicate a compute-once expression. Plain and pushable definitions are
772+
// cheap to duplicate.
773+
let kept_columns: std::collections::HashSet<String> = child
774+
.schema
775+
.iter()
776+
.zip(child.expr.iter())
777+
.filter_map(|((qualifier, field), expr)| {
778+
if expr.placement().should_push_to_leaves() {
779+
None
780+
} else {
781+
Some(Column::from((qualifier, field)).flat_name())
782+
}
783+
})
784+
.collect();
785+
if kept_columns.is_empty() {
786+
return false;
787+
}
788+
789+
// A pair that references the same kept column twice still inlines one copy,
790+
// so count each pair at most once per column.
791+
let mut usage: HashMap<String, usize> = HashMap::new();
792+
for (expr, _alias) in pairs {
793+
let mut seen: std::collections::HashSet<String> =
794+
std::collections::HashSet::new();
795+
for col in expr.column_refs() {
796+
let name = col.flat_name();
797+
if kept_columns.contains(&name) && seen.insert(name.clone()) {
798+
*usage.entry(name).or_insert(0) += 1;
799+
}
800+
}
801+
}
802+
usage.values().any(|&count| count > 1)
803+
}
804+
760805
/// Splits a projection into extractable pieces, pushes them towards leaf
761806
/// nodes, and adds a recovery projection if needed.
762807
///
@@ -899,6 +944,14 @@ fn split_and_push_projection(
899944
return Ok(None);
900945
}
901946

947+
// Merging here would re-inline an expression `CommonSubexprEliminate` hoisted
948+
// to run once, undoing it (issue #23655); leave the projection in place.
949+
if let LogicalPlan::Projection(child) = input.as_ref()
950+
&& merge_would_duplicate_kept_expr(&extraction_pairs, child)
951+
{
952+
return Ok(None);
953+
}
954+
902955
// ── Phase 2: Push down ──────────────────────────────────────────────
903956
let proj_input = Arc::clone(&proj.input);
904957
let pushed = push_extraction_pairs(
@@ -1246,8 +1299,9 @@ fn try_push_into_inputs(
12461299
mod tests {
12471300

12481301
use super::*;
1302+
use crate::common_subexpr_eliminate::CommonSubexprEliminate;
12491303
use crate::optimize_projections::OptimizeProjections;
1250-
use crate::test::udfs::PlacementTestUDF;
1304+
use crate::test::udfs::{PlacementTestUDF, get_field_like};
12511305
use crate::test::*;
12521306
use crate::{Optimizer, OptimizerContext};
12531307
use datafusion_expr::expr::ScalarFunction;
@@ -3163,4 +3217,57 @@ mod tests {
31633217

31643218
Ok(())
31653219
}
3220+
3221+
/// Regression test for issue #23655: `f(c)['a'], f(c)['b']` on a
3222+
/// struct-returning UDF must evaluate `f(c)` once, not once per field access.
3223+
///
3224+
/// `keep_in_place_udf` / `get_field_like` model `f(c)` and `f(c)[key]` — the
3225+
/// crate can't depend on the real `get_field` (see [`get_field_like`]).
3226+
/// Without the guard, `PushDownLeafProjections` re-inlines the shared UDF that
3227+
/// `CommonSubexprEliminate` hoisted and the passes oscillate to a double
3228+
/// evaluation.
3229+
#[test]
3230+
fn test_struct_returning_udf_evaluated_once() -> Result<()> {
3231+
let udf = ScalarUDF::new_from_impl(
3232+
PlacementTestUDF::new().with_placement(ExpressionPlacement::KeepInPlace),
3233+
);
3234+
let f_c = udf.call(vec![col("c")]);
3235+
let plan = LogicalPlanBuilder::from(test_table_scan()?)
3236+
.project(vec![
3237+
get_field_like(f_c.clone(), "a"),
3238+
get_field_like(f_c, "b"),
3239+
])?
3240+
.build()?;
3241+
3242+
// The full four-rule interaction, default pass budget so any oscillation
3243+
// would surface here as a double evaluation.
3244+
let ctx = OptimizerContext::new();
3245+
let optimizer = Optimizer::with_rules(vec![
3246+
Arc::new(CommonSubexprEliminate::new()),
3247+
Arc::new(ExtractLeafExpressions::new()),
3248+
Arc::new(PushDownLeafProjections::new()),
3249+
Arc::new(OptimizeProjections::new()),
3250+
]);
3251+
let optimized = optimizer.optimize(plan, &ctx, |_, _| {})?;
3252+
3253+
// Every field access reads the UDF back through `__common_expr_1`; none
3254+
// wraps a fresh `keep_in_place_udf(...)` call. Asserted explicitly so a
3255+
// careless snapshot re-accept can't quietly restore the double evaluation.
3256+
let formatted = format!("{optimized}");
3257+
assert_eq!(
3258+
formatted
3259+
.matches("get_field_like(keep_in_place_udf")
3260+
.count(),
3261+
0,
3262+
"struct-returning UDF must not be re-evaluated inside a field access; got:\n{formatted}"
3263+
);
3264+
3265+
insta::assert_snapshot!(formatted, @r#"
3266+
Projection: get_field_like(__common_expr_1 AS keep_in_place_udf(test.c), Utf8("a")), get_field_like(__common_expr_1 AS keep_in_place_udf(test.c), Utf8("b"))
3267+
Projection: keep_in_place_udf(test.c) AS __common_expr_1
3268+
TableScan: test projection=[c]
3269+
"#);
3270+
3271+
Ok(())
3272+
}
31663273
}

‎datafusion/optimizer/src/test/udfs.rs‎

Lines changed: 65 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ use arrow::datatypes::DataType;
1919
use datafusion_common::Result;
2020
use datafusion_expr::{
2121
ColumnarValue, Expr, ExpressionPlacement, ScalarFunctionArgs, ScalarUDF,
22-
ScalarUDFImpl, Signature, TypeSignature,
22+
ScalarUDFImpl, Signature, TypeSignature, Volatility, lit,
2323
};
2424

2525
/// A configurable test UDF for optimizer tests.
@@ -44,7 +44,7 @@ impl PlacementTestUDF {
4444
// The actual types don't matter since this UDF is not intended for execution.
4545
signature: Signature::new(
4646
TypeSignature::OneOf(vec![TypeSignature::Any(1), TypeSignature::Any(2)]),
47-
datafusion_expr::Volatility::Immutable,
47+
Volatility::Immutable,
4848
),
4949
placement: ExpressionPlacement::MoveTowardsLeafNodes,
5050
id: 0,
@@ -96,3 +96,66 @@ pub fn leaf_udf_expr(arg: Expr) -> Expr {
9696
);
9797
udf.call(vec![arg])
9898
}
99+
100+
/// A `get_field`-like test UDF for modeling struct-field access (`f(c)['a']`),
101+
/// so tests need not depend on `datafusion-functions` (which this crate avoids —
102+
/// see its `Cargo.toml`).
103+
///
104+
/// Placement mirrors `GetFieldFunc`: `MoveTowardsLeafNodes` when the base is
105+
/// pushable and the keys are literals, else `KeepInPlace`. That flip drives the
106+
/// issue #23655 pushdown interaction: the access is `KeepInPlace` until
107+
/// `CommonSubexprEliminate` hoists the UDF into a column, then becomes pushable.
108+
#[derive(Debug, PartialEq, Eq, Hash)]
109+
pub struct GetFieldLikeUDF {
110+
signature: Signature,
111+
}
112+
113+
impl Default for GetFieldLikeUDF {
114+
fn default() -> Self {
115+
Self::new()
116+
}
117+
}
118+
119+
impl GetFieldLikeUDF {
120+
pub fn new() -> Self {
121+
Self {
122+
signature: Signature::new(TypeSignature::Any(2), Volatility::Immutable),
123+
}
124+
}
125+
}
126+
127+
impl ScalarUDFImpl for GetFieldLikeUDF {
128+
fn name(&self) -> &str {
129+
"get_field_like"
130+
}
131+
fn signature(&self) -> &Signature {
132+
&self.signature
133+
}
134+
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
135+
Ok(DataType::UInt32)
136+
}
137+
fn invoke_with_args(&self, _args: ScalarFunctionArgs) -> Result<ColumnarValue> {
138+
panic!("GetFieldLikeUDF: not intended for execution")
139+
}
140+
fn placement(&self, args: &[ExpressionPlacement]) -> ExpressionPlacement {
141+
// Mirror `GetFieldFunc`: cheap to push down only when the base is a
142+
// column (or already pushable) and every field key is a literal.
143+
match args.first() {
144+
Some(
145+
ExpressionPlacement::Column | ExpressionPlacement::MoveTowardsLeafNodes,
146+
) if args[1..]
147+
.iter()
148+
.all(|p| matches!(p, ExpressionPlacement::Literal)) =>
149+
{
150+
ExpressionPlacement::MoveTowardsLeafNodes
151+
}
152+
_ => ExpressionPlacement::KeepInPlace,
153+
}
154+
}
155+
}
156+
157+
/// Create a `get_field_like(base, key)` expression that models `base[key]`.
158+
pub fn get_field_like(base: Expr, key: &str) -> Expr {
159+
let udf = ScalarUDF::new_from_impl(GetFieldLikeUDF::new());
160+
udf.call(vec![base, lit(key)])
161+
}

0 commit comments

Comments
 (0)