Skip to content

Commit 1e77af8

Browse files
authored
fix: don't duplicate volatile expressions when pushing projection into file scan (apache#23395)
## Which issue does this PR close? - Closes apache#23220. ## Rationale for this change The reported bug showed that when volatile (non-deterministic) expressions are referenced multiple times in an outer function the query results can be incorrect. To produce correct results, a volatile expression (e.g. `random()`, `uuid()`) aliased once in a subquery and referenced multiple times must be evaluated once and reused. Since 52.0.0 this evaluation pattern for volatile functions has been broken; the physical projection-pushdown rule merges the outer projection into the file `DataSourceExec`, inlining the aliased volatile expression at each reference site. Instead of being evaluated once, the volatile expression is evaluated N times, so the references diverge: ```sql SELECT s.r AS x, s.r AS y FROM (SELECT random() AS r FROM t) AS s; -- x != y on >= 52.0.0 ``` This was correct in 51.0.0 and regressed in 52.0.0/53.0.0. It reproduces on file scans (Parquet/CSV) but not in-memory tables, and was surfaced downstream in Ibis. Worth noting, apache#10337 appears to report the same class of bug but a different cause (I'll try to look into that issue soon as well). ## What changes are included in this PR? `FileScanConfig::try_swapping_with_projection` now declines to merge a projection into the file source when the merge would inline a volatile expression that the incoming projection references. The check reuses the existing `is_volatile()` utility from `datafusion_physical_expr_common` — the same volatility gate the physical `ProjectionPushdown` and `FilterPushdown` rules already apply — so file-source projection merging is now consistent with them. Deterministic expressions still merge freely. The guard blocks when a volatile inner expression is referenced at least once (not only more than once), because a single outer expression can itself duplicate the value (e.g.`r + r`). ## Are these changes tested? Yes: - Unit tests in `file_scan_config.rs`: - volatile referenced ≥1× → blocked; - deterministic computed and column-only → allowed; - unreferenced volatile → allowed; - single-expression self-reference (`r + r`) → blocked; - volatile nested in arithmetic → blocked; - empty → allowed. - A regression test in `projection_pushdown.slt` asserting `x = y` for the aliased-`random()` pattern over a Parquet scan. - Full `datafusion-sqllogictest` suite passes. ## Are there any user-facing changes? No API changes! Query results for the affected pattern are corrected (a volatile expression aliased in a subquery is no longer duplicated by projection pushdown into a file scan). Physical plans for such queries now retain a `ProjectionExec` above the scan rather than inlining the volatile expression into the `DataSourceExec` projection.
1 parent f4a700d commit 1e77af8

2 files changed

Lines changed: 260 additions & 1 deletion

File tree

‎datafusion/datasource/src/file_scan_config/mod.rs‎

Lines changed: 230 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ use datafusion_physical_expr::projection::{ProjectionExprs, ProjectionMapping};
4444
use datafusion_physical_expr::utils::reassign_expr_columns;
4545
use datafusion_physical_expr::{EquivalenceProperties, Partitioning, split_conjunction};
4646
use datafusion_physical_expr_adapter::PhysicalExprAdapterFactory;
47-
use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
47+
use datafusion_physical_expr_common::physical_expr::{PhysicalExpr, is_volatile};
4848
use datafusion_physical_expr_common::sort_expr::{LexOrdering, PhysicalSortExpr};
4949
use datafusion_physical_plan::SortOrderPushdownResult;
5050
use datafusion_physical_plan::coop::cooperative;
@@ -625,6 +625,51 @@ fn project_output_partitioning(
625625
}
626626
}
627627

628+
/// Returns `true` if merging `outer` into `inner` would duplicate a volatile
629+
/// expression; the caller should then decline the merge.
630+
///
631+
/// `inner` is the scan's current projection and `outer` the projection being
632+
/// pushed into it; merging substitutes each `inner` expression into every
633+
/// `outer` reference to it. If a volatile `inner` expression (e.g. `random()`,
634+
/// `uuid()`) is referenced more than once, that single value gets inlined at
635+
/// each site and re-evaluated independently, so references meant to share a
636+
/// "locked-in" value diverge. This is the volatility guard the physical
637+
/// `ProjectionPushdown` and `FilterPushdown` rules already apply (see
638+
/// `datafusion_physical_expr_common::physical_expr::is_volatile`).
639+
///
640+
/// References are counted with multiplicity by walking each `outer` expression
641+
/// (as `try_collapse_projection_chain` does), so a self-duplicating expression
642+
/// such as `r + r` counts as two references. A volatile expression referenced
643+
/// exactly once has nothing to duplicate and is left to merge.
644+
fn would_duplicate_volatile_exprs(
645+
inner: &ProjectionExprs,
646+
outer: &ProjectionExprs,
647+
) -> bool {
648+
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};
649+
650+
let inner_exprs = inner.as_ref();
651+
652+
let mut ref_counts = vec![0usize; inner_exprs.len()];
653+
for proj_expr in outer.as_ref() {
654+
proj_expr
655+
.expr
656+
.apply(|e| {
657+
if let Some(col) = e.as_ref().downcast_ref::<Column>()
658+
&& let Some(count) = ref_counts.get_mut(col.index())
659+
{
660+
*count += 1;
661+
}
662+
Ok(TreeNodeRecursion::Continue)
663+
})
664+
.expect("infallible closure should not fail");
665+
}
666+
667+
ref_counts
668+
.iter()
669+
.enumerate()
670+
.any(|(idx, &count)| count > 1 && is_volatile(&inner_exprs[idx].expr))
671+
}
672+
628673
impl DataSource for FileScanConfig {
629674
fn open(
630675
&self,
@@ -911,6 +956,15 @@ impl DataSource for FileScanConfig {
911956
&self,
912957
projection: &ProjectionExprs,
913958
) -> Result<Option<Arc<dyn DataSource>>> {
959+
// Don't merge a projection into the scan if it would inline a volatile
960+
// expression that the outer projection references, which would turn a
961+
// single "locked-in" value (e.g. `random()` aliased in a subquery) into
962+
// multiple independent evaluations. See #23220.
963+
if let Some(inner) = self.file_source.projection()
964+
&& would_duplicate_volatile_exprs(inner, projection)
965+
{
966+
return Ok(None);
967+
}
914968
match self.file_source.try_pushdown_projection(projection)? {
915969
Some(new_source) => {
916970
let mut new_file_scan_config = self.clone();
@@ -3314,4 +3368,179 @@ mod tests {
33143368
);
33153369
Ok(())
33163370
}
3371+
3372+
/// Helper: build a `ProjectionExprs` from `(expr, alias)` pairs.
3373+
fn make_projection(pairs: Vec<(Arc<dyn PhysicalExpr>, &str)>) -> ProjectionExprs {
3374+
ProjectionExprs::new(
3375+
pairs
3376+
.into_iter()
3377+
.map(|(expr, alias)| ProjectionExpr::new(expr, alias)),
3378+
)
3379+
}
3380+
3381+
/// Helper: create a volatile (non-deterministic) function expression,
3382+
/// e.g. `random()`.
3383+
fn make_volatile_expr() -> Arc<dyn PhysicalExpr> {
3384+
use datafusion_common::config::ConfigOptions;
3385+
use datafusion_expr::ScalarUDF;
3386+
use datafusion_functions::math::random::RandomFunc;
3387+
use datafusion_physical_expr::ScalarFunctionExpr;
3388+
3389+
Arc::new(ScalarFunctionExpr::new(
3390+
"random",
3391+
Arc::new(ScalarUDF::from(RandomFunc::new())),
3392+
vec![],
3393+
Arc::new(Field::new("random", DataType::Float64, false)),
3394+
Arc::new(ConfigOptions::default()),
3395+
))
3396+
}
3397+
3398+
/// Column-only inner projections always merge safely, even when
3399+
/// the outer projection references them multiple times.
3400+
#[test]
3401+
fn test_would_duplicate_allows_column_only_inner() {
3402+
let col_a: Arc<dyn PhysicalExpr> = Arc::new(Column::new("a", 0));
3403+
let col_b: Arc<dyn PhysicalExpr> = Arc::new(Column::new("b", 1));
3404+
3405+
let inner =
3406+
make_projection(vec![(Arc::clone(&col_a), "a"), (Arc::clone(&col_b), "b")]);
3407+
3408+
// Outer references col 0 twice
3409+
let outer = make_projection(vec![
3410+
(Arc::new(Column::new("a", 0)), "x"),
3411+
(Arc::new(Column::new("a", 0)), "y"),
3412+
]);
3413+
3414+
assert!(!would_duplicate_volatile_exprs(&inner, &outer));
3415+
}
3416+
3417+
/// Deterministic computed expressions (arithmetic) referenced multiple
3418+
/// times are allowed to merge — only volatile expressions are protected.
3419+
#[test]
3420+
fn test_would_duplicate_allows_deterministic_computed_multi_ref() {
3421+
let col_a: Arc<dyn PhysicalExpr> = Arc::new(Column::new("a", 0));
3422+
let col_b: Arc<dyn PhysicalExpr> = Arc::new(Column::new("b", 1));
3423+
// Inner: [a + b, b] (index 0 is deterministic computed)
3424+
let inner = make_projection(vec![
3425+
(
3426+
Arc::new(BinaryExpr::new(
3427+
Arc::clone(&col_a),
3428+
Operator::Plus,
3429+
Arc::clone(&col_b),
3430+
)),
3431+
"sum",
3432+
),
3433+
(Arc::clone(&col_b), "b"),
3434+
]);
3435+
3436+
// Outer references index 0 twice
3437+
let outer = make_projection(vec![
3438+
(Arc::new(Column::new("sum", 0)), "x"),
3439+
(Arc::new(Column::new("sum", 0)), "y"),
3440+
]);
3441+
3442+
// Deterministic arithmetic → allow merge even though duplicated
3443+
assert!(!would_duplicate_volatile_exprs(&inner, &outer));
3444+
}
3445+
3446+
/// A volatile expression the outer projection does not reference is
3447+
/// safe to merge (it is projected away, not duplicated).
3448+
#[test]
3449+
fn test_would_duplicate_allows_unreferenced_volatile() {
3450+
let col_a: Arc<dyn PhysicalExpr> = Arc::new(Column::new("a", 0));
3451+
// Inner: [random(), a]
3452+
let inner =
3453+
make_projection(vec![(make_volatile_expr(), "r"), (Arc::clone(&col_a), "a")]);
3454+
3455+
// Outer references only index 1 (the column), not the volatile expr
3456+
let outer = make_projection(vec![(Arc::new(Column::new("a", 1)), "a")]);
3457+
3458+
assert!(!would_duplicate_volatile_exprs(&inner, &outer));
3459+
}
3460+
3461+
/// A volatile expression referenced multiple times must block merge:
3462+
/// this is the #23220 regression (`random()` aliased then referenced as
3463+
/// `x` and `y`).
3464+
#[test]
3465+
fn test_would_duplicate_blocks_multi_ref_volatile() {
3466+
// Inner: [random()]
3467+
let inner = make_projection(vec![(make_volatile_expr(), "r")]);
3468+
3469+
// Outer references index 0 twice
3470+
let outer = make_projection(vec![
3471+
(Arc::new(Column::new("r", 0)), "x"),
3472+
(Arc::new(Column::new("r", 0)), "y"),
3473+
]);
3474+
3475+
assert!(would_duplicate_volatile_exprs(&inner, &outer));
3476+
}
3477+
3478+
/// A volatile expression referenced exactly once has nothing to duplicate,
3479+
/// so the merge is allowed.
3480+
#[test]
3481+
fn test_would_duplicate_allows_single_ref_volatile() {
3482+
let col_a: Arc<dyn PhysicalExpr> = Arc::new(Column::new("a", 0));
3483+
// Inner: [random(), a]
3484+
let inner =
3485+
make_projection(vec![(make_volatile_expr(), "r"), (Arc::clone(&col_a), "a")]);
3486+
3487+
// Outer references the volatile expression exactly once
3488+
let outer = make_projection(vec![
3489+
(Arc::new(Column::new("r", 0)), "x"),
3490+
(Arc::new(Column::new("a", 1)), "a"),
3491+
]);
3492+
3493+
assert!(!would_duplicate_volatile_exprs(&inner, &outer));
3494+
}
3495+
3496+
/// References are counted with multiplicity, so a single outer expression
3497+
/// that duplicates the value (e.g. `r + r`) still blocks the merge.
3498+
#[test]
3499+
fn test_would_duplicate_blocks_single_expr_self_ref_volatile() {
3500+
// Inner: [random()]
3501+
let inner = make_projection(vec![(make_volatile_expr(), "r")]);
3502+
3503+
// Outer: [r + r] — one expression referencing `random()` twice
3504+
let outer = make_projection(vec![(
3505+
Arc::new(BinaryExpr::new(
3506+
Arc::new(Column::new("r", 0)),
3507+
Operator::Plus,
3508+
Arc::new(Column::new("r", 0)),
3509+
)),
3510+
"x",
3511+
)]);
3512+
3513+
assert!(would_duplicate_volatile_exprs(&inner, &outer));
3514+
}
3515+
3516+
/// A volatile expression buried inside a larger expression (e.g.
3517+
/// `random() + 1`) is still detected and blocks merge.
3518+
#[test]
3519+
fn test_would_duplicate_blocks_volatile_nested_in_arithmetic() {
3520+
// Inner: [random() + 1]
3521+
let inner = make_projection(vec![(
3522+
Arc::new(BinaryExpr::new(
3523+
make_volatile_expr(),
3524+
Operator::Plus,
3525+
Arc::new(Literal::new(ScalarValue::Float64(Some(1.0)))),
3526+
)),
3527+
"expr",
3528+
)]);
3529+
3530+
// Outer references index 0 twice
3531+
let outer = make_projection(vec![
3532+
(Arc::new(Column::new("expr", 0)), "x"),
3533+
(Arc::new(Column::new("expr", 0)), "y"),
3534+
]);
3535+
3536+
assert!(would_duplicate_volatile_exprs(&inner, &outer));
3537+
}
3538+
3539+
/// Empty projections should not block merging.
3540+
#[test]
3541+
fn test_would_duplicate_empty_projections() {
3542+
let inner = make_projection(vec![]);
3543+
let outer = make_projection(vec![]);
3544+
assert!(!would_duplicate_volatile_exprs(&inner, &outer));
3545+
}
33173546
}

‎datafusion/sqllogictest/test_files/projection_pushdown.slt‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2077,3 +2077,33 @@ SELECT s, id FROM simple_struct WHERE s['value'] > 100 AND id < 4;
20772077
# reset it explicitly.
20782078
statement ok
20792079
SET datafusion.execution.target_partitions = 4;
2080+
2081+
#####################
2082+
# Section: volatile expressions are not duplicated by projection pushdown
2083+
#
2084+
# Regression test for #23220: a volatile expression (e.g. `random()`) aliased
2085+
# once in a subquery and referenced multiple times must be evaluated once and
2086+
# reused. Projection pushdown must not merge the outer projection into the file
2087+
# scan when doing so would inline and duplicate the volatile expression.
2088+
# Reproduces only against a file scan (not an in-memory table); if the volatile
2089+
# expression is duplicated, the two references diverge and `x = y` is false.
2090+
#####################
2091+
2092+
statement ok
2093+
COPY (SELECT 1 AS id UNION ALL SELECT 2 UNION ALL SELECT 3)
2094+
TO 'test_files/scratch/projection_pushdown/volatile.parquet'
2095+
STORED AS PARQUET;
2096+
2097+
statement ok
2098+
CREATE EXTERNAL TABLE volatile_scan STORED AS PARQUET
2099+
LOCATION 'test_files/scratch/projection_pushdown/volatile.parquet';
2100+
2101+
# The two references to the aliased `random()` value must be equal on every
2102+
# row: the expression is evaluated once and reused, not inlined twice.
2103+
query B rowsort
2104+
SELECT s.x = s.y
2105+
FROM (SELECT r AS x, r AS y FROM (SELECT random() AS r FROM volatile_scan) AS t) AS s;
2106+
----
2107+
true
2108+
true
2109+
true

0 commit comments

Comments
 (0)