Skip to content

Commit 92290cb

Browse files
Move filter for HAVING and QUALIFY when volatile aliases are used above projection (apache#26850)
## Which issue does this PR close? - Closes apache#7976 - Commit [158a1b8](apache@158a1b8) -> Makes test that fails, as expected, on the expected correct behavior - Commit [49e5f9c](apache@49e5f9c) -> Dont push down volatile filter expressions - new tests still fail as we arent rewriting to subquery - Commit [3895261](apache@3895261) -> Move the filter of volatile expression above projection - Commit [2f358c0](apache@2f358c0) -> Split conjuncts so we can plan more cases following @zhuqi-lucas feedback. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> The current behavior returns unexpected / incorrect results since the volatile expression used for evaluating predicate is different from the one used for projection. ## What changes are included in this PR? When a filter on a volatile alias is used with `HAVING` / `QUALIFY` it is moved above the projection so it operates on the same value that is projected. ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> SLT tests are added for the triggering cases and for cases that arent allowed with the new plan (primarily cases when there are filters on fields that arent in the projection so they are unavailable for filtering above the projection). ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 7ee8f52 commit 92290cb

5 files changed

Lines changed: 415 additions & 35 deletions

File tree

‎datafusion/physical-plan/src/projection.rs‎

Lines changed: 89 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ use datafusion_expr::ExpressionPlacement;
4242
use datafusion_physical_expr::EquivalenceProperties;
4343
use datafusion_physical_expr::equivalence::ProjectionMapping;
4444
use datafusion_physical_expr::projection::Projector;
45-
use datafusion_physical_expr_common::physical_expr::{PhysicalExprRef, fmt_sql};
45+
use datafusion_physical_expr_common::physical_expr::{
46+
PhysicalExprRef, fmt_sql, is_volatile,
47+
};
4648
use datafusion_physical_expr_common::sort_expr::{
4749
LexOrdering, LexRequirement, PhysicalSortExpr,
4850
};
@@ -524,10 +526,17 @@ impl ExecutionPlan for ProjectionExec {
524526
for filter in parent_filters {
525527
// Check that every column is a valid output column of this
526528
// projection, then replace each one with the expression at that
527-
// output position.
529+
// output position. A filter that becomes volatile after the
530+
// rewrite must stay above the projection: inlining a volatile
531+
// expression evaluates it independently, so the filter would test
532+
// a different value than the one the projection outputs.
528533
if let Some(reassigned) = remapper.try_remap(&filter)? {
529534
let rewritten = self.projection_expr().unproject_expr(&reassigned)?;
530-
child_parent_filters.push(PushedDownPredicate::supported(rewritten));
535+
if is_volatile(&rewritten) {
536+
child_parent_filters.push(PushedDownPredicate::unsupported(filter));
537+
} else {
538+
child_parent_filters.push(PushedDownPredicate::supported(rewritten));
539+
}
531540
} else {
532541
child_parent_filters.push(PushedDownPredicate::unsupported(filter));
533542
}
@@ -1574,6 +1583,7 @@ mod tests {
15741583

15751584
use datafusion_expr::{Operator, ScalarUDF};
15761585
use datafusion_functions::core::arrow_metadata::ArrowMetadataFunc;
1586+
use datafusion_functions::math::random::RandomFunc;
15771587
use datafusion_physical_expr::ScalarFunctionExpr;
15781588
use datafusion_physical_expr::expressions::{
15791589
BinaryExpr, CastExpr, Column, DynamicFilterPhysicalExpr, Literal, binary, col,
@@ -2236,6 +2246,82 @@ mod tests {
22362246
Ok(())
22372247
}
22382248

2249+
#[test]
2250+
fn test_filter_pushdown_volatile_projection_expr() -> Result<()> {
2251+
let input_schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]);
2252+
let input = Arc::new(StatisticsExec::new(
2253+
Statistics::new_unknown(&input_schema),
2254+
input_schema.clone(),
2255+
));
2256+
2257+
// project "a", random() as "r"
2258+
let random_expr = Arc::new(ScalarFunctionExpr::new(
2259+
"random",
2260+
Arc::new(ScalarUDF::new_from_impl(RandomFunc::new())),
2261+
vec![],
2262+
Arc::new(Field::new("r", DataType::Float64, false)),
2263+
Arc::new(ConfigOptions::default()),
2264+
)) as Arc<dyn PhysicalExpr>;
2265+
let projection = ProjectionExec::try_new(
2266+
vec![
2267+
ProjectionExpr {
2268+
expr: Arc::new(Column::new("a", 0)),
2269+
alias: "a".to_string(),
2270+
},
2271+
ProjectionExpr {
2272+
expr: random_expr,
2273+
alias: "r".to_string(),
2274+
},
2275+
],
2276+
input,
2277+
)?;
2278+
2279+
// filter "a > 5" is deterministic and is pushed down, rewritten to
2280+
// reference the input column
2281+
let deterministic_filter = Arc::new(BinaryExpr::new(
2282+
Arc::new(Column::new("a", 0)),
2283+
Operator::Gt,
2284+
Arc::new(Literal::new(ScalarValue::Int32(Some(5)))),
2285+
)) as Arc<dyn PhysicalExpr>;
2286+
2287+
// filter "r > 0.5" must stay above the projection: unprojecting it
2288+
// would inline an independent evaluation of random(), so the filter
2289+
// would test a different value than the one the projection outputs
2290+
let volatile_filter = Arc::new(BinaryExpr::new(
2291+
Arc::new(Column::new("r", 1)),
2292+
Operator::Gt,
2293+
Arc::new(Literal::new(ScalarValue::Float64(Some(0.5)))),
2294+
)) as Arc<dyn PhysicalExpr>;
2295+
2296+
let description = projection.gather_filters_for_pushdown(
2297+
FilterPushdownPhase::Post,
2298+
vec![deterministic_filter, Arc::clone(&volatile_filter)],
2299+
&ConfigOptions::default(),
2300+
)?;
2301+
2302+
let expected_pushed = Arc::new(BinaryExpr::new(
2303+
Arc::new(Column::new("a", 0)),
2304+
Operator::Gt,
2305+
Arc::new(Literal::new(ScalarValue::Int32(Some(5)))),
2306+
)) as Arc<dyn PhysicalExpr>;
2307+
2308+
let pushed_filters = &description.parent_filters()[0];
2309+
assert_eq!(pushed_filters.len(), 2);
2310+
assert!(matches!(pushed_filters[0].discriminant, PushedDown::Yes));
2311+
assert_eq!(
2312+
format!("{}", pushed_filters[0].predicate),
2313+
format!("{}", expected_pushed)
2314+
);
2315+
assert!(matches!(pushed_filters[1].discriminant, PushedDown::No));
2316+
// the unsupported filter is kept unchanged
2317+
assert_eq!(
2318+
format!("{}", pushed_filters[1].predicate),
2319+
format!("{}", volatile_filter)
2320+
);
2321+
2322+
Ok(())
2323+
}
2324+
22392325
#[test]
22402326
fn test_filter_pushdown_with_multiple_aliases() -> Result<()> {
22412327
let input_schema = Schema::new(vec![

0 commit comments

Comments
 (0)