Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
test: check per rule that the pushdown rules reach a fixed point
Address review: add an observer test that fails when PushDownFilter or
PushDownLeafProjections changes the plan in the last optimizer pass, and
correct the rule precedence wording in the optimizer guide.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
  • Loading branch information
adriangb and claude committed Oct 6, 2026
commit 406e428657115bbd268bf1ed13bd746a3080835e
59 changes: 59 additions & 0 deletions datafusion/optimizer/src/push_down_filter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4627,6 +4627,65 @@ mod tests {
)
}

/// Runs the default optimizer and returns, for each pass, the names of the
/// rules that changed the plan in that pass.
fn rules_that_changed_plan_per_pass(plan: LogicalPlan) -> Result<Vec<Vec<String>>> {
let optimizer = Optimizer::new();
let rules_per_pass = optimizer.rules.len();
let mut previous = plan.display_indent().to_string();
let mut calls = 0;
let mut passes: Vec<Vec<String>> = vec![];
optimizer.optimize(plan, &OptimizerContext::new(), |plan, rule| {
if calls % rules_per_pass == 0 {
passes.push(vec![]);
}
calls += 1;
let current = plan.display_indent().to_string();
if current != previous {
passes.last_mut().unwrap().push(rule.name().to_string());
previous = current;
}
})?;
Ok(passes)
}

/// `PushDownFilter` and `PushDownLeafProjections` must not undo each other
/// for a filter next to a pure extraction projection
/// (<https://github.com/apache/datafusion/issues/14540>). Neither rule may
/// change the plan in the last optimizer pass. The source does not absorb
/// filters, so the `Filter` node stays in the plan.
#[test]
fn filter_and_extraction_projection_reach_fixed_point() -> Result<()> {
let scan = || {
table_scan_with_pushdown_provider_builder(
TableProviderFilterPushDown::Unsupported,
vec![],
None,
)
};
let simple = scan()?
.filter(col("b").gt(lit(5)))?
.project(vec![leaf_udf_expr(col("a"))])?
.build()?;
// Two filters, one of them on an extracted leaf.
let two_filters = scan()?
.filter(col("b").gt(lit(5)))?
.filter(leaf_udf_expr(col("a")).eq(lit(1)))?
.project(vec![leaf_udf_expr(col("a")), col("b")])?
.build()?;

for plan in [simple, two_filters] {
let passes = rules_that_changed_plan_per_pass(plan)?;
let last = passes.last().unwrap();
assert!(
!last.iter().any(|rule| rule == "push_down_filter"
|| rule == "push_down_leaf_projections"),
"pushdown rules changed the plan in the last pass: {passes:?}"
);
}
Ok(())
}

/// A projection that mixes an extraction alias with a computed expression is
/// not a pure extraction projection, so the filter still moves below it.
#[test]
Expand Down
4 changes: 2 additions & 2 deletions docs/source/library-user-guide/query-optimizer.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,9 +164,9 @@ the actual optimization. This approach is used in projection push down and filte

### Rule Precedence

Two rules can want the opposite order for the same pair of adjacent plan nodes. Each rule then undoes the work of the other one on every optimizer pass. The loop stops at `datafusion.optimizer.max_passes`, and the position of the two rules in the rule list decides the plan. This wastes a pass and makes the plan depend on the rule list.
Two rules can want the opposite order for the same pair of adjacent plan nodes. Each rule then undoes the work of the other one on every optimizer pass. The plan does not reach a fixed point, so the optimizer runs more passes than necessary, and the position of the two rules in the rule list decides the plan. The optimizer can stop early when the plan at the end of a pass is the same as the plan at the end of an earlier pass, but that does not make the result independent of the rule order.

Do not let two rules compete. Give one rule precedence, make that rule yield, and record the decision in the module documentation of both rules.
Do not let two rules compete. Give one rule precedence, make the competing rule yield, and record the decision in the module documentation of both rules.

There is one such decision today:

Expand Down
Loading