Skip to content
Merged
Prev Previous commit
Next Next commit
Add fetch method to LogicalPlan
  • Loading branch information
shivbhatia10 committed Mar 20, 2026
commit 0e83891753d055f62dea152c9cacad2a9f4a642c
34 changes: 34 additions & 0 deletions datafusion/expr/src/logical_plan/plan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1392,6 +1392,40 @@ impl LogicalPlan {
}
}

/// Returns the fetch (limit) of this plan node, if it has one.
///
/// Only [`LogicalPlan::Sort`] and [`LogicalPlan::TableScan`] carry a fetch
/// value directly; all other variants return `None`.
pub fn fetch(&self) -> Option<usize> {
match self {
LogicalPlan::Sort(Sort { fetch, .. }) => *fetch,
LogicalPlan::TableScan(TableScan { fetch, .. }) => *fetch,
LogicalPlan::Projection(_) => None,
LogicalPlan::Filter(_) => None,
LogicalPlan::Window(_) => None,
LogicalPlan::Aggregate(_) => None,
LogicalPlan::Join(_) => None,
LogicalPlan::Repartition(_) => None,
LogicalPlan::Union(_) => None,
LogicalPlan::EmptyRelation(_) => None,
LogicalPlan::Subquery(_) => None,
LogicalPlan::SubqueryAlias(_) => None,
LogicalPlan::Limit(_) => None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am surprised LogicalPlan::Limit does not return a value here as well;

I looked into the code a bit to try and understand why Limits were needed. Initially I thought it was because the following code claims that we only push down Filters after Limits

// Filters can't be pushed down past Limits, we should do PushDownFilter after PushDownLimit
Arc::new(PushDownLimit::new()),
Arc::new(PushDownFilter::new()),

However, at some point we started running multiple optimizer passes so PushdownFilter can (effectively) run after PushDownLimit

while i < options.optimizer.max_passes {
log_plan(&format!("Optimizer input (pass {i})"), &new_plan);
for rule in &self.rules {
// If skipping failed rules, copy plan before attempting to rewrite
// as rewriting is destructive
let prev_plan = options
.optimizer
.skip_failed_rules
.then(|| new_plan.clone());
let starting_schema = Arc::clone(new_plan.schema());
let result = match rule.apply_order() {
// optimizer handles recursion
Some(apply_order) => new_plan.rewrite_with_subqueries(
&mut Rewriter::new(apply_order, rule.as_ref(), config),

So this bug was probably introduced when we started running multiple optimizer passes

LogicalPlan::Statement(_) => None,
LogicalPlan::Values(_) => None,
LogicalPlan::Explain(_) => None,
LogicalPlan::Analyze(_) => None,
LogicalPlan::Extension(_) => None,
LogicalPlan::Distinct(_) => None,
LogicalPlan::Dml(_) => None,
LogicalPlan::Ddl(_) => None,
LogicalPlan::Copy(_) => None,
LogicalPlan::DescribeTable(_) => None,
LogicalPlan::Unnest(_) => None,
LogicalPlan::RecursiveQuery(_) => None,
}
}

/// If this node's expressions contains any references to an outer subquery
pub fn contains_outer_reference(&self) -> bool {
let mut contains = false;
Expand Down
20 changes: 6 additions & 14 deletions datafusion/optimizer/src/push_down_filter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -796,6 +796,12 @@ impl OptimizerRule for PushDownFilter {
filter.predicate = new_predicate;
}

// If the child has a fetch (limit), pushing a filter below it would

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

// change semantics: the limit should apply before the filter, not after.
if filter.input.fetch().is_some() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should also either explicitly check here for Limit or (better yet) change fetch() to also return Some for LogicalPlan::Limit for the reasons I explain above

@shivbhatia10 shivbhatia10 Mar 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes a lot of sense, the reason I initially skipped Limit was because we already handle it correctly in PushDownFilter::rewrite, although very indirectly, because Limit is not covered in the match statement and therefore it falls back to leaving the filter unchanged. However this is a fragile assumption - I think the right thing to do here is to add another method to LogicalPlan called skip, since Limit can have both fetch and skip variants, and if either of them is non-empty we should block pushdown. This is true in general for any logical plan node I believe. So I've added that method, used the get_skip_type and get_fetch_type methods on Limit to extract these Optional<usize> types, and now we check for the presence of both skip and fetch in push down filter.

return Ok(Transformed::no(LogicalPlan::Filter(filter)));
}

match Arc::unwrap_or_clone(filter.input) {
LogicalPlan::Filter(child_filter) => {
let parents_predicates = split_conjunction_owned(filter.predicate);
Expand Down Expand Up @@ -832,13 +838,6 @@ impl OptimizerRule for PushDownFilter {
insert_below(LogicalPlan::Distinct(distinct), new_filter)
}
LogicalPlan::Sort(sort) => {
// If the sort has a fetch (limit), pushing a filter below
// it would change semantics: the limit should apply before
// the filter, not after.
if sort.fetch.is_some() {
filter.input = Arc::new(LogicalPlan::Sort(sort));
return Ok(Transformed::no(LogicalPlan::Filter(filter)));
}
let new_filter =
Filter::try_new(filter.predicate, Arc::clone(&sort.input))
.map(LogicalPlan::Filter)?;
Expand Down Expand Up @@ -1137,13 +1136,6 @@ impl OptimizerRule for PushDownFilter {
}
LogicalPlan::Join(join) => push_down_join(join, Some(&filter.predicate)),
LogicalPlan::TableScan(scan) => {
// If the scan has a fetch (limit), pushing filters into it
// would change semantics: the limit should apply before the
// filter, not after.
if scan.fetch.is_some() {
filter.input = Arc::new(LogicalPlan::TableScan(scan));
return Ok(Transformed::no(LogicalPlan::Filter(filter)));
}
let filter_predicates = split_conjunction(&filter.predicate);

let (volatile_filters, non_volatile_filters): (Vec<&Expr>, Vec<&Expr>) =
Expand Down
Loading