You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: OFFSET pushdown with WHERE filters using fully-matched RG statistics #21916
#21828 implements OFFSET pushdown for parquet queries without filters. Queries with WHERE clauses still use GlobalLimitExec for offset handling because row counts may be inaccurate after filtering.
Problem
For queries like SELECT * FROM table WHERE date >= '2020-01-01' LIMIT 5 OFFSET 1000000, the offset is handled by GlobalLimitExec even when statistics prove all rows in some RGs satisfy the filter.
Opportunity
prune_by_statistics already marks RGs as is_fully_matched when column statistics prove ALL rows satisfy the predicate (e.g., min(date) >= '2020-01-01'). For these RGs, num_rows is the exact qualifying row count — safe to use for offset calculation.
Stop at the first non-fully-matched RG (qualifying row count unknown)
GlobalLimitExec handles the remaining offset (reduced by skipped rows)
Need mechanism to communicate skipped row count from parquet opener back to GlobalLimitExec (reduce its skip)
Challenge
The key difficulty is coordinating between parquet-level RG skipping and GlobalLimitExec's skip counter. The optimizer sets GlobalLimitExec(skip=N) at plan time, but the actual RG-level skipping happens at runtime. Options:
Shared counter between opener and GlobalLimitExec
Dynamic adjustment of GlobalLimitExec skip based on DataSourceExec's output
Activity