Repository navigation
Propagate window ranking statistics into rank filters #25622
Description
Describe the bug
Filtering a window ranking to the first ten rows retains the full input
estimate. The reduced example estimates 73,049 rows and emits ten; the window
input itself is estimated correctly.
To Reproduce
From the repository root, with
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/date_dim.parquet \
-o "$repro_dir/data/date_dim.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT * FROM (
SELECT d_date_sk, ROW_NUMBER() OVER (ORDER BY d_date_sk) AS rn
FROM date_dim
) t WHERE rn <= 10;
SELECT * FROM (
SELECT d_date_sk, RANK() OVER (ORDER BY d_date_sk) AS rn
FROM date_dim
) t WHERE rn <= 10;
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
| Predicate | FilterExec node | Estimated rows | Actual rows |
|---|---|---|---|
| ROW_NUMBER() <= 10 | 0.0 |
73,049 | 10 |
| RANK() <= 10 | 0.0 |
73,049 | 10 |
Expected behavior
Expose usable ranking-column statistics. A global ROW_NUMBER() <= 10 has an
upper bound of ten rows. RANK needs tie-aware estimation, and partitioned
windows need partition-count estimates; neither can generally use that same
global bound.
Additional context
Reduced from the ranking filters in TPC-DS Q49/Q67/Q70, with ROW_NUMBER added as
a stronger semantic control. BoundedWindowAggExec currently appends unknown
column statistics for its window outputs. This concerns estimates, independently
of whether execution can stop early.
Part of #25610.
Activity
mohitgurav20 commentedon Sep 24, 2026
Hi @gabotechs, thanks for putting together this detailed issue and repro!
This looks like a great cardinality estimation improvement for the TPC-DS epic. I took a brief look at BoundedWindowAggExec (and WindowAggExec), and it seems we're currently just appending ColumnStatistics::new_unknown() for the generated window expression columns.
I'd love to take this on! My plan would be to update the statistics() implementation for the window execution nodes to properly derive bounds for ranking functions.
For unpartitioned ROW_NUMBER(), we can easily establish an upper bound based on the input row count estimate.
For partitioned windows or tie-aware functions like RANK(), we can compute tighter bounds using the input's partition count estimates and row counts.
Could you assign this to me if no one else is currently working on it?
mohitgurav20 commentedon Sep 24, 2026
take
kosiew commentedon Nov 12, 2026
hi @mohitgurav20
Are you working on this?
mohitgurav20 commentedon Nov 12, 2026
@kosiew no sir it's not assigned to me or no reply from the author so didn't carry it on if assigned would like to work on it
kosiew commentedon Nov 12, 2026
Assigned.
mohitgurav20 commentedon Nov 12, 2026
@kosiew thanks sir will work on it