Repository navigation
Account for partitions in local TopK statistics #25619
Description
Describe the bug
Local TopK statistics cap the total estimate at one partition’s limit, while
runtime metrics sum rows across all partitions. A simplified grouped query
exposes the mismatch without the joins in TPC-H Q3.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SET datafusion.execution.target_partitions = 1;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10;
SET datafusion.execution.target_partitions = 4;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10;
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| SELECT | Operator / node | Estimated rows | Actual rows |
|---|---|---|---|
| One partition | SortExec(TopK), 0.0 |
10 | 10 |
| Four partitions | SortExec(TopK), 0.0.0 |
10 | 40 |
| Four partitions | SortPreservingMergeExec, 0 |
10 | 10 |
Expected behavior
Account for the emitter count in local TopK estimates while preserving the
global merge’s limit. Per-partition and overall statistics must use consistent
units.
Additional context
Dynamic filtering is disabled. Early cancellation can reduce intermediate output
in other plans; this reproducer emits all 40 local rows.
Part of #25610.
Activity
hassaanch23 commentedon Sep 23, 2026
take
fix: account for every partition in local TopK and limit statistics (a…
fix: account for every partition in local TopK and limit statistics (a…