Repository navigation
fix: PartitionedTopKRank counts tie rows twice in output_rows - #25535
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25535 +/- ##
==========================================
- Coverage 82.38% 82.38% -0.01%
==========================================
Files 1138 1138
Lines 434491 434519 +28
Branches 434491 434519 +28
==========================================
+ Hits 357969 357981 +12
- Misses 54875 54884 +9
- Partials 21647 21654 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kosiew
approved these changes
Sep 21, 2026
kosiew
left a comment
kosiew
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the fix. This looks good to me.
Removing the early record_output call for tie batches makes sense since those rows are subsequently pushed through the coalescer and counted when the actual output batch is emitted. This keeps the RANK path consistent with the other Top-K implementations and avoids double-counting both rows and batches.
The regression test also captures the issue nicely by checking that the metrics match the rows and batches actually emitted, including the tie at the K-th value.
LGTM!
Contributor
Author
|
Thanks @kosiew ! |
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Sep 21, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 22, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Sep 23, 2026
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
…#25535) ## Which issue does this PR close? - N/A — no issue filed; found while reading the code. ## Rationale for this change For a `RANK()` window Top-N (`PartitionedTopKExec: fn=rank`), the `output_rows` and `output_batches` metrics shown by `EXPLAIN ANALYZE` are too high whenever rows tie at the K-th value. `PartitionedTopKRank::emit` calls `record_output` on every tie batch before pushing it into the `BatchCoalescer`, and then calls `record_output` again on each completed batch coming out of that coalescer — which already contains those tie rows. Heap rows are only counted at the second site, so each tie row is counted twice and each tie batch adds a phantom output batch. Example: K = 2, one partition with values `5, 5, 10, 5`. Three rows are emitted in one batch, but the metrics report `output_rows=4`, `output_batches=2`. `PartitionedTopK` (`ROW_NUMBER`) and `PartitionedTopKDenseRank` only record at the coalescer output and are not affected. ## What changes are included in this PR? Remove the extra `record_output` call on tie batches so every emitted row is counted once, at the coalescer output, as in the other two operators. Query results are unchanged; only the metrics are corrected. ## What is the testing strategy for this PR? New unit test `test_partitioned_topk_rank_output_rows_counts_ties_once`, which emits heap rows plus a boundary tie and asserts `output_rows` / `output_batches` equal what the stream actually produced. It fails on `main` with `(2, 4)` vs `(1, 3)`. ## Are there any user-facing changes? `output_rows` / `output_batches` reported for `PartitionedTopKExec` with `fn=rank` are now accurate when ties are present. No API changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
For a
RANK()window Top-N (PartitionedTopKExec: fn=rank), theoutput_rowsandoutput_batchesmetrics shown byEXPLAIN ANALYZEare too high whenever rows tie at the K-th value.PartitionedTopKRank::emitcallsrecord_outputon every tie batch before pushing it into theBatchCoalescer, and then callsrecord_outputagain on each completed batch coming out of that coalescer — which already contains those tie rows. Heap rows are only counted at the second site, so each tie row is counted twice and each tie batch adds a phantom output batch.Example: K = 2, one partition with values
5, 5, 10, 5. Three rows are emitted in one batch, but the metrics reportoutput_rows=4,output_batches=2.PartitionedTopK(ROW_NUMBER) andPartitionedTopKDenseRankonly record at the coalescer output and are not affected.What changes are included in this PR?
Remove the extra
record_outputcall on tie batches so every emitted row is counted once, at the coalescer output, as in the other two operators. Query results are unchanged; only the metrics are corrected.What is the testing strategy for this PR?
New unit test
test_partitioned_topk_rank_output_rows_counts_ties_once, which emits heap rows plus a boundary tie and assertsoutput_rows/output_batchesequal what the stream actually produced. It fails onmainwith(2, 4)vs(1, 3).Are there any user-facing changes?
output_rows/output_batchesreported forPartitionedTopKExecwithfn=rankare now accurate when ties are present. No API changes.