Repository navigation
fix: TopK dynamic filter returns wrong rows when one sort key contains another - #27118
Open
shahwazreza wants to merge 1 commit into
Open
shahwazreza wants to merge 1 commit into
shahwazreza wants to merge 1 commit into
Conversation
shahwazreza
commented
Nov 5, 2026
Comment on lines
+1203
to
+1204
| 02)--SortExec: TopK(fetch=3), expr=[<slt:ignore>], preserve_partitioning=[false] | ||
| 03)----ProjectionExec: expr=[<slt:ignore>] |
Contributor
Author
There was a problem hiding this comment.
Note on the two <slt:ignore> lines: when I ran this query repeatedly, the middle ProjectionExec didn't always list its columns in the same order (p, b, q in some runs, p, q, b in others), which also moves b between @1 and @2 in the SortExec. This also happens without this PR's change, and the results are the same either way, so I only ignored those two lines. The test still checks for the TopK SortExec and the DynamicFilter on the Parquet scan. I'm going to look into why the plan isn't stable and open a separate issue for it, and I'll link it here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
With
datafusion.execution.parquet.pushdown_filters = true, anORDER BY ... LIMITreturns the wrong rows when one sort key contains another sort key, for exampleORDER BY CASE WHEN b % 2 = 0 THEN p ELSE q END, b. With other column types (e.g.basUInt64) the same query fails with anInvalid comparison operationerror instead. Turning offenable_topk_dynamic_filter_pushdowngives the right answer, so the problem is in the TopK dynamic filter that gets pushed into the Parquet scan.What changes are included in this PR?
When the filter is pushed below a projection,
DynamicFilterPhysicalExpr::remap_childrenrewrites each original sort key into the new input's columns. It usedtransform_up, so in the example above the innerbwas rewritten first. After that the outerCASEno longer equalled the original sort key, so it was never rewritten, and itsp@0reached the scan, where index 0 isb.remap_childrennow usestransform_down, so each sort key is matched as a whole before anything inside it changes.TreeNodeRecursion::Jump, so the replacement is not searched again. The replacement is already written against the new input; withoutJump, a column inside it can be rewritten a second time.This is the approach that was approved in review on #26031. That PR was closed without being merged, so I've picked the issue up here. The change is in the shared remapping code, so it applies to every dynamic filter, not only TopK.
What is the testing strategy for this PR?
Two sqllogictest cases at the end of
dynamic_filter_pushdown_config.slt, each with anEXPLAINthat checks TopK's filter reaches the Parquet scan, followed by a check of the query results:mainit returns9, 10, 11instead of0, 1, 2.SELECT b + 1 AS a, a AS b ... ORDER BY a, b LIMIT 3. In the scan, the sort keyabecomesb@1 + 1, and thatb@1is equal to the other sort keyb. This passes onmain, but fails ifJumpis removed, because theb@1gets rewritten again and the wrong rows come back.The
EXPLAINchecks follow the review feedback on #26031, so the tests can't silently stop testing the bug if the filter no longer reaches the scan. In the first case, the middle projection's column order changes between runs, so those two plan lines use<slt:ignore>. Both cases can be reached from SQL, so I didn't add Rust unit tests.Are there any user-facing changes?
Queries like the ones above now return the correct rows, or no longer fail with a comparison error.