You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2f95eef
Browse filesBrowse the repository at this point in the historyBrowse files
fix: declare the streamed order only over operators that keep a JVM input's order
From review:
- with_declared_order walks down from the probe input through operators
that report keeping their input's order and declares only when it
reaches a ScanExec or ShuffleScanExec. DataFusion reports a right hash
join as keeping its probe order even when it moves unmatched rows, so
the walk stops at any hash join that is not an inner join. The
null-aware anti join stops it, so an order Spark reports above NOT IN
no longer reaches the aggregates above the outer join.
- doConvert sends the longest prefix of the streamed order that Comet can
express, since declaring any order keeps the probe rows in place, and
falls back only when it cannot express the first key.
- SortedInputExec's doc names the top-k sort as another consumer, and the
CometHashJoin comment names the null-aware anti join as the exception.
- Tests: dwsmith1983's NOT IN repro, a residual join condition, a
descending, nulls last streamed order with null keys, the prefix case,
the aggregate test's join and exchange checks, and planner tests for
the walk and the planned join type.
0 commit comments