You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3090d85
Browse filesBrowse the repository at this point in the historyBrowse files
fix: find shared subexpressions where shuffle keys and TakeOrderedAndProject are serialized
The native shuffle writer serializes its partitioning keys, and CometTakeOrderedAndProjectExec its
sort keys and projection, when the plan runs, outside the operator conversion that sets the shared
subexpressions. canShortCircuitNulls took the missing set as an empty one, so a cast that two
partitioning keys share was skipped on the rows whose array is NULL, where Spark's projection
evaluates it for every row and raises.
canShortCircuitNulls now skips no argument outside withSharedSubexpressions. The shuffle writer
calls it with the partitioning keys, which Spark evaluates together in one projection, and
CometTakeOrderedAndProjectExec with its own expressions.
0 commit comments