You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9c9ff57
Browse filesBrowse the repository at this point in the historyBrowse files
Evaluate a masked argument through one copy, so shuffle keeps its state
A masked argument was evaluated through its rebuilt copy, which reads its
columns from a batch of just those columns, only for a batch with a NULL
in an earlier argument, and through the original for every other batch.
Rebuilding an argument whose column indices change gives it new state, so
array_union(a, shuffle(b)) restarted shuffle's random sequence each time
evaluation moved between the two, and drew different permutations from
Spark's. The rebuilt copy now evaluates a masked argument for every batch.
A unit test evaluates array_union(a, shuffle(b)) over a batch without a
NULL array, one with, and one without again, against the unwrapped
function, which draws the same permutations as Spark when b is NULL
wherever a is. A Spark 4.0+ SQL fixture reads 8200 rows from one file,
so that only the scan's second batch has NULL arrays. Both fail with
the previous evaluation.
0 commit comments