-
Notifications
You must be signed in to change notification settings - Fork 2k
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
- Posthumously filing to track fix: validate inter-file ordering in eq_properties() #20329
TLDR: just because the files themselves are sorted doesn't mean the partition streams are sorted.
eq_properties()inFileScanConfigblindly trustedoutput_ordering(set from Parquetsorting_columnsmetadata) without verifying that files within a group are in the correct inter-file orderEnforceSortingthen removedSortExecbased on this unvalidated ordering, producing wrong results when filesystem order didn't match data order
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working