Repository navigation
Spark reader returns a file's rows out of order (scan isn't ordered) #48
Description
Summary
The Spark connector reads a file's rows out of order. VortexPartitionReader builds its ScanOptions without ordered(true), so the native scan returns chunks as they finish. A file read as one Spark partition comes back as sorted runs in shuffled order.
This doesn't matter for jobs that don't rely on row order, but it does for:
- readers that expect a file's order, such as merge-joining files sorted by key;
- order checks in tests.
Repro
We have a file of 442,632 rows sorted by a string key, written through VortexWriter.
| How the file is read | Order breaks |
|---|---|
DataSource.scan(... ordered(true)) |
0 |
spark.read.format("vortex").load(file) (1 input partition), local[1] |
3, at rows 88,527, 265,578 and 354,105 |
Same, local[4] |
2, at different rows on each run |
Each break is a jump back to a smaller key at a chunk boundary, and every run between breaks is sorted. That's the pattern of an unordered scan.
Fix
Set ordered(true) in VortexPartitionReader, or make it an option. Only the rows inside one Spark partition need ordering; Spark doesn't promise an order across partitions.
Concurrency within a single file split may drop as a result; that needs measuring.
Activity
ChristopherWirt commentedon Nov 3, 2026
Closing as not planned. The chunk reordering is reproducible, but Spark's PartitionReader contract does not require rows to be returned in physical file order. We should therefore not enable ordered native scans for every Spark read. PR #76 is being withdrawn.
Explicit, verified sort-order reporting through
SupportsReportOrderingis tracked in #80. That work must ensure each Spark input partition honors any ordering it advertises; preserving physical file order alone does not establish a sort order.