You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[JNI] List layout scans read the same segments several times #65
A Spark read of a file written with vortex.listLayout=true reads several times the bytes it needs, because the
JNI data source opens files without a segment cache. On a 47 MB list-layout file of nested event data, reading 29
of its columns with the connector on develop moved 79 MB, though the segments those columns need hold 13 MB. The same
read on the same data written without the list layout moved 47 MB. So turning the list layout on made reads more
expensive.
Why
A scan reads a file as several row ranges, and a segment that spans ranges is requested once per range. Flat
columns get these requests deduplicated while they are in flight, because every range asks at once. A list
column's elements are chunked apart from its rows, and a range can only ask for its elements after it has read its
offsets, so those requests come later, after the first read has finished. Nothing keeps the finished segment, so
each range reads it from storage again. Spark's file splitting multiplies this: every split is its own scan.
The 1 MiB read-merging gap the JNI reader uses adds to it: with a cache, a merged read that spans a segment the
cache already holds reads it again.
Measured
A native trace of every byte range read (scan through the data source API, JNI's merge settings, Arrow export as
JNI does), on one 269k-row file:
Read
Needs
Today, 1 split
Today, 4 splits
Cache + 64 KiB gap, 1 split
Cache + 64 KiB gap, 4 splits
29 columns, list layout
13.4 MB
39.5 MB
68.8 MB
13.7 MB
15.7 MB
Every column, list layout
47.2 MB
103.0 MB
296.0 MB
52.8 MB
29 columns, without list layout
28.7 MB
30.0 MB
47.2 MB
28.8 MB
30.1 MB
With a segment cache and no read merging, every case reads each segment exactly once. A 16 MiB cache per file is
enough for this file; larger caches change nothing.
Proposed fix
Give each file the JNI data source opens its own bounded segment cache (MokaSegmentCache, 64 MiB), living as
long as the open file.
Merge reads only across gaps up to 64 KiB, instead of 1 MiB. The reads per file stay about the same (45 to 48
for the 29-column read above).
Activity