Skip to content

[JNI] List layout scans read the same segments several times #65

Closed

Description

@gitRasheed

A Spark read of a file written with vortex.listLayout=true reads several times the bytes it needs, because the
JNI data source opens files without a segment cache. On a 47 MB list-layout file of nested event data, reading 29
of its columns with the connector on develop moved 79 MB, though the segments those columns need hold 13 MB. The same
read on the same data written without the list layout moved 47 MB. So turning the list layout on made reads more
expensive.

Why

A scan reads a file as several row ranges, and a segment that spans ranges is requested once per range. Flat
columns get these requests deduplicated while they are in flight, because every range asks at once. A list
column's elements are chunked apart from its rows, and a range can only ask for its elements after it has read its
offsets, so those requests come later, after the first read has finished. Nothing keeps the finished segment, so
each range reads it from storage again. Spark's file splitting multiplies this: every split is its own scan.

The 1 MiB read-merging gap the JNI reader uses adds to it: with a cache, a merged read that spans a segment the
cache already holds reads it again.

Measured

A native trace of every byte range read (scan through the data source API, JNI's merge settings, Arrow export as
JNI does), on one 269k-row file:

Read Needs Today, 1 split Today, 4 splits Cache + 64 KiB gap, 1 split Cache + 64 KiB gap, 4 splits
29 columns, list layout 13.4 MB 39.5 MB 68.8 MB 13.7 MB 15.7 MB
Every column, list layout 47.2 MB 103.0 MB 296.0 MB 52.8 MB
29 columns, without list layout 28.7 MB 30.0 MB 47.2 MB 28.8 MB 30.1 MB

With a segment cache and no read merging, every case reads each segment exactly once. A 16 MiB cache per file is
enough for this file; larger caches change nothing.

Proposed fix

  • Give each file the JNI data source opens its own bounded segment cache (MokaSegmentCache, 64 MiB), living as
    long as the open file.
  • Merge reads only across gaps up to 64 KiB, instead of 1 MiB. The reads per file stay about the same (45 to 48
    for the 29-column read above).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions