Skip to content

fix(jni): cache each open file's segments and merge reads across small gaps only - #66

Merged
ChristopherWirt merged 1 commit into
developfrom
rasheed/jni-segment-cache
Nov 2, 2026
Merged

ChristopherWirt merged 1 commit into
developfrom
rasheed/jni-segment-cache

Conversation

@gitRasheed

Copy link
Copy Markdown
Collaborator

Summary

Closes #65. Spark reads of list-layout files read several times the bytes they need, because the JNI data
source opens files without a segment cache and a list column's element segments are requested once per row range
of a scan. On a 47 MB list-layout file of nested event data, a 29-column read now reads 13.7 MB instead of 39.5 MB,
and 15.7 MB instead of 68.8 MB when Spark splits the file in four. Before this change, the same read of the file
written without the list layout read 30.0 MB, so the list layout now pays off instead of costing more.

Changes

  • Each file the JNI data source opens gets its own segment cache, capped at 64 MiB and dropped with the open file.
    Both NativeDataSource.open and openFiles use it.
  • JavaReadable merges reads across gaps of at most 64 KiB instead of 1 MiB. With a cache, a merged read that
    spans a segment the cache already holds reads it again; at 1 MiB that left list-layout reads at 1.3x to 1.8x
    the bytes they need, at 64 KiB 1.0x to 1.2x.

Testing

  • writer::tests::list_layout_scan_reads_each_segment_once: writes a list-of-struct column with the list layout,
    scans it as four row ranges through the data source, and checks that no segment is read twice with the JNI open
    options. It also checks that without them some segments are, so the test keeps reproducing the problem.
  • cargo test -p vortex-jni, cargo clippy -p vortex-jni --all-targets.
  • Spark, local mode, the connector on develop with this native library, same file, bytes the JVM read for the
    29-column read: list layout 21.5 to 25.0 MB before, 14.5 MB after; split in four, 77 to 92 MB before, 48.4 MB
    after. Without the list layout: 30.6 MB before, 29.5 MB after. Row counts match.

Not fixed here

Spark reads each split of a file in its own task, so splits can't share a cache, and an elements segment that
spans the whole file is read once per split. Reading a list-layout file in one split avoids that
(vortex.splitFiles=false); writing list elements in chunks that line up with the outer row chunks would fix it
for split reads too.

@ChristopherWirt

Copy link
Copy Markdown
Collaborator

this looks good

@ChristopherWirt ChristopherWirt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the cache wiring, ownership and existing Python precedent. No blocking correctness findings. The per-open cache safely isolates segment IDs, and both JNI opening paths install it. Non-blocking follow-ups: clarify that the 64 MiB budget counts segment lengths rather than retained backing allocations, and add focused coverage for the 64 KiB coalescing change; the added regression currently covers caching with coalescing disabled. No additional tests were run during this review.

@ChristopherWirt
ChristopherWirt merged commit 769938b into develop Nov 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[JNI] List layout scans read the same segments several times

2 participants