You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
jni: split oversized Java reads into chunks instead of capping coalesce max size #104
JavaReadable::read_at rejects any read over i32::MAX bytes ("exceeds ByteBuffer limit"), because each read is one readFully upcall into a single Java ByteBuffer.
To keep merged reads under that limit, checked_coalesce_config (vortex-jni/src/io/read_at.rs) caps coalesceMaxSize at i32::MAX - (Alignment::MAX_UNTRUSTED - 1). This has two drawbacks:
Coupled to another crate. The headroom copies how FileSegmentSource (vortex-file/src/segments/source.rs) pads the coalescing window. If that padding changes, the cap is silently wrong.
Only covers merged reads. A single segment over 2 GiB still fails.
Proposal
Split oversized reads in JavaReadable::read_at into readFully upcalls of at most i32::MAX bytes each, every one over a slice of the same aligned buffer. JavaWritable already does this for writes (vortex-jni/src/io/write.rs, bytes.chunks(i32::MAX as usize)).
Then:
Remove the MAX_UNTRUSTED clamp and its i32::MAX test case.
Remove the "native code caps values near Integer.MAX_VALUE" caveat from the NativeReadable.coalesceMaxSize() Javadoc and java/vortex-spark/README.md.
Activity
feat(jni): make Java read coalescing configurable (#95)