Skip to content

jni: split oversized Java reads into chunks instead of capping coalesce max size #104

Open

Description

@ChristopherWirt

Problem

JavaReadable::read_at rejects any read over i32::MAX bytes ("exceeds ByteBuffer limit"), because each read is one readFully upcall into a single Java ByteBuffer.

To keep merged reads under that limit, checked_coalesce_config (vortex-jni/src/io/read_at.rs) caps coalesceMaxSize at i32::MAX - (Alignment::MAX_UNTRUSTED - 1). This has two drawbacks:

  • Coupled to another crate. The headroom copies how FileSegmentSource (vortex-file/src/segments/source.rs) pads the coalescing window. If that padding changes, the cap is silently wrong.
  • Only covers merged reads. A single segment over 2 GiB still fails.

Proposal

Split oversized reads in JavaReadable::read_at into readFully upcalls of at most i32::MAX bytes each, every one over a slice of the same aligned buffer. JavaWritable already does this for writes (vortex-jni/src/io/write.rs, bytes.chunks(i32::MAX as usize)).

Then:

  • Remove the MAX_UNTRUSTED clamp and its i32::MAX test case.
  • Remove the "native code caps values near Integer.MAX_VALUE" caveat from the NativeReadable.coalesceMaxSize() Javadoc and java/vortex-spark/README.md.

Follow-up from #95.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions