Skip to content

Make JavaReadable's coalescing gap configurable from Java #94

Closed

Description

@ChristopherWirt

JavaReadable hard-codes its coalescing config to a 64 KiB gap and a 16 MiB maximum (vortex-jni/src/io/read_at.rs:92-98). Java readers, the Spark connector included, can't change it.

Why it matters. Files are laid out column by column, in runs of about 2 MB (BufferedStrategy in vortex-file/src/strategy.rs). So a scan that reads a few rows from many columns issues roughly one request per projected column buffer, and a 64 KiB gap rarely merges them. On object storage, each request is a GET.

Measured on real 476k-row files, reading the rows of one key (about 6 rows) with a 58-leaf projection, with the reads coalesced as JavaReadable does:

Gap Requests per file Bytes read per file
64 KiB (today) 120 25 MB
256 KiB 69 31 MB
1 MiB 34 49 MB
4 MiB 17 85 MB

With many GETs and in-region reads, 1 MiB is far cheaper. About 14 requests is the floor at any gap.

Proposal.

  • Take the gap, and optionally the maximum, from HadoopReadable / NativeReadable, and default to today's 64 KiB.
  • Expose them as read options, e.g. vortex.read.coalesce.distance and vortex.read.coalesce.maxSize, which VortexIo passes through.
  • The comment's concern stays valid: a wider gap can re-read segments the file's segment cache already holds. So the default stays as it is, and callers opt in.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions