You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make JavaReadable's coalescing gap configurable from Java #94
JavaReadable hard-codes its coalescing config to a 64 KiB gap and a 16 MiB maximum (vortex-jni/src/io/read_at.rs:92-98). Java readers, the Spark connector included, can't change it.
Why it matters. Files are laid out column by column, in runs of about 2 MB (BufferedStrategy in vortex-file/src/strategy.rs). So a scan that reads a few rows from many columns issues roughly one request per projected column buffer, and a 64 KiB gap rarely merges them. On object storage, each request is a GET.
Measured on real 476k-row files, reading the rows of one key (about 6 rows) with a 58-leaf projection, with the reads coalesced as JavaReadable does:
Gap
Requests per file
Bytes read per file
64 KiB (today)
120
25 MB
256 KiB
69
31 MB
1 MiB
34
49 MB
4 MiB
17
85 MB
With many GETs and in-region reads, 1 MiB is far cheaper. About 14 requests is the floor at any gap.
Proposal.
Take the gap, and optionally the maximum, from HadoopReadable / NativeReadable, and default to today's 64 KiB.
Expose them as read options, e.g. vortex.read.coalesce.distance and vortex.read.coalesce.maxSize, which VortexIo passes through.
The comment's concern stays valid: a wider gap can re-read segments the file's segment cache already holds. So the default stays as it is, and callers opt in.
Activity