Repository navigation
feat(jni): make Java read coalescing configurable - #11369
Merged
robert3005 merged 3 commits intoNov 10, 2026
Merged
robert3005 merged 3 commits into
robert3005 merged 3 commits into
Conversation
Signed-off-by: Christopher Wirt <[email protected]>
ChristopherWirt
added a commit
to quantcast/vortex
that referenced
this pull request
Nov 10, 2026
Sets the default Java read coalescing gap back to 1 MiB, from the 64 KiB #95 chose. - Upstream's port (vortex-data#11369) keeps 1 MiB, so this keeps our defaults the same as upstream's. - Readers that want 64 KiB can still set `vortex.read.coalesce.distance`. Tested: `:vortex-spark-4.0_2.13:test --tests 'dev.vortex.spark.io.*'` (28 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
robert3005
approved these changes
Nov 10, 2026
robert3005
enabled auto-merge (squash)
November 10, 2026 14:09
Signed-off-by: Christopher Wirt <[email protected]>
auto-merge was automatically disabled
November 10, 2026 15:31
Head branch was pushed to by a user without write access
robert3005
enabled auto-merge (squash)
November 10, 2026 15:36
Contributor
|
I think you still have spotless issues here |
Signed-off-by: Christopher Wirt <[email protected]>
auto-merge was automatically disabled
November 10, 2026 16:34
Head branch was pushed to by a user without write access
robert3005
enabled auto-merge (squash)
November 10, 2026 16:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Java-backed readers always use
CoalesceConfig::object_storage()(a 1 MiB gap, a 16 MiB maximum merged span). Callers can't change it.The right values depend on the storage underneath. Local SSD, spinning disk and object stores each trade request count against extra bytes read differently. This PR lets Java and Spark callers set them, keeping the current 1 MiB / 16 MiB as the defaults.
Changes
NativeReadable.coalesceDistance()andcoalesceMaxSize()methods. Native code reads them once, when it registers a scan or metadata/statistics source.vortex.read.coalesce.distanceandvortex.read.coalesce.maxSize, in bytes. They're case-insensitive and validated before broadcast.API changes
Two default interface methods and two Spark read options. Existing implementations keep today's behaviour.
Validation
Maintainer action: please add
changelog/feature. I don't have permission to set labels.No tests, builds or lint were run on this branch. The same change was tested in our fork. No benchmarks were run.
AI assistance
AI tools helped write this change and its description. Christopher Wirt is the responsible contributor.