Repository navigation
feat(jni): make Java read coalescing configurable - #95
Merged
Merged
Conversation
Signed-off-by: Christopher Wirt <[email protected]>
- Parse Spark read options once into a ReadSettings record shared by create/broadcast - Drop HadoopReadable's duplicate coalescing checks; native registration validates them - Give non-positive maxSize a single '> 0' error and bump VortexIo serialVersionUID - Restore the rationale for the 64 KiB default gap - Dedupe test dispatch and use the named default constants in tests Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Christopher Wirt <[email protected]>
a392n328
approved these changes
Nov 9, 2026
This was referenced Nov 9, 2026
ChristopherWirt
added a commit
that referenced
this pull request
Nov 10, 2026
Sets the default Java read coalescing gap back to 1 MiB, from the 64 KiB #95 chose. - Upstream's port (vortex-data#11369) keeps 1 MiB, so this keeps our defaults the same as upstream's. - Readers that want 64 KiB can still set `vortex.read.coalesce.distance`. Tested: `:vortex-spark-4.0_2.13:test --tests 'dev.vortex.spark.io.*'` (28 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #94.
Java-backed scans hard-code a 64 KiB coalescing gap and a 16 MiB max merged read. Callers can't trade extra bytes read for fewer object-storage requests.
This PR makes both limits configurable and keeps the current defaults.
Changes
NativeReadable: new default methodscoalesceDistance()andcoalesceMaxSize(). Native code reads them once when each source is registered, for scans and metadata/statistics reads.vortex.read.coalesce.distanceandvortex.read.coalesce.maxSize, in bytes and case-insensitive. They flow throughVortexIotoHadoopReadableand are validated on the driver before broadcast.>= 0and the max size> 0. Max sizes nearInteger.MAX_VALUEare capped to leave room for alignment padding within theByteBufferlimit. Chunking oversized reads instead is tracked in jni: split oversized Java reads into chunks instead of capping coalesce max size #104.Upstream has no equivalent Java setting. The closest work is vortex-data#11338, which changes DataFusion coalescing.
Validation
cargo clippy -p vortex-jni --all-targets --all-features -- -D warningscargo nextest run -p vortex-jni: 102 passed:vortex-jni:test --tests NativeIOBridgeTest: passed:vortex-spark-4.0_2.13:test --tests 'dev.vortex.spark.io.*': passedgit diff --checkSpark 3.5 modules and benchmarks were not run.
🤖 Generated with Claude Code