Repository navigation
perf(io): use local coalescing limits for local DataFusion reads - #11338
Draft
joseph-isaacs wants to merge 1 commit into
Draft
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
Limit local read gaps to 8 KiB and select the existing 4 MiB local request cap for the default DataFusion file reader. Remote stores keep their existing coalescing policy and custom reader factories remain authoritative. Signed-off-by: Joe Isaacs <[email protected]>
joseph-isaacs
changed the base branch from
ji/clickbench-v2-20261109-01-io-wakeups
to
develop
November 9, 2026 11:56
joseph-isaacs
added
the
action/bench-sql
ChatGPT Codex Connector
joseph-isaacs
changed the base branch from
develop
to
ji/clickbench-v2-20261109-01-io-wakeups
November 9, 2026 11:56
github-actions
Bot
removed
the
action/bench-sql
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.061x ➖, 2↑ 3↓)
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.923x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.023x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.048x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.023x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.956x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
ChristopherWirt
added a commit
to quantcast/vortex
that referenced
this pull request
Nov 9, 2026
## Summary Fixes #94. Java-backed scans hard-code a 64 KiB coalescing gap and a 16 MiB max merged read. Callers can't trade extra bytes read for fewer object-storage requests. This PR makes both limits configurable and keeps the current defaults. ## Changes - **`NativeReadable`:** new default methods `coalesceDistance()` and `coalesceMaxSize()`. Native code reads them once when each source is registered, for scans and metadata/statistics reads. - **Spark:** new options `vortex.read.coalesce.distance` and `vortex.read.coalesce.maxSize`, in bytes and case-insensitive. They flow through `VortexIo` to `HadoopReadable` and are validated on the driver before broadcast. - **Native validation:** the distance must be `>= 0` and the max size `> 0`. Max sizes near `Integer.MAX_VALUE` are capped to leave room for alignment padding within the `ByteBuffer` limit. Chunking oversized reads instead is tracked in #104. - **README:** documents the options and the trade-off between request count and bytes read. Upstream has no equivalent Java setting. The closest work is vortex-data#11338, which changes DataFusion coalescing. ## Validation - `cargo clippy -p vortex-jni --all-targets --all-features -- -D warnings` - `cargo nextest run -p vortex-jni`: 102 passed - `:vortex-jni:test --tests NativeIOBridgeTest`: passed - `:vortex-spark-4.0_2.13:test --tests 'dev.vortex.spark.io.*'`: passed - Spotless; `git diff --check` Spark 3.5 modules and benchmarks were not run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Christopher Wirt <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack dependency: #11334. This PR targets that branch so its diff contains only this change.
Summary
Local reads currently inherit a coalescing policy that can include large gaps between requested segments. Use the local-file policy in DataFusion's default local reader and narrow the shared local gap limit to reduce unnecessary bytes copied between segments.
Stack base:
ji/clickbench-v2-20261109-01-io-wakeups(da10665e5b0f) (s01). This branch changes local IO configuration; its scope extends beyond the V2 benchmark path.Changes
CoalesceConfig::file()from a 1 MiB gap to 8 KiB, retaining its 4 MiB coalescing limit; larger individual reads are not split. This also affects other users of that shared local-file default, including standard and CUDA file readers.DefaultVortexReaderFactory::with_coalesce_configand select the local policy for DataFusion's default local-filesystem reader. Remote-store defaults and explicitly supplied reader factories retain their policies.Benchmark evidence
Warm ClickBench timings against this branch's exact dependency parent. Query IDs are zero-based; these are individual query gains, not whole-suite or additive gains.
24 fresh adjacent process pairs in balanced order; 100 executions/query, first 20 discarded; 8 interspersed identical-binary A/A pairs. Effects use mean paired log ratios of process warm medians. Displayed times are medians across processes, so their ratio need not exactly equal the paired estimate. Intervals are pointwise 99% bootstrap intervals. Each listed gain passes both the >1% interval rule and the exact paired sign test with Holm 1% correction across all 76 planned latency directions. Both execution orders and chronological sensitivity were reviewed; no post-warmup observations were excluded or controls subtracted. The sign test concerns how often a paired improvement exceeds 1%; the displayed interval estimates the mean log effect.
DataFusion Q36 and Q39 did not pass the corrected gain rule and are not claimed as wins.
32-core aarch64 host, full 100-file partitioned ClickBench fixture, Vortex V2, ordinary local reads with no mmap, 32 workers/partitions/threads, DuckDB
--reuse. DataFusion 55.2 retainsenable_topk_aggregation=true. Compiler, Cargo feature union, profile and runtime environment are fixed across parent/candidate. Executable and library hashes, canonical SQL and fixture identities are recorded.Full43 screening
The complete series screen finished 168/168 processes, with no failures or overlaps and successful source/SQL/binary/fixture checks. Each source/engine comparison used four balanced exact-parent process pairs and two matching identical-binary A/A pairs, with 10 executions/query and the first 3 discarded. This screen is descriptive: no new gain, equivalence, or regression-free claim is inferred, and no observations or controls were subtracted.
Focused Q38/Q41 gains remain valid in both engines. The all43 screen adds a coherent DuckDB Q9 concern: medians 1.64% and means 1.54% slower, with all four pairs adverse. DataFusion Q2 has repeated candidate tail events; its 35.53% mean increase must not be described as typical latency. Broad sums and CPU favor the candidate, but do not erase those endpoints. Keep the draft's focused claims and disclose these unresolved landing concerns.
Validation
read_at::tests::test_coalesce_config.--locked, therelease_debugprofile,profile.release_debug.debug=1, andRUSTFLAGS="-C target-cpu=native -C force-frame-pointers=yes". Rust changes were formatted with pinned nightly; patch whitespace checks passed. Workspace-wide clippy was not run; CI status is separate.Measured local source commit:
614cc21d766c12fce6c3564c7093a65b2ae5a827; full source tree:3c43dac2fee0f91008d19d9f0be7bfe3b6cc8a80. Measured parent:a2765d42438062db73d431ef6d296930e69f5b28.PR source commit:
922926672e62282ae0b95817fbefd0f1b5db63e2; mapped parent:da10665e5b0fad748a1e0e6550cc19880edd0317. The complete source tree and mapped parent graph match the measured source. GitHub regenerated commit metadata/timestamps, so the measured local commit and executable hashes remain the benchmark identities. The complete remote Git object, original message and author/committer signoffs were verified.Measurements use the develop snapshot
b2f8d0f946e003d3e463d1447ed357ed88990686. Develop advanced during the campaign; these results retain the recorded source and binary identities.Repository SQL benchmark
The SQL workflow completed 8/8 suites on this exact PR head. 2 suite(s) required one failed-jobs-only retry after EC2 capacity failures before checkout; the head stayed unchanged. The ClickBench bot verdict is No clear signal (low confidence).
The preset covers V2 ClickBench, TPC-H, TPC-DS and FineWeb, plus a legacy PolarSignals control. These cloud runs use suite-specific historical develop baselines and
FLAT_LAYOUT_INLINE_ARRAY_NODE=1; they are separate from the local exact-parent comparisons and cannot isolate this stack's incremental effect. Ordinary draft CI was skipped; SQL benchmark success is not an overall CI pass.