You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1aa2ad7
Browse filesBrowse the repository at this point in the historyBrowse files
feat(scan): let a scan request bound the splits it decodes at once (#64)
Closes#63. Helps with #61, but doesn't fix it.
## Problem
- A scan decodes `4 × available_parallelism()` splits at once.
- In Spark, every task in an executor runs its own scan sized to the
whole machine.
- Each split holds the decoded chunks of every column it overlaps.
- On wide, nested files that's GiBs of native memory per executor (#61).
## Change
An absolute bound on splits decoded at once, settable per scan:
| Layer | API |
|---|---|
| Rust | `ScanRequest::split_concurrency` →
`ScanBuilder::with_some_max_splits_in_flight` |
| JNI | `ScanOptions.splitConcurrency()` |
| Spark | `vortex.splitConcurrency` read option |
- It's a cap kept apart from the per-worker `with_concurrency`. A later
`with_concurrency` (as the DataFusion opener calls) leaves it in place,
and `concurrency()` keeps its meaning.
- JNI also bounds the Arrow conversion it runs ahead of Java
(`buffered(worker_count)`) by the same value.
- It doesn't bound I/O: segment reads for later splits may still be
coalesced ahead of them. The docs say so.
## Effect
A full scan of a 180 MB list-layout file, peak live native bytes:
| splits at once | peak live | time |
|---|---|---|
| 48 (default, 4 × 12 cores) | 3.65 GiB | 1.0 s |
| 12 | 1.18 GiB | 1.0 s |
## Tests
- **Rust:** a scan of 64 one-row splits decodes exactly 1, and exactly
2, at once with the bound set, against 48 by default. A later
`with_concurrency` keeps the bound.
- **JNI:** `ScanOptions` validation, and a scan with the bound reads
every row.
- **Spark:** option parsing, validation and serialisation, plus V1 and
V2 reads with the option set.
- **Suites:** Rust (vortex-layout, vortex-scan, vortex-jni,
vortex-file), JNI, Spark 3.5 and Spark 4.0 all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Christopher Wirt <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
|`vortex.readConcurrency`|`0`| Maximum Hadoop read upcalls per file; `0` uses the native default. |
244
+
|`vortex.splitConcurrency`|`0`| Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing output. This does not cap total native memory or I/O read-ahead. |
244
245
|`vortex.workerThreads`|`4`| Background threads driving Vortex futures. JVM-wide, and set once per JVM by the first read or write. |
245
246
|`vortex.session.provider`| — | Fully-qualified name of a `VortexSessionProvider` supplying a custom session on the driver and on every executor. |
246
247
|`vortex.aggregatePushdown`|`true`| Answer `COUNT(*)` from file footers. |
0 commit comments