You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 31283ee
Browse filesBrowse the repository at this point in the historyBrowse files
feat(scan): let a scan request bound the splits it decodes at once (#11127)
- Tracking Issue: #11124
A scan decodes `4 × available_parallelism()` splits at once. In Spark,
every task in an executor runs its own scan sized to the whole machine,
and each split holds the decoded chunks of every column it overlaps. On
wide, nested files that is GiBs of native memory per executor (#11123).
This adds a per-scan limit on how many splits are decoded at once.
| Layer | API |
|---|---|
| Rust | `ScanRequest::split_concurrency` →
`ScanBuilder::with_some_max_splits_in_flight` |
| JNI | `ScanOptions.splitConcurrency()` |
| Spark | `vortex.splitConcurrency` read option |
**Result:** a full scan of a 180 MB list-layout file, peak live native
bytes:
| Splits at once | Peak live | Time |
|---|---|---|
| 48 (default, 4 × 12 cores) | 3.65 GiB | 1.0 s |
| 12 | 1.18 GiB | 1.0 s |
---------
Signed-off-by: Christopher Wirt <[email protected]>
|`vortex.readConcurrency`|`0`| Maximum Hadoop read upcalls per file; `0` uses the native default. |
199
+
|`vortex.splitConcurrency`|`0`| Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing output. This does not cap total native memory or I/O read-ahead. |
199
200
|`vortex.workerThreads`|`4`| Background threads driving Vortex futures. JVM-wide, and set once per JVM by the first read or write. |
200
201
|`vortex.session.provider`| — | Fully-qualified name of a `VortexSessionProvider` supplying a custom session on the driver and on every executor. |
201
202
|`vortex.aggregatePushdown`|`true`| Answer `COUNT(*)` from file footers. |
0 commit comments