Skip to content

Commit 11e695f

Browse files
docs(scan): clarify split concurrency memory limits
Signed-off-by: Christopher Wirt <[email protected]>
1 parent 31ae92c commit 11e695f

3 files changed

Lines changed: 6 additions & 4 deletions

File tree

‎docs/user-guide/spark.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -236,7 +236,7 @@ depends on, on the driver and executor classpath, for example
236236
| Option | Default | Description |
237237
|----------------------------|---------|-------------|
238238
| `vortex.readConcurrency` | `0` | Maximum Hadoop read upcalls per file; `0` uses the native default. |
239-
| `vortex.splitConcurrency` | `0` | Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. It does not cap total native memory or I/O read-ahead. |
239+
| `vortex.splitConcurrency` | `0` | Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing output. This does not cap total native memory or I/O read-ahead. |
240240
| `vortex.workerThreads` | `4` | Background threads driving Vortex futures. JVM-wide, and set once per JVM by the first read or write. |
241241
| `vortex.session.provider` | — | Fully-qualified name of a `VortexSessionProvider` supplying a custom session on the driver and on every executor. |
242242
| `vortex.aggregatePushdown` | `true` | Answer `COUNT(*)` from file footers. |

‎java/vortex-jni/src/main/java/dev/vortex/api/ScanOptions.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,9 @@ default boolean ordered() {
6565
* to Arrow ahead of the caller. Absent keeps the native default, which scales with the machine's cores.
6666
*
6767
* <p>Each split being decoded holds the decoded chunks of every column it overlaps, so a scan running alongside
68-
* others, such as one Spark task of several in an executor, can set this to bound its native memory. It does not
69-
* bound I/O: segment reads for later splits may still be coalesced ahead of them.
68+
* others, such as one Spark task of several in an executor, can lower this to reduce memory held by concurrent
69+
* decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split
70+
* while preparing output. This does not cap total native memory or I/O read-ahead.
7071
*/
7172
OptionalInt splitConcurrency();
7273

‎java/vortex-spark/common/src/main/java/dev/vortex/spark/io/VortexIo.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ public final class VortexIo implements Serializable {
3434
* Option bounding how many splits of a file the native scan reads at once, in all. Each split in flight holds the
3535
* decoded chunks of every column it overlaps, and every task in an executor runs its own scan, which by default
3636
* keeps several splits in flight per core. Lowering this reduces memory held by concurrent split decoding and Arrow
37-
* conversion; it does not cap total native memory or I/O read-ahead. Zero keeps the native default.
37+
* conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing
38+
* output. This does not cap total native memory or I/O read-ahead. Zero keeps the native default.
3839
*/
3940
public static final String SPLIT_CONCURRENCY_OPTION = "vortex.splitConcurrency";
4041

0 commit comments

Comments
 (0)