You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 31ae92c
Browse filesBrowse the repository at this point in the historyBrowse files
fix(scan): keep capped JNI scans within Arrow lookahead
Drive capped Arrow streams directly instead of feeding the runtime result queue, which can retain one extra completed batch per core. Preserve the existing queue for uncapped scans. Add paused-consumer regressions with two background workers.
Saturate per-worker concurrency scaling before applying the absolute cap and add an overflow regression. Clarify that split concurrency reduces decoded memory rather than bounding total native memory or I/O read-ahead.
Signed-off-by: Christopher Wirt <[email protected]>
|`vortex.readConcurrency`|`0`| Maximum Hadoop read upcalls per file; `0` uses the native default. |
239
-
|`vortex.splitConcurrency`|`0`| Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Each split holds the decoded chunks of every column it reads, and every task in an executor runs its own scan, so lowering this bounds a task's native memory. |
239
+
|`vortex.splitConcurrency`|`0`| Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. It does not cap total native memory or I/O read-ahead. |
240
240
|`vortex.workerThreads`|`4`| Background threads driving Vortex futures. JVM-wide, and set once per JVM by the first read or write. |
241
241
|`vortex.session.provider`| — | Fully-qualified name of a `VortexSessionProvider` supplying a custom session on the driver and on every executor. |
242
242
|`vortex.aggregatePushdown`|`true`| Answer `COUNT(*)` from file footers. |
0 commit comments