You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Let a scan bound the splits it decodes at once #63
A scan keeps 4 × available_parallelism() splits in flight (RepeatedScan::execute_stream), and nothing in ScanRequest, JNI or Spark can lower that.
In Spark, every task in an executor runs its own scan, and each sizes itself to the container's cores. A 4-core executor running 4 tasks therefore decodes 4 × 16 = 64 splits at once. Each split holds the decoded chunks of every column it overlaps. On wide, nested files this is GiBs of native memory per executor, and executors are OOM-killed (#61).
What one file shows
A full scan of a 180 MB list-layout file, measuring live native bytes:
splits in flight
peak live
time
48 (default, 4 × 12 cores)
3.65 GiB
1.0 s
12
1.18 GiB
1.0 s
Lowering concurrency costs nothing in time on this file and cuts memory by two thirds.
Proposal
An absolute bound on splits being decoded, per scan:
ScanRequest::split_concurrency: Option<NonZeroUsize>, honoured by both layout data sources;
ScanOptions.splitConcurrency() in JNI;
a vortex.splitConcurrency Spark read option.
This is a mitigation, not the fix for #61. The root cause there is list-layout scans splitting into tiny ranges that each decode whole column chunks.
Activity