Skip to content

Let a scan bound the splits it decodes at once #63

Closed

Description

@ChristopherWirt

Problem

A scan keeps 4 × available_parallelism() splits in flight (RepeatedScan::execute_stream), and nothing in ScanRequest, JNI or Spark can lower that.

In Spark, every task in an executor runs its own scan, and each sizes itself to the container's cores. A 4-core executor running 4 tasks therefore decodes 4 × 16 = 64 splits at once. Each split holds the decoded chunks of every column it overlaps. On wide, nested files this is GiBs of native memory per executor, and executors are OOM-killed (#61).

What one file shows

A full scan of a 180 MB list-layout file, measuring live native bytes:

splits in flight peak live time
48 (default, 4 × 12 cores) 3.65 GiB 1.0 s
12 1.18 GiB 1.0 s

Lowering concurrency costs nothing in time on this file and cuts memory by two thirds.

Proposal

An absolute bound on splits being decoded, per scan:

  • ScanRequest::split_concurrency: Option<NonZeroUsize>, honoured by both layout data sources;
  • ScanOptions.splitConcurrency() in JNI;
  • a vortex.splitConcurrency Spark read option.

This is a mitigation, not the fix for #61. The root cause there is list-layout scans splitting into tiny ranges that each decode whole column chunks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions