Skip to content

[JNI] Scans queue a completed Arrow batch per CPU core ahead of Java #11197

Closed

Description

@ChristopherWirt

Problem or motivation

A JNI scan hands Arrow batches to Java through CurrentThreadRuntime::block_on_stream_thread_safe (NativePartition.scanArrow).

  • That bridge queues up to one completed item per CPU core (block_on_stream_thread_safe: increase channel size to number of cores #9116). This helps DuckDB, where many threads pull from one stream.
  • Each JNI stream has a single consumer, so the queue only holds finished batches that Java hasn't asked for yet.
  • On wide or nested Spark scans those queued batches add up, so peak memory grows with the host's core count.

Local Spark measurements, medians of macOS peak physical footprint, one task with four native workers:

Workload Runs Peak GiB, current → one batch ahead Change
Flat-layout read 3 1.46 → 1.05 −28%
List-layout read 2 3.25 → 2.22 −32%
Ordered Vortex rewrite 2 1.86 → 1.43 −23%

Proposed solution

Add a single-consumer bridge, CurrentThreadRuntime::block_on_stream_prefetch, and use it for JNI scans.

  • It drives the stream in a background task, so native work still overlaps Java consumption.
  • It keeps at most one completed item ahead of the consumer.
  • It cancels the producer on drop and re-raises a producer panic at end of stream.

block_on_stream_thread_safe and its other callers stay as they are.

Additional context

  • Batch sizes, compression and decode concurrency don't change. The rewrite's output was identical in size and row count.
  • Timing was too noisy on a shared host to claim a speedup or rule out a regression.
  • This complements Let a scan limit how many splits it decodes at once #11124, which caps how many splits a scan decodes at once. That bounds decoding, and this bounds the finished batches waiting for Java.

I have a fix ready.

Activity

  1. robert3005 commented on Nov 11, 2026

    @robert3005
    Contributor

    fixed in #11198

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions