perf(jni): bound completed scan outputs with background prefetch - #11198

Summary

• Tracking Issue: [JNI] Scans queue a completed Arrow batch per CPU core ahead of Java #11197

A JNI scan hands batches to Java through block_on_stream_thread_safe. That bridge queues one completed batch per CPU core, but each JNI stream has only one consumer. On wide or nested Spark scans those queued Arrow batches add up, so peak memory grows with the host's core count.

This adds a single-consumer bridge that keeps one completed batch ahead of Java, and uses it for JNI scans.

Result: local Spark measurements, medians of macOS peak physical footprint:

• Batch sizes, compression and decode concurrency don't change. The rewrite's output was identical in size and row count.
• Timing was too noisy on a shared host to claim a speedup or rule out a regression. Only one Spark task with four native workers was measured.

Changes

• vortex-io: CurrentThreadRuntime::block_on_stream_prefetch, returning a PrefetchIterator.
  • It drives the stream in a background task, so native work still overlaps Java consumption.
  • It keeps at most one completed item ahead of the consumer, and skips entering the executor when an item is already waiting.
  • Dropping it cancels the producer. A producer panic is re-raised at end of stream.
• JNI: NativePartition.scanArrow uses it instead of block_on_stream_thread_safe. Other callers, including DuckDB, are unchanged.

Notes for reviewers

• Once a batch is queued, the producer stops pulling the next one until Java takes it. Conversions already in flight keep running, but with a single native worker there is little overlap.
• A native panic during a scan is still re-raised inside the Arrow C stream callback, which aborts the JVM. This PR doesn't change that.

Tests

• vortex-io: the one-item-ahead bound, and propagation of a driver panic.

On the fork (quantcast#78, same change), the vortex-io change is identical. The fork's JNI side goes through a scan_iterator helper from #11127 and has a JNI test for it. develop doesn't have that helper yet, so this branch changes the block_on_stream_thread_safe call directly and leaves out the JNI test. Only git diff --check has been run on the port.

API Changes

• New public CurrentThreadRuntime::block_on_stream_prefetch and PrefetchIterator in vortex-io. Additive only.

AI assistance

Disclosed per the AI policy: AI tools helped write and review this change. Claude Code prepared this upstream port, issue and PR description at my request. I am the responsible contributor.

🤖 Generated with Claude Code