perf(jni): bound completed scan outputs with background prefetch - #78

Problem

• A JNI scan hands batches to Java through block_on_stream_thread_safe.
• That bridge queues one completed batch per CPU core, but each JNI stream has only one consumer.
• On wide or nested Spark scans those queued Arrow batches add up, so peak memory grows with host core count.

Fix

• Add CurrentThreadRuntime::block_on_stream_prefetch to vortex-io. It is a single-consumer bridge that:
  • runs the stream in a background task, so native work still overlaps Java consumption;
  • keeps at most one completed item ahead of the consumer;
  • cancels the producer on drop and re-raises its panic at end of stream.
• Use it for the default JNI scan path. Scans with an explicit splitConcurrency keep the direct iterator.

The new API is additive. JNI is its only caller, and other bindings are unchanged.

Results

Local Spark measurements, baseline QC develop 29348ee39. Values are medians of macOS peak physical footprint; n is runs per variant.

• Batch sizes, compression and decode concurrency are unchanged, and the rewrite output was byte-identical in size and row count.
• Timing was too noisy on a shared host to claim a speedup or rule out a regression.
• Only one Spark task with four native workers was measured.
• The benchmark harness is not part of this PR.

Notes for reviewers

• Once a batch is queued, the producer stops pulling the next one until Java takes it. Conversions already running keep going, but with a single native worker there is little overlap.
• This PR does not change how a native panic during a scan is handled. It is still re-raised inside the Arrow C stream callback, which aborts the JVM. A follow-up could surface it as an ArrowError instead.

Tests

• vortex-io: one-item-ahead bound and driver panic propagation.
• vortex-jni: the default scan keeps one completed output ahead.
• Not run yet: no tests, clippy or fmt have been run locally on this branch.

🤖 Generated with Claude Code