Spark reader close retains native file caches until Java GC #96

Closing a Spark Vortex reader releases its Arrow stream and Hadoop readable, but leaves the native data source and its populated file cache alive until Java GC/Cleaner processing. Successive completed readers can therefore accumulate native memory even when no batches remain open.

Reproduced on QC develop at aefef5f089989dae140083ba47a476a7113ff337. Current upstream develop still has Cleaner-only DataSource/Scan handles and clears these references without explicit native release in VortexPartitionReader.close().

Evidence

A 300,000-row nested fixture with 1,007 leaf fields, read through the actual VortexPartitionReader with a one-leaf projection:

• Thirty sequential complete reads retained 225.3 MiB of native input-buffer backing allocations after every reader had closed. Arrow allocator accounting was zero at each post-close snapshot, and no GC occurred during the measured loop.
• Invoking the already-registered data-source Cleaner action after close returned retained input bytes to zero. Releasing only the exhausted scan did not.
• Disabling only the per-file segment cache also returned retained input bytes to zero, with the same input-allocation count and total bytes as the original run.
• A final GC reclaimed everything. This proves delayed reclamation, not a permanent leak.
• Thirty first-batch-only reads showed the same mechanism, retaining 151.3 MiB until GC.

A matched local Spark scan over 24 fixture-file paths, projecting one nested string field, also reproduces it. Five warm-up actions followed by eight measured actions, local[4], G1, and the same native diagnostic library on both sides:

Every action returned 7,200,000 rows and the expected byte checksum, independently derived from the Parquet input. Timings were similar after warming. These are local dev-library experiments; process RSS includes Java heap and other native memory, and does not establish production OOM behavior.

Reproduction outline

1. Write a wide nested file and read one leaf through VortexPartitionReader.
2. Fully consume it, close the reader, and repeat using the same JVM/session without forcing GC.
3. Track native input-buffer allocation/deallocation and data-source creation/destruction separately from Arrow allocations and process RSS.
4. Compare against explicit invocation of the existing data-source release action, then against the original path after GC.

Expected behavior

Provide idempotent explicit close for native data-source and scan handles and use it from Spark reader/footer lifecycles. Release owned resources at EOF, early close, empty splits, and failure, while retaining resources still needed by returned partitions or batches. A partition prefetched by Scan.hasNext() but never returned by next() must also be released. Keep Cleaner as a fallback and shared session caches at their intended lifetime.

Upstream and existing QC coverage checked

No matching issue or PR was found in open/closed upstream or QC searches for Cleaner/AutoCloseable, data-source close, or reader memory retention. Related work has distinct scope:

• upstream #8075: session lifetime/crashes and cache configuration; it proposes session close, not reader-owned source/cache cleanup.
• upstream #11123, #11124, #11127: tiny splits and active decode concurrency.
• upstream #11197, #11198: completed Arrow output lookahead.
• QC #66: adds the per-open-file segment cache and reduces rereads; it does not close the Java-owned native source deterministically.
• QC #94, #95: coalescing configuration, separate from handle lifetime.

A local cleanup implementation and focused tests are ready for complete-review before a linked PR.