You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Java and Spark readers retain native file resources until Cleaner processing #11364
Closing or exhausting a Spark Vortex reader releases its Arrow stream and Hadoop readable, but the Java DataSource and Scan handles release native ownership only when their Cleaners run. Completed readers can therefore retain per-file native caches until Java GC even when no Arrow batches remain open.
The Cleaner-only handles and reader cleanup are still present on upstream develop at 2e1b05b84.
In local QC experiments, 30 complete one-leaf reads of a wide nested fixture retained 225.3 MiB of native input buffers after closing every reader; explicitly releasing the data source returned that accounting to zero. This establishes delayed reclamation, not a permanent leak or a production OOM claim. Details: QC issue #96.
Provide idempotent explicit close for source and scan handles and use it at EOF, early close, empty splits, footer reads and failures. Preserve returned partitions/batches and Cleaner fallback. Also release prefetched-but-unreturned partitions and make Arrow stream handoff exception-safe.
QC PR #103 implements this. Its unrelated list-view writer normalization is excluded from the upstream port. Session lifetime (#8075) and decode/output buffering (#11127, #11198) are separate concerns.
Activity
fix(spark): release native read resources deterministically (vortex-d…