You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Spark] Release the exported Arrow batch if writeBatch throws before native takes it #11
VortexDataWriter hands each batch to VortexWriter.writeBatch inside a try-with-resources ArrowArray. If writeBatch throws before the native side takes ownership, every buffer of that batch leaks.
Why
ArrowArray.close() only frees the struct and never calls release(), so the buffers the export retained are never released.
Two paths throw before native code takes ownership:
Preconditions.checkState(!closed.get(), "writer already closed") in VortexWriter.writeBatch;
writer_ptr <= 0 in native writeBatch.
Once native code is entered there's no leak: FFI_ArrowArray::from_raw takes ownership straight away.
Impact
Latent: no current caller can trigger it.
VortexDataWriter only flushes while vortexWriter is non-null, and nulls it as soon as the writer is finished or closed.
The builder rejects a zero native pointer.
A future caller that writes after close(), or reorders cleanup, would leak the batch.
Release the array if writeBatch throws before native code takes it: call arrowArray.release() in a catch around writeBatch, or check the writer's state before exporting.
PR #35 supersedes #22 as the fix for this issue. The FileFormat Spark writer uses the Java-root batch API, which releases an exported array in a finally block if ownership was not transferred to native code. PR #35 now includes the closing reference. This issue remains open until #35 merges.
Activity
ChristopherWirt commentedon Nov 2, 2026
PR #35 supersedes #22 as the fix for this issue. The FileFormat Spark writer uses the Java-root batch API, which releases an exported array in a finally block if ownership was not transferred to native code. PR #35 now includes the closing reference. This issue remains open until #35 merges.