Skip to content

[Spark] Release the exported Arrow batch if writeBatch throws before native takes it #11

Closed

Description

@ChristopherWirt

VortexDataWriter hands each batch to VortexWriter.writeBatch inside a try-with-resources ArrowArray. If writeBatch throws before the native side takes ownership, every buffer of that batch leaks.

Why

  • ArrowArray.close() only frees the struct and never calls release(), so the buffers the export retained are never released.
  • Two paths throw before native code takes ownership:
    • Preconditions.checkState(!closed.get(), "writer already closed") in VortexWriter.writeBatch;
    • writer_ptr <= 0 in native writeBatch.
  • Once native code is entered there's no leak: FFI_ArrowArray::from_raw takes ownership straight away.

Impact

Latent: no current caller can trigger it.

  • VortexDataWriter only flushes while vortexWriter is non-null, and nulls it as soon as the writer is finished or closed.
  • The builder rejects a zero native pointer.

A future caller that writes after close(), or reorders cleanup, would leak the batch.

Where

Fix

Release the array if writeBatch throws before native code takes it: call arrowArray.release() in a catch around writeBatch, or check the writer's state before exporting.

Activity

  1. ChristopherWirt commented on Nov 2, 2026

    @ChristopherWirt
    CollaboratorAuthor

    PR #35 supersedes #22 as the fix for this issue. The FileFormat Spark writer uses the Java-root batch API, which releases an exported array in a finally block if ownership was not transferred to native code. PR #35 now includes the closing reference. This issue remains open until #35 merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions