Skip to content

fix(java): release borrowed batch schemas in address-based callers - #84

Open
ChristopherWirt wants to merge 2 commits into
developfrom
cwirt/jni-batch-schema-release
Open

ChristopherWirt wants to merge 2 commits into
developfrom
cwirt/jni-batch-schema-release

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Address-based writeBatch callers export an Arrow schema for each batch but close its outer struct without releasing the export. Native import takes the array and only borrows the schema; Arrow 19's ArrowSchema.close() does not invoke its release callback. This leaves the exported schema's child allocations alive.

Closes #53.

Upstream comparison

The issue remains in both repositories, reviewed on 2026-11-04:

The managed writeBatch(VectorSchemaRoot, BufferAllocator) overload exports no per-batch schema and already cleans up its temporary array. Current Spark production writes use that overload. #18's writer-creation cleanup does not cover these per-batch exports.

Changes

  • Release schema exports in finally blocks at eleven leaking call sites across JNI and Spark fixtures. Also release arrays that were not consumed if export/import fails, before closing their outer structs.
  • Document array transfer versus schema borrowing on both address-based Java APIs. Schemas remain reusable across calls; native ownership behavior is unchanged.
  • Give the existing flat and nested round-trip tests isolated child allocators and assert zero remaining allocations after cleanup.
  • Add flat/nested ownership cases that reuse one schema for three batches, on success and native column-count rejection. Check that import consumes the array, preserves the borrowed schema, and leaves no allocator allocations after finish and cleanup.

Validation

Completed independent reuse, maintainability, efficiency and correctness reviews. No substantive defects were found; applied the repository formatter to the changed Java files and reviewed the final diff.

  • Scoped Spotless Java formatting completed for all nine changed files.
  • Java compilation with ErrorProne and -Werror passed.
  • Relevant JNI suites: 62 passed, one existing disabled test skipped (JNIWriterTest, NativeIOBridgeTest, ListContainsFilterTest, SpatialTypesTest, TestMinimal, IndependentBuffersTest).
  • Spark 4.0 / Scala 2.13: 56 passed across VortexNestedSchemaResolutionTest and VortexReadTypeParityTest.
  • JNI Javadoc generation and git diff --check passed.
  • Regression sensitivity confirmed: temporarily removing only the schema release in testWriteBatch made the address-based case fail with 294 bytes still allocated; the managed-root case passed. Restoring the release made both cases pass again.

Tests used a cached JNI build with identical writer/runtime sources and Cargo dependency files; its only Rust source delta is PR #82's unrelated file-listing/deletion utilities. No Rust code changed in this PR. Spark 3.5 variants and the full repository suite were not run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JNI address-based writer callers leak exported batch schemas

1 participant