Skip to content

Commit 8dbcdaf

Browse files
fix(jni): retain Java Arrow buffers independently (#49)
## Summary Fixes #46. Provides a zero-copy alternative to the batch-copy workaround in #47. A retained validity bitmap or column slice currently keeps an entire imported Arrow batch alive through its shared C Data Interface release callback. The new Java-root write path retains each Java allocation independently, so the writer releases unrelated columns as soon as it no longer needs them. Spark uses this path by default. In the nested-string regression workload (16 batches of 8,192 rows), memory retained before `finish` dropped from about **65 MiB to 8.1 MiB**, with all allocations released at finish. These are synthetic results; the production 615-column workload has not been rerun. ## Changes - Add `VortexWriter.writeBatch(VectorSchemaRoot, BufferAllocator)`. It checks the batch schema, exports the array, and imports it using the writer's stored schema, avoiding a schema export for every batch. - Retain Java Arrow reference managers independently and replace the Rust buffers' owners without copying their data. The last Rust reference releases its Java allocation, including from native worker threads. - Preserve Arrow's existing alignment copies for unaligned input. Track the original exported addresses so missing Java owners still fail import rather than silently keeping whole batches alive. - Switch the Spark V2 writer to the new path. The address-based API remains compatible and keeps the producer's shared release callback; its callers must use the Java-root overload to get independent allocation lifetimes. Dictionary-encoded roots still require the address-based API. - Serialize both write methods with `finish` to prevent access to a freed native writer, and release the temporary schema used when creating a writer. Buffers sharing one underlying Java allocation still share that allocation's lifetime. ## Testing - `IndependentBuffersTest`: memory retained before finish and full release afterward; partial-import and sink-failure cleanup; empty batches and string views; unaligned integer buffers; schema mismatch rejection; concurrent write/finish stress for both APIs (16 iterations each). - `JNIWriterTest`: both APIs round-trip strings and nested maps. **18 JNI tests pass; one existing test is disabled.** - Rust ownership tests: unchanged buffer addresses, independent column release, sliced validity offsets, and release/global-reference cleanup from a native thread. **3 pass.** - Spark 2.13 `VortexDataSourceWriteTest`: **11 pass.** - `cargo clippy -p vortex-jni --all-targets --all-features -- -D warnings`, pinned Rust formatting, Java formatting checks, and patch whitespace checks pass. - Four independent reviews covered reuse, maintainability, efficiency and correctness; confirmed findings were fixed and tested. Known separately: the existing Vortex string-view concatenation path reads metadata from null slots. The view round-trip fixture initializes those slots; this PR does not fix that issue. Checked locally against QC `develop`. I used AI tools to help write and review this change. Signed-off-by: Christopher Wirt <[email protected]>
1 parent 7ad32dd commit 8dbcdaf

12 files changed

Lines changed: 804 additions & 37 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎java/vortex-jni/src/main/java/dev/vortex/api/VortexWriter.java‎

Lines changed: 43 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,19 @@
66
import com.google.common.base.Preconditions;
77
import dev.vortex.VortexCleaner;
88
import dev.vortex.io.NativeWritable;
9+
import dev.vortex.jni.NativeBuffer;
910
import dev.vortex.jni.NativeWriter;
1011
import java.io.Closeable;
1112
import java.io.IOException;
1213
import java.util.HashMap;
1314
import java.util.Map;
1415
import java.util.Objects;
1516
import java.util.concurrent.atomic.AtomicBoolean;
17+
import org.apache.arrow.c.ArrowArray;
1618
import org.apache.arrow.c.ArrowSchema;
1719
import org.apache.arrow.c.Data;
1820
import org.apache.arrow.memory.BufferAllocator;
21+
import org.apache.arrow.vector.VectorSchemaRoot;
1922
import org.apache.arrow.vector.types.pojo.Schema;
2023

2124
/**
@@ -24,22 +27,25 @@
2427
* <p>Open one with {@link #builder(Session, String, Schema, BufferAllocator)} to write to a URI, or
2528
* {@link #builder(Session, NativeWritable, Schema, BufferAllocator)} to write into a caller-provided byte sink.
2629
*
27-
* <p>Batches are accepted via the Arrow C Data Interface: callers export an Arrow record batch to an {@code ArrowArray}
28-
* / {@code ArrowSchema} pair and pass the addresses to {@link #writeBatch(long, long)}. The writer accepts up to four
29-
* in-flight batches on the session's runtime thread before back-pressuring the caller.
30+
* <p>Use {@link #writeBatch(VectorSchemaRoot, BufferAllocator)} for zero-copy Java batches with independent buffer
31+
* lifetimes. Batches are also accepted via the Arrow C Data Interface: callers export an Arrow record batch to an
32+
* {@code ArrowArray} / {@code ArrowSchema} pair and pass the addresses to {@link #writeBatch(long, long)}. The writer
33+
* accepts up to four in-flight batches on the session's runtime thread before back-pressuring the caller.
3034
*
3135
* <p>Call {@link #close()} to flush remaining batches and finalize the file. If the writer becomes unreachable without
3236
* an explicit {@code close()}, {@link VortexCleaner} will flush and release native resources as a backstop — but
3337
* callers should always finalize explicitly so that I/O errors surface through the normal call path.
3438
*/
3539
public final class VortexWriter implements AutoCloseable {
3640
private final long pointer;
41+
private final Schema arrowSchema;
3742
private final AtomicBoolean closed = new AtomicBoolean(false);
3843
private volatile VortexWriteSummary summary;
3944

40-
private VortexWriter(long pointer) {
45+
private VortexWriter(long pointer, Schema arrowSchema) {
4146
Preconditions.checkArgument(pointer != 0, "invalid writer pointer");
4247
this.pointer = pointer;
48+
this.arrowSchema = arrowSchema;
4349
AtomicBoolean closedRef = this.closed;
4450
VortexCleaner.register(this, () -> {
4551
if (closedRef.compareAndSet(false, true)) {
@@ -170,7 +176,7 @@ public VortexWriter build() throws IOException {
170176
if (pointer <= 0) {
171177
throw new IOException("failed to create writer for " + destination() + " (ptr=" + pointer + ")");
172178
}
173-
return new VortexWriter(pointer);
179+
return new VortexWriter(pointer, arrowSchema);
174180
}
175181
}
176182

@@ -183,8 +189,38 @@ private void checkOptionsApply() {
183189
}
184190
}
185191

186-
/** Write a batch directly from Arrow C Data Interface addresses. */
187-
public void writeBatch(long arrowArrayAddr, long arrowSchemaAddr) throws IOException {
192+
/**
193+
* Write a Java Arrow batch without copying its data. Native buffers retain their allocations independently, so a
194+
* retained column does not pin the entire batch. The caller may close or clear the root after this method returns,
195+
* but must not overwrite its retained buffers.
196+
*
197+
* @param root batch to write; dictionary-encoded fields require the address-based API
198+
* @param allocator allocator for temporary C Data Interface metadata
199+
*/
200+
public synchronized void writeBatch(VectorSchemaRoot root, BufferAllocator allocator) throws IOException {
201+
Preconditions.checkState(!closed.get(), "writer already closed");
202+
if (!arrowSchema.equals(root.getSchema())) {
203+
throw new IOException("batch schema does not match writer schema");
204+
}
205+
try (ArrowArray array = ArrowArray.allocateNew(allocator)) {
206+
try {
207+
Data.exportVectorSchemaRoot(allocator, root, null, array);
208+
if (!NativeWriter.writeBatchBuffers(pointer, array.memoryAddress(), NativeBuffer.collect(root))) {
209+
throw new IOException("failed to write batch");
210+
}
211+
} finally {
212+
array.release();
213+
}
214+
} catch (RuntimeException e) {
215+
throw new IOException("failed to write batch", e);
216+
}
217+
}
218+
219+
/**
220+
* Write a batch directly from Arrow C Data Interface addresses. These buffers share the producer's batch release
221+
* callback; use {@link #writeBatch(VectorSchemaRoot, BufferAllocator)} for independently retained Java allocations.
222+
*/
223+
public synchronized void writeBatch(long arrowArrayAddr, long arrowSchemaAddr) throws IOException {
188224
Preconditions.checkState(!closed.get(), "writer already closed");
189225
final boolean ok;
190226
try {
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// SPDX-FileCopyrightText: Copyright the Vortex contributors
3+
4+
package dev.vortex.jni;
5+
6+
import java.util.ArrayList;
7+
import java.util.List;
8+
import org.apache.arrow.memory.ArrowBuf;
9+
import org.apache.arrow.memory.ReferenceManager;
10+
import org.apache.arrow.vector.FieldVector;
11+
import org.apache.arrow.vector.VectorSchemaRoot;
12+
13+
/** Per-allocation ownership used by the native zero-copy importer. */
14+
public final class NativeBuffer {
15+
private final ReferenceManager manager;
16+
private final long address;
17+
private final long capacity;
18+
19+
private NativeBuffer(ArrowBuf buffer) {
20+
manager = buffer.getReferenceManager();
21+
address = buffer.memoryAddress();
22+
capacity = buffer.capacity();
23+
}
24+
25+
/** Capture buffer ownership metadata while the root remains alive. */
26+
public static NativeBuffer[] collect(VectorSchemaRoot root) {
27+
List<NativeBuffer> buffers = new ArrayList<>();
28+
for (FieldVector vector : root.getFieldVectors()) {
29+
collect(vector, buffers);
30+
}
31+
return buffers.toArray(new NativeBuffer[0]);
32+
}
33+
34+
private static void collect(FieldVector vector, List<NativeBuffer> buffers) {
35+
for (ArrowBuf buffer : vector.getFieldBuffers()) {
36+
if (buffer.capacity() != 0) {
37+
buffers.add(new NativeBuffer(buffer));
38+
}
39+
}
40+
for (FieldVector child : vector.getChildrenFromFields()) {
41+
collect(child, buffers);
42+
}
43+
}
44+
45+
/** Buffer address, called by JNI. */
46+
public long address() {
47+
return address;
48+
}
49+
50+
/** Buffer capacity, called by JNI. */
51+
public long capacity() {
52+
return capacity;
53+
}
54+
55+
/** Acquire an independent native reference, called by JNI. */
56+
public void retain() {
57+
manager.retain();
58+
}
59+
60+
/** Release the native reference, possibly from a writer worker thread. */
61+
public void release() {
62+
manager.release();
63+
}
64+
}

‎java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriter.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@ public static native long createStream(
4545
*/
4646
public static native boolean writeBatch(long writerPointer, long arrowArrayAddress, long arrowSchemaAddress);
4747

48+
/**
49+
* Import a Java batch with independently retained buffer allocations. The exported array must match the schema
50+
* passed to {@link #create}; the caller must serialize writes and close.
51+
*/
52+
public static native boolean writeBatchBuffers(long writerPointer, long arrowArrayAddress, NativeBuffer[] buffers);
53+
4854
/** Number of bytes successfully written to the underlying sink so far. */
4955
public static native long bytesWritten(long writerPointer);
5056

0 commit comments

Comments
 (0)