Repository navigation
Commit 8dbcdaf
authored
fix(jni): retain Java Arrow buffers independently (#49)
## Summary
Fixes #46. Provides a zero-copy alternative to the batch-copy workaround
in #47.
A retained validity bitmap or column slice currently keeps an entire
imported Arrow batch alive through its shared C Data Interface release
callback. The new Java-root write path retains each Java allocation
independently, so the writer releases unrelated columns as soon as it no
longer needs them. Spark uses this path by default.
In the nested-string regression workload (16 batches of 8,192 rows),
memory retained before `finish` dropped from about **65 MiB to 8.1
MiB**, with all allocations released at finish. These are synthetic
results; the production 615-column workload has not been rerun.
## Changes
- Add `VortexWriter.writeBatch(VectorSchemaRoot, BufferAllocator)`. It
checks the batch schema, exports the array, and imports it using the
writer's stored schema, avoiding a schema export for every batch.
- Retain Java Arrow reference managers independently and replace the
Rust buffers' owners without copying their data. The last Rust reference
releases its Java allocation, including from native worker threads.
- Preserve Arrow's existing alignment copies for unaligned input. Track
the original exported addresses so missing Java owners still fail import
rather than silently keeping whole batches alive.
- Switch the Spark V2 writer to the new path. The address-based API
remains compatible and keeps the producer's shared release callback; its
callers must use the Java-root overload to get independent allocation
lifetimes. Dictionary-encoded roots still require the address-based API.
- Serialize both write methods with `finish` to prevent access to a
freed native writer, and release the temporary schema used when creating
a writer.
Buffers sharing one underlying Java allocation still share that
allocation's lifetime.
## Testing
- `IndependentBuffersTest`: memory retained before finish and full
release afterward; partial-import and sink-failure cleanup; empty
batches and string views; unaligned integer buffers; schema mismatch
rejection; concurrent write/finish stress for both APIs (16 iterations
each).
- `JNIWriterTest`: both APIs round-trip strings and nested maps. **18
JNI tests pass; one existing test is disabled.**
- Rust ownership tests: unchanged buffer addresses, independent column
release, sliced validity offsets, and release/global-reference cleanup
from a native thread. **3 pass.**
- Spark 2.13 `VortexDataSourceWriteTest`: **11 pass.**
- `cargo clippy -p vortex-jni --all-targets --all-features -- -D
warnings`, pinned Rust formatting, Java formatting checks, and patch
whitespace checks pass.
- Four independent reviews covered reuse, maintainability, efficiency
and correctness; confirmed findings were fixed and tested.
Known separately: the existing Vortex string-view concatenation path
reads metadata from null slots. The view round-trip fixture initializes
those slots; this PR does not fix that issue.
Checked locally against QC `develop`.
I used AI tools to help write and review this change.
Signed-off-by: Christopher Wirt <[email protected]>1 parent 7ad32dd commit 8dbcdaf
12 files changed
Lines changed: 804 additions & 37 deletions
File tree
- java
- vortex-jni/src
- main/java/dev/vortex
- api
- jni
- test/java/dev/vortex/jni
- vortex-spark/src/main/java/dev/vortex/spark/write
- vortex-jni
- src
- buffer
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
| 21 | + | |
19 | 22 | | |
20 | 23 | | |
21 | 24 | | |
| |||
24 | 27 | | |
25 | 28 | | |
26 | 29 | | |
27 | | - | |
28 | | - | |
29 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
30 | 34 | | |
31 | 35 | | |
32 | 36 | | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
36 | 40 | | |
| 41 | + | |
37 | 42 | | |
38 | 43 | | |
39 | 44 | | |
40 | | - | |
| 45 | + | |
41 | 46 | | |
42 | 47 | | |
| 48 | + | |
43 | 49 | | |
44 | 50 | | |
45 | 51 | | |
| |||
170 | 176 | | |
171 | 177 | | |
172 | 178 | | |
173 | | - | |
| 179 | + | |
174 | 180 | | |
175 | 181 | | |
176 | 182 | | |
| |||
183 | 189 | | |
184 | 190 | | |
185 | 191 | | |
186 | | - | |
187 | | - | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
188 | 224 | | |
189 | 225 | | |
190 | 226 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
48 | 54 | | |
49 | 55 | | |
50 | 56 | | |
| |||
0 commit comments