Repository navigation
refactor(spark): make ArrowFieldWriter generic over its vector - #17
Merged
Merged
Conversation
Each writer kept a typed copy of the base vector field. Hold it as `final V vector` instead, and advance `count` only in the base class: a null struct's children now get `writeNull()` rather than having their `count` bumped by hand. Signed-off-by: gitRasheed <[email protected]>
ChristopherWirt
approved these changes
Oct 30, 2026
ChristopherWirt
added a commit
that referenced
this pull request
Nov 2, 2026
Preserve the generic Arrow field writers from #17 and exception-safe schema cleanup and regression coverage from #18 alongside independent JNI buffer ownership. Signed-off-by: Christopher Wirt <[email protected]>
gitRasheed
added a commit
that referenced
this pull request
Nov 2, 2026
Develop changed old-connector files that this branch removes (#17) and the JNI writer files this branch already carries with the new writer port (#18, #49). The resolution keeps this branch's versions and takes the broader schema release test from #18. Signed-off-by: gitRasheed <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #14
Two structural tidy-ups in
ArrowFieldWriter. No behaviour change.Changes
ArrowFieldWriter<V extends FieldVector>holds the vector asfinal V vector. The 16 subclasses drop their typed copy of it (bits,bytes,shorts, ...) and the constructors that only set it.createreturnsArrowFieldWriter<?>.countadvances only in the base class.writeNull()appends a null and advancescount, andStructWriter.setNullcalls it for each child instead of incrementing the children'scountby hand.Testing
Existing tests cover it.
VortexNestedSchemaPruningTestwrites null structs with struct children, and one of its cases fails ifwriteNull()doesn't advancecount.Checked locally:
./gradlew --parallel --continue javadoc check, and:vortex-spark_2.13:test -PsparkTestVersion=4.2.0.I used AI tools to help write and review this change.