You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: explore bulk copies for no-null fixed-width Arrow writes #5299
ArrowWriter.writeColNoNull still copies fixed-width columns one value at a time with setValueUnsafe. This avoids Arrow's per-value capacity check, but retains a getter call and setter call for every value.
A bulk copy into the Arrow data buffer could remove that per-row dispatch for eligible Spark column vectors. This was identified during review of #5046.
Describe the potential solution
Investigate a specialized no-null path for fixed-width types whose Spark and Arrow physical layouts match:
copy the contiguous value region into BaseFixedWidthVector.getDataBuffer;
initialize the destination validity bitmap in bulk;
preserve the existing per-value path for unsupported vectors and layouts;
handle sliced input offsets, dictionaries, endianness, and both on-heap and off-heap inputs;
benchmark the specialized path across representative types and batch sizes before adopting it.
There are API constraints to resolve first:
OffHeapColumnVector.valuesNativeAddress exposes the values address but is annotated @VisibleForTesting;
OnHeapColumnVector keeps its primitive arrays private, while its public bulk getters allocate new arrays;
ColumnarArray keeps its backing vector and offset private, so the current writer API does not expose the information needed for a direct copy;
booleans are byte-per-value in Spark but bit-packed in Arrow, and decimals/intervals do not generally share a directly copyable layout.
Avoid reflection or private-field access that would be brittle across supported Spark versions. The issue should only proceed if benchmarks show that a stable source-access approach beats the current loop.
Activity
peterxcli commentedon Aug 7, 2026
take