You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve ArrowWriter performance for fixed-length vectors #888
Writing to Arrow vectors through Comet's ArrowWriter class only uses setSafe on the underlying vectors. This incurs a capacity and resize check for every write. For CometSparkToColumnarExec we know the sizes ahead of time, so this is unnecessary overhead when performing a lot of writes.
Describe the potential solution
For ArrowWriter we should:
Expose a way to set the capacities for column vectors. Note that setInitialCapacity on the underlying BaseFixedWidthVector doesn't actually allocate anything, so we may need to call allocateNew as well.
Extend ArrowWriter to have unsafe methods that call set on the underlying vectors, probably with assertions that we can elide in release builds.
Activity
coderfender commentedon May 6, 2026
@mbutrovich not sure if this is still relevant ?
mbutrovich commentedon Jun 11, 2026
Seems like it still hasn't been done, just not high priority.
peterxcli commentedon Jun 25, 2026
take