ARROW-1192: [JAVA] Use buffer slice for splitAndTransfer in List and …#819
ARROW-1192: [JAVA] Use buffer slice for splitAndTransfer in List and …#819StevenMPhillips wants to merge 1 commit intoapache:masterfrom
Conversation
…Union vectors Includes a fix for splitAndTransfer in BitVector for the case when split length is 0
|
@StevenMPhillips Can you give a bit more context on this change so that I'm able to review it? |
|
Needs a unit test also |
|
@siddharthteotia, can you take over this pull request and add some unit tests? The goal of this PR was to update some vectors so they were more efficient at splitting vectors into subvectors. Many vectors are already efficient, minimizing manipulation of data. However, the ListVector and BitVector were doing far more work than necessary. In Dremio we use splitAndTransfer when unrolling lists. Ensuring that these apis are efficient substantially reduces CPU overhead during this operation. See the code using it here: https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/sabot/op/flatten/FlattenTemplate.java#L205. Probably makes sense to move our Dremio code for this class over to Arrow as well: |
|
@jacques-n |
|
I have opened a new PR -- #901 |
|
This one can be closed |
|
Can this be closed? |
|
@StevenMPhillips, please close. |
|
@StevenMPhillips can you close? |
…Union vectors
Includes a fix for splitAndTransfer in BitVector for the case when split length is 0