You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Arrow] Add bitwise operations BooleanArray that potentially reuse the underlying allocation #8809
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It is common to apply successive filtering operations that progressively narrow a bitmap with rows that pass a predicate -- for example a AND b AND c AND d AND ...
This would be evaluated typically like
t1 = a AND b
t2 = t1 AND c
t3 = t2 AND d
Where t1, t2, and t3 are newly allocated arrays; t3 holds the final output
For a typical array size of 8192 elements, t1, t2 and t3 are eack 1KB allocations
The current boolean kernels in arrow pretty much require allocating a new buffer as they take a reference
Now that @rluvaton has added optimized code to apply mutable bitwise operations in the following PR
So that would mean adding new functions to BooleanArray, like
implBooleanArray{
...
// apply a bitwise operation to this BooleanArray, using u64 operations// returning a new BooleanArrayfnbitwise_unary(&self,mutop:F) -> SelfwhereF:FnMut(u64) -> u64{.. }// Try to apply a bitwise operation to this BooleanArray in place, using u64 operations// If the underlying buffer is shared, returns Err<self>fnbitwise_unary_mut(self,mutop:F) -> Result<Self,Self>whereF:FnMut(u64) -> u64{.. }// Try to apply a bitwise operation to this BooleanArray in place, using u64 operations// If the underlying buffer is shared, returns a new // BooleanArrayfnbitwise_unary_mut_or_clone(self,mutop:F) -> SelfwhereF:FnMut(u64) -> u64{.. }
And we would also add similar functions for binary_bitwise functions (that takes a second array argument and right_offset_in_bits)
These APIs are likely fairly straightforward if we implement the plan in
changed the title [-][Arrow] Add in place / mutable bitwise operations `BooleanArray`[/-][+][Arrow] Add `mut` bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/+]on Nov 9, 2025
changed the title [-][Arrow] Add `mut` bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/-][+][Arrow] Add bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/+]on Nov 9, 2025
changed the title [-][Arrow] Add bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/-][+][Arrow] Add bitwise operations `BooleanArray` that potentially reuse the underlying allocation[/+]on Jan 28, 2026
Activity
rluvaton commentedon Nov 9, 2025
I think the title is misleading, there can't be mutation on Arrow Array due to the nature of immutable arrays in arrow
[-][Arrow] Add in place / mutable bitwise operations `BooleanArray`[/-][+][Arrow] Add `mut` bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/+][-][Arrow] Add `mut` bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/-][+][Arrow] Add bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/+]alamb commentedon Nov 9, 2025
Good call -- I tried to update it to better reflect the idea
[-][Arrow] Add bitwise operations `BooleanArray` that potentially reuse the underlying allocaton[/-][+][Arrow] Add bitwise operations `BooleanArray` that potentially reuse the underlying allocation[/+]alamb commentedon Mar 18, 2026
@Dandandan just added in place bitwise operations:
Implement apache#8809 and modify union_many to use that. Remove Boole…
Add mutable bitwise operations to `BooleanArray` and `NullBuffer::uni…
Add mutable bitwise operations to `BooleanArray` and `NullBuffer::uni…