Skip to content

Memory accounting in GenericBytesView overcounts allocation size #7099

New issue
New issue
Closed
Closed
Memory accounting in GenericBytesView overcounts allocation size#7099
Labels
arrowChanges to the arrow cratebuggood first issueGood for newcomers
@alamb

Description

@alamb
alamb
opened on Feb 8, 2025
          This is something I also noticed in #7082

One thing to note is that NullBufferBuilder::allocated_size() is used here:

arrow-rs/arrow-array/src/builder/generic_bytes_view_builder.rs

Lines 397 to 408 in 92cfd99

/// Return the allocated size of this builder in bytes, useful for memory accounting.
pub fn allocated_size(&self) -> usize {
let views = self.views_builder.capacity() * std::mem::size_of::<u128>();
let null = self.null_buffer_builder.allocated_size();
let buffer_size = self.completed.iter().map(|b| b.capacity()).sum::<usize>();
let in_progress = self.in_progress.capacity();
let tracker = match &self.string_tracker {
Some((ht, _)) => ht.capacity() * std::mem::size_of::<usize>(),
None => 0,
};
buffer_size + in_progress + tracker + views + null
}

And I think it's used with assumption it provides bytes not bits, so may need adjustment.

Originally posted by @Jefffrey in #7089 (review)

I think the adjustment is to divide the allocated_size by 8.

Another solution would be to deprecate allocated_size entirely and make a new function that returns allocated size in bytes

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow cratebuggood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions