Add a crate for HeapSize trait by adriangb · Pull Request #9138 · apache/arrow-rs

adriangb · 2026-01-10T20:08:57Z

The rationale is that we want to use it in DataFusion but don't want to force a dependency on Parquet to use this trait.

adriangb · 2026-01-10T21:33:25Z

arrow-array/src/heap_size.rs

+impl HeapSize for StringArray {
+    fn heap_size(&self) -> usize {
+        self.get_buffer_memory_size()
+    }
+}


This could probably be a blanket implementation:

impl<T> HeapSize for T where T: Array

alamb · 2026-01-12T17:46:18Z

I am not sure we need a new crate just for the HeapSize trait in a downstream crate (DataFUsion)

What about making a trait like this in DataFusion common:

pub trait DFHeapSize {
...
}

And then providing a blanket implementation for DFHeapSize for everything that implements HeapSize

impl <T: HeapSize> for DFHeapSize { 
...
}

That way we can use the arrow implementations, but we don't have to make a whole new crate etc

alamb · 2026-01-12T17:46:54Z

Cargo.toml

 arrow-data = { version = "57.2.0", path = "./arrow-data" }
 arrow-ipc = { version = "57.2.0", path = "./arrow-ipc" }
 arrow-json = { version = "57.2.0", path = "./arrow-json" }
+arrow-memory-size = { version = "57.2.0", path = "./arrow-memory-size" }


If we are going to do this I would suggest we put it in arrow-buffer or something

I'm happy to just move it there instead of a new crate, certainly easier

adriangb · 2026-01-12T18:52:06Z

I agree we can move this into arrow-buffer and then make a trait in DataFusion if that sounds better. We still a different crate for the macros though 😞

alamb · 2026-01-12T21:26:39Z

What do we think about just adding the new crate in DataFusion?

Or is there more to this PR that I am missing?

alamb · 2026-01-12T21:26:56Z

I haven't had a chance to look through it (2k lines is a lot!)

alchemist51

Pretty neat PR overall @adriangb ! Few minor comments on the PR.

alchemist51 · 2026-01-29T09:54:41Z

arrow-array/src/heap_size.rs

+    fn test_primitive_array_heap_size() {
+        let array = Int32Array::from(vec![1, 2, 3, 4, 5]);
+        let size = array.heap_size();
+        assert!(size >= 5 * std::mem::size_of::<i32>());


Can it not be accurately compared?

Changed to be exact comparisons

alchemist51 · 2026-01-29T10:06:55Z

arrow-memory-size-derive/src/lib.rs

+//!
+//! #[derive(HeapSize)]
+//! struct MyStruct {
+//!     #[heap_size(size_fn = custom_size)]


Loved the interface to add custom size for custom objects!

alchemist51 · 2026-01-29T10:25:56Z

arrow-memory-size/README.md

+
+Key motivations:
+
+- **Minimal API**: Just two methods (`heap_size()` and `total_size()`)


I didn't see the total_size() being exposed, can you point me to that implementation? In case we are targeting for total_size. One can't use the ignore type in FieldAttr right? Since the memory allocation has to belong to either stack or heap? Should we keep that type?

Here you go: https://github.com/apache/arrow-rs/pull/9138/changes#r2751327162.

The ignore option is useful e.g. for caches, shared references where you know / want to semantically treat them as owned by someone else. E.g. Vec<Arc<T>> should still count the Vec and pointers, but not the T.

That makes sense, thanks @adriangb!

This PR introduces two new crates: - `arrow-memory-size`: Contains the `HeapSize` trait for estimating heap memory usage, with implementations for std types, arrow-buffer types, and all arrow-array types. - `arrow-memory-size-derive`: Provides `#[derive(HeapSize)]` proc macro for automatically implementing HeapSize on structs and enums. The HeapSize infrastructure was previously in the parquet crate. Moving it to a standalone crate allows users who don't use parquet to depend on memory size estimation utilities. **Breaking Change**: The `#[derive(HeapSize)]` macro has been removed from `parquet_derive`. Users should switch to `arrow-memory-size-derive`. The parquet crate re-exports `HeapSize` from `arrow_memory_size` for backward compatibility. Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Move HeapSize implementations from arrow-memory-size to their respective crates (arrow-buffer, arrow-array) - arrow-memory-size now contains only the trait and std impls - Re-export HeapSize and HeapSizeDerive from arrow::util - Add derive macro attributes: #[heap_size(ignore)], #[heap_size(size = N)], #[heap_size(size_fn = path)] - Add type coverage: tuples (up to 12), arrays [T; N], Mutex, RwLock - Add README files with comparison to deepsize and get-size2 crates This allows users who don't use arrow to depend on just arrow-memory-size for the trait, while arrow crates implement HeapSize for their own types. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Remove prescriptive recommendations about which crates to use - Explain motivations: minimal API, customizable Arc/Rc semantics, Arrow integration - Simplify comparison section to just explain our approach Co-Authored-By: Claude Opus 4.5 <[email protected]>

Tests for the HeapSize derive macro are not parquet-specific, so move them from parquet_derive_test to arrow-memory-size-derive/tests/. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Proc-macro crates cannot resolve intra-doc links to external crates, causing rustdoc errors for the arrow_memory_size::HeapSize references. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Replace individual HeapSize implementations with a macro to reduce code duplication. A blanket impl<T: Array> is not possible due to Rust's orphan rules (E0210). Co-Authored-By: Claude Opus 4.5 <[email protected]>

adriangb · 2026-02-01T14:53:58Z

arrow-memory-size/src/lib.rs

+
+    /// Return the total size of this object including heap allocations
+    /// and the size of the object itself.
+    fn total_size(&self) -> usize {


Here is total_size()

github-actions bot added parquet Changes to the parquet crate arrow Changes to the arrow crate labels Jan 10, 2026

adriangb commented Jan 10, 2026

View reviewed changes

adriangb marked this pull request as ready for review January 11, 2026 20:34

adriangb mentioned this pull request Jan 12, 2026

Add heap memory estimation for statistics apache/datafusion#19599

Closed

alamb reviewed Jan 12, 2026

View reviewed changes

mkleen mentioned this pull request Jan 28, 2026

Add a memory bound FileStatisticsCache for the Listing Table apache/datafusion#20047

Open

alchemist51 reviewed Jan 29, 2026

View reviewed changes

adriangb and others added 9 commits January 31, 2026 07:14

Fix formatting in arrow-memory-size crates

4d0e68d

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Fix formatting in arrow-array heap_size module

6f71be2

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Fix formatting in parquet_derive_test

1c2eb09

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Move HeapSize derive tests to arrow-memory-size-derive

4722a72

Tests for the HeapSize derive macro are not parquet-specific, so move them from parquet_derive_test to arrow-memory-size-derive/tests/. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Remove broken intra-doc links in proc-macro crate

23d2648

Proc-macro crates cannot resolve intra-doc links to external crates, causing rustdoc errors for the arrow_memory_size::HeapSize references. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Use macro to reduce HeapSize impl repetition

7245a2a

Replace individual HeapSize implementations with a macro to reduce code duplication. A blanket impl<T: Array> is not possible due to Rust's orphan rules (E0210). Co-Authored-By: Claude Opus 4.5 <[email protected]>

adriangb force-pushed the heap-size-crate branch from aa2d756 to 7245a2a Compare January 31, 2026 13:14

adriangb commented Feb 1, 2026

View reviewed changes

touch up

54bf894


		Key motivations:

		- Minimal API: Just two methods (`heap_size()` and `total_size()`)

Comments

Conversation

adriangb commented Jan 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Jan 12, 2026

Uh oh!

alamb commented Jan 12, 2026

Uh oh!

alamb commented Jan 12, 2026

Uh oh!

alchemist51 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants