The arrow crate's use of shared Buffer allows for fast and memory efficient implementation for operations such as "slice" because slicing is simply a view to the same Buffer

However it creates problems for memory management when there is more than one Array pointing at the same underlying buffer as it is not always clear how much memory should be "charged" to any particular Array and it is hard to avoid double counting buffers when they are shared

• This is discussed in [DISSCUSSION] the memory tracking approach that better handles shared buffers #6439

@waynexia is adding a new MemoryPool API and this ticket tracks the progress, for visibility and hopefully so others can contribute a bit more to the project

Tasks / PRs

• memory tracking with memory pool #7303
• Memory consumption grows for interleave kernel when input is StringViewArray #7151
• Expose memory tracking API to Array and ArrayData #8040

Related issues

• [DISSCUSSION] the memory tracking approach that better handles shared buffers #6439