You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Interleave for String Views duplicates buffers #11280
When investigating some shortcomings in apache/datafusion#23565 I found that the interleave method doesn't reuse existing buffers, and clones them.
While this is fine for memory/perf, since cloning a buffer is a refcount rather than a full data clone, the problem arises in things that don't know about this accounting:
The issue as is in DF, writing out IPC can inflate the output drastically & gc() can make things much worse
BatchCoalescer gc check can inflate values as well
To Reproduce
use arrow::array::{Array,AsArray,StringViewArray};use arrow::compute::interleave;let array = StringViewArray::from_iter_values((0..4).map(|i| format!("a string long enough to not be inlined {i}")),);assert_eq!(array.data_buffers().len(),1);let a = array.slice(0,2);let b = array.slice(2,2);let result = interleave(&[&a,&b],&[(0,0),(1,0),(0,1),(1,1)]).unwrap();// fails: the single shared buffer is emitted twiceassert_eq!(result.as_string_view().data_buffers().len(),1);
The effect grows with the number of inputs sharing buffers. Interleaving all 8192 rows of a StringViewArray (5 data buffers, 638,976 bytes) back from k equal slices:
k
output buffers
get_buffer_memory_size
IPC stream bytes
1
5
638,976
517,640
2
6
770,048
648,776
4
8
1,097,728
853,704
8
12
1,654,784
1,288,072
Expected behavior
I'd expect interleave to reuse existing view buffers
Activity
Reuse buffers when interleaving byte views (apache#11281)
Reuse buffers when interleaving byte views (apache#11281)