Skip to content

Interleave for String Views duplicates buffers #11280

Closed

Description

@cetra3

Describe the bug

When investigating some shortcomings in apache/datafusion#23565 I found that the interleave method doesn't reuse existing buffers, and clones them.

While this is fine for memory/perf, since cloning a buffer is a refcount rather than a full data clone, the problem arises in things that don't know about this accounting:

  • The issue as is in DF, writing out IPC can inflate the output drastically & gc() can make things much worse
  • BatchCoalescer gc check can inflate values as well

To Reproduce

use arrow::array::{Array, AsArray, StringViewArray};
use arrow::compute::interleave;

let array = StringViewArray::from_iter_values(
    (0..4).map(|i| format!("a string long enough to not be inlined {i}")),
);
assert_eq!(array.data_buffers().len(), 1);

let a = array.slice(0, 2);
let b = array.slice(2, 2);
let result = interleave(&[&a, &b], &[(0, 0), (1, 0), (0, 1), (1, 1)]).unwrap();

// fails: the single shared buffer is emitted twice
assert_eq!(result.as_string_view().data_buffers().len(), 1);

The effect grows with the number of inputs sharing buffers. Interleaving all 8192 rows of a StringViewArray (5 data buffers, 638,976 bytes) back from k equal slices:

k output buffers get_buffer_memory_size IPC stream bytes
1 5 638,976 517,640
2 6 770,048 648,776
4 8 1,097,728 853,704
8 12 1,654,784 1,288,072

Expected behavior

I'd expect interleave to reuse existing view buffers

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions