Skip to content

Implement std::fmt::Write for StringViewBuilder to permit non contiguous data to be written #6373

Open

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

DataFusion has an optimized version of concat(col1, ...) for StringView added by @devanbenz in apache/datafusion#12224 that uses a StringViewArrayBuilder which is similar but not the same as StringViewBuilder in arrow: https://github.com/apache/datafusion/blob/9bc39a0522840ed90de2a4d23157de2e192cd00f/datafusion/functions/src/string/common.rs#L464-L536

The major differences are:

  1. You can call write to incrementally build up each string and then call append_offset to create each string. StringBuilder requires each input to be a single contiguous string to call
  2. You can avoid creating the null buffer and pass it in to the finish

Describe the solution you'd like
I would like the APIs in arrow-rs to be sufficiently complete that we could use the arrow-rs versions rather than our own custom versions in DataFusion

Describe alternatives you've considered
StringBuilder allows this like

use std::fmt::Write;
use arrow_array::builder::GenericStringBuilder;
let mut builder = GenericStringBuilder::<i32>::new();

// Write data in multiple `write!` calls
write!(builder, "foo").unwrap();
write!(builder, "bar").unwrap();
// The next call to append_value finishes the current string
// including all previously written strings.
builder.append_value("baz");

let array = builder.finish();
assert_eq!(array.value(0), "foobarbaz");

I think it would be cool to try and follow the same API for StringViewBuilder -- though note that API may be more complicated to ensure we don't get any performance regressions

Additional context

  1. See Improve GenericStringBuilder documentation #6372 for better docs of what StringBuilder does
  2. Ability to append non contiguous strings to StringBuilder #6347 lists more background

Activity

  1. alamb commented on Sep 9, 2024

    @alamb
    ContributorAuthor

    cc @devanbenz in case you are interested in trying to figure out what we need to add to the upstream arrow code to get rid of the custom implementation in DataFusion

  2. devanbenz commented on Sep 19, 2024

    @devanbenz
    Contributor

    Just now getting eyes on this FYI

  3. alamb commented on Sep 19, 2024

    @alamb
    ContributorAuthor

    No rush -- I think we have plenty of other things going on -- I would say this is a nice to have in my mind

  4. alamb commented on Nov 22, 2024

    @alamb
    ContributorAuthor

    BTW @tlm365 made a PR here #6719 that needed it for some other reasons, However, it seems like it needs a bit more effort 🤔

  5. alamb commented on Jul 31, 2025

    @alamb
    ContributorAuthor

    Copying an idea @waynexia and I had in DataFusion: apache/datafusion#16878 (comment)

    Something that might be worth looking into is to use the underlying buffer as temp space, and then "rewind" the values (aka call current_buffer.truncate(original_size) if the data fit entirely in the view 🤔 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions