You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Implement std::fmt::Write for StringViewBuilder to permit non contiguous data to be written #6373
You can call write to incrementally build up each string and then call append_offset to create each string. StringBuilder requires each input to be a single contiguous string to call
You can avoid creating the null buffer and pass it in to the finish
Describe the solution you'd like
I would like the APIs in arrow-rs to be sufficiently complete that we could use the arrow-rs versions rather than our own custom versions in DataFusion
Describe alternatives you've considered StringBuilder allows this like
use std::fmt::Write;use arrow_array::builder::GenericStringBuilder;letmut builder = GenericStringBuilder::<i32>::new();// Write data in multiple `write!` callswrite!(builder,"foo").unwrap();write!(builder,"bar").unwrap();// The next call to append_value finishes the current string// including all previously written strings.
builder.append_value("baz");let array = builder.finish();assert_eq!(array.value(0),"foobarbaz");
I think it would be cool to try and follow the same API for StringViewBuilder -- though note that API may be more complicated to ensure we don't get any performance regressions
cc @devanbenz in case you are interested in trying to figure out what we need to add to the upstream arrow code to get rid of the custom implementation in DataFusion
Something that might be worth looking into is to use the underlying buffer as temp space, and then "rewind" the values (aka call current_buffer.truncate(original_size) if the data fit entirely in the view 🤔 )
Activity
alamb commentedon Sep 9, 2024
cc @devanbenz in case you are interested in trying to figure out what we need to add to the upstream arrow code to get rid of the custom implementation in DataFusion
devanbenz commentedon Sep 19, 2024
Just now getting eyes on this FYI
alamb commentedon Sep 19, 2024
No rush -- I think we have plenty of other things going on -- I would say this is a nice to have in my mind
alamb commentedon Nov 22, 2024
BTW @tlm365 made a PR here #6719 that needed it for some other reasons, However, it seems like it needs a bit more effort 🤔
alamb commentedon Jul 31, 2025
Copying an idea @waynexia and I had in DataFusion: apache/datafusion#16878 (comment)