ScalarValue::try_from_array holds onto input array for Struct, Lists types #15205

Activity

1. Dandandan

   added

   bugSomething isn't workingSomething isn't working

   on Mar 13, 2025
2. Dandandan

   changed the title [-]`ScalarValue::try_from_array` holds onto input array for Struct, Union, Lists types[/-] [+]`ScalarValue::try_from_array` holds onto input array for Struct, Lists types[/+] on Mar 13, 2025
3. alamb

   added

   performanceMake DataFusion fasterMake DataFusion faster

   on Mar 13, 2025
4. chenkovsky commented on Mar 16, 2025

   

   chenkovsky

   on Mar 16, 2025

   Contributor

   More actions

   take

   👍React with 👍1Reacted by Daniël Heres
5. github-actions

   assigned

   chenkovsky

   on Mar 16, 2025
6. chenkovsky commented on Mar 18, 2025

   

   chenkovsky

   on Mar 18, 2025

   Contributor

   More actions

   Hi, @Dandandan .

   it seems that there's no direct way to deep copy array in arrow?
   so I tested the following code. it seems that it works.

   let a = array.slice(index, 1);
   
   // copy
   let data = a.to_data();
   let mut copy = MutableArrayData::new(vec![&data], true, data.len());
   copy.extend(0, 0, data.len());
   let a = make_array(copy.freeze());
   
   Self::Struct(Arc::new(a.as_struct().to_owned()))

   But I'm wondering whether it's better to provide a deep copy method for ScalarValue rather than modify try_from_array.
   then we can just copy the scalar value when it's needed since deep copy is expensive.
7. bert-beyondloops commented on May 21, 2026

   

   bert-beyondloops

   on May 21, 2026

   Last edited by bert-beyondloops

   Contributor

   More actions

   I personally would go for the fix behind the compact() functionality on ScalarValue

   See #21928

   In this way, the developer still has the option to just use the ScalarValue::try_from_array as is without any side effects of recopying memory and possible performance impact.

   If reducing memory footprint is really needed, a compact() call can be made explicitly.