Refactoring the io and in-memory lazy vs non-lazy representation #293

Activity

1. giovp

   self-assigned this

   on Jun 13, 2023
2. LucaMarconato commented on Jun 27, 2023

   

   LucaMarconato

   on Jun 27, 2023

   MemberAuthor

   More actions

   @lopollar reported that saving extra columns in shapes layers is not working at the moment #311. I think it was working but we didn't have a test for that.

   Please @giovp when working on this PR also add a test for this.

   👍React with 👍1Reacted by Giovanni Palla
3. LucaMarconato commented on Jul 2, 2023

   

   LucaMarconato

   on Jul 2, 2023

   MemberAuthor

   More actions

   Refactoring the geometry argument of the shapes model: #315
4. giovp commented on Jul 30, 2023

   

   giovp

   on Jul 30, 2023

   Member

   More actions

   question regarding incremental, plan is to remove entirely the write/read steps when __setitem__ is called, unless overwrite=True, in that case a call to write follows and hence makes sense to reload data lazy?
5. LucaMarconato commented on Jul 30, 2023

   

   LucaMarconato

   on Jul 30, 2023

   Last edited by LucaMarconato

   MemberAuthor

   More actions

   I'd maybe remove add_image(), etc and leave only __setitem__. In this way there is no option to pass overwrite=True. I would then add a method to resave to disk a specifc element. So the workflow would be:

   sdata.images['my_image'] = im
   # or even the following, that is already implemented and uses single dispatch to forward 
   # the right `__setitem__`
   sdata['my_image'] = im
   sdata.write('my_data.zarr')
   # example of in-place operation (can be both lazy or in-memory)
   sdata['my_image'] = sdata['my_image'] + 1
   sdata.write(element='my_image')

   We could add a parameter reload=True to write() that does the lazy re-reading, and let the user disable it for specific cases, like when the data is all full in-memory and the user wants to keep it there.

   👍React with 👍2Reacted by Giovanni Palla and lopollar
6. LucaMarconato commented on Jul 30, 2023

   

   LucaMarconato

   on Jul 30, 2023

   MemberAuthor

   More actions

   I'd then have something like this to load an element in-memory.

   # all is read lazy by default
   sdata = read_zarr('my_data.zarr')
   sdata.load_in_memory(element='my_image')

   👍React with 👍2Reacted by Giovanni Palla and lopollar
7. LucaMarconato commented on Jun 11, 2024

   

   LucaMarconato

   on Jun 11, 2024

   MemberAuthor

   More actions

   We should also consider adding the option to keep data lazy but making it persistent (.persist()).
8. giovp

   added

   format

   on Sep 8, 2024