Store conversion methods #137

Activity

1. jakirkham commented on Mar 1, 2017

   

   jakirkham

   on Mar 1, 2017

   MemberAuthor

   More actions

   Musing on this a bit, maybe leveraging the update methods from the MutableMapping interface might be better. One would then provide it the store they would like to move their data to. For instance, converting to a DirectoryStore or a TempStore should be able to follow similar code paths, which should just work thanks to inheritance. This should also allow it to remain extensible in the future.
2. jakirkham

   changed the title [-]Adding to_zipfile/to_dir methods[/-] [+]Store conversion methods[/+] on Mar 1, 2017
3. alimanfoo commented on Mar 1, 2017

   

   alimanfoo

   on Mar 1, 2017 via email

   Member

   More actions

   That is a very elegant idea!

   …

   On Wed, Mar 1, 2017 at 5:15 PM, jakirkham ***@***.***> wrote: Musing on this a bit, maybe leveraging the update methods from the MutableMapping interface might be better. One would then provide it the store they would like to move their data to. For instance, converting to a DirectoryStore or a TempStore should be able to follow similar code paths, which should just work thanks to inheritance. This should also allow it to remain extensible in the future. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/137#issuecomment-283405709>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QkveW6Cvgx6P7Kh97iYaKyYCEpwEks5rhaecgaJpZM4MP6u-> .

   -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721
4. alimanfoo commented on Mar 1, 2017

   

   alimanfoo

   on Mar 1, 2017

   Member

   More actions

   I think it would be worth adding an example to the tutorial to show how to copy data from one store to another.
5. alimanfoo

   added this to the v2.2 milestone on Nov 19, 2017
6. jakirkham commented on Nov 20, 2017

   

   jakirkham

   on Nov 20, 2017

   Last edited by jakirkham

   MemberAuthor

   More actions

   Just to clarify, we are thinking of this as a docs/examples item? If so, I think I agree. The existing interface is already sufficient, but a couple line example would be good. Any thoughts on where this would be documented?
7. alimanfoo commented on Nov 20, 2017

   

   alimanfoo

   on Nov 20, 2017 via email

   Last edited by alimanfoo

   Member

   More actions

   I was actually thinking that a convenience function like zarr.copy_store(source, dest, source_path=None, dest_path=None) might be useful. If source_path and dest_path were None this would be equivalent to copying all key/value pairs from source to dest. If source_path were given this would restrict copying to keys with this prefix, and then dest_path would give a new prefix to prepend before storing. Basically, it's a convenient way to copy data from one hierarchy directly to another, without having to do any decompression/recompression, and allowing for the path within source and dest to be provided so that parts of a hierarchy can be selectively copied. What do you think?
8. jakirkham commented on Nov 20, 2017

   

   jakirkham

   on Nov 20, 2017

   MemberAuthor

   More actions

   No complaints from me. This is somewhat similar to how I imagined the Group copy method. Does that fit the use cases you are imagining here or do you see other benefits from handling this at the Store level instead of the Group level?
9. alimanfoo commented on Nov 20, 2017

   

   alimanfoo

   on Nov 20, 2017

   Member

   More actions

   I was thinking there is potentially a use for both a low-level zarr.copy_store(source, dest, ...) where source and dest are store objects, and a higher-level zarr.copy(source, dest, ...) where source is array-like or group-like and dest is group-like.

   The low-level copy_store() is for the case where you want to replicate data exactly, and so you just copy key/value pairs from one store to another, which is going to be faster because there is no need to decompress/recompress chunks, and because only initialized chunks will get copied.

   The higher-level copy() would be for a more general case where you want to copy an entity (group or array), going via the create_group/create_dataset API. This could mean potentially that source and dest could be anything array- or group-like, e.g., either zarr or h5py, i.e., this would provide a way to migrate data between two zarr hierarchies, or zarr to/from h5py. This could also allow for copying but using different compression to store data in the destination than is used in the source.
10. jakirkham commented on Nov 20, 2017

    

    jakirkham

    on Nov 20, 2017

    MemberAuthor

    More actions

    I see. Sure that seems fine.

    My initial thinking with copy was that it would special case Zarr objects and thus handle direct data replication with the performance of copy_store, but through the same easy high level interface.
11. alimanfoo

    added

    enhancementNew features or improvementsNew features or improvements

    on Nov 21, 2017
12. alimanfoo commented on Nov 21, 2017

    

    alimanfoo

    on Nov 21, 2017 via email

    Member

    More actions

    I've had situations where I'd like to copy between two zarr objects, but change the compressor, in which case it would not be possible to do a direct store-to-store copy. So it would be nice to handle that use case, I was thinking via copy().

    …

    On Monday, November 20, 2017, jakirkham ***@***.***> wrote: I see. Sure that seems fine. My initial thinking with copy was that it would special case Zarr objects and thus handle direct data replication with the performance of copy_store, but through the same easy high level interface. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/137#issuecomment-345868495>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QtFSlxpZWvnqycP77ThyoZgk5Qiiks5s4g6MgaJpZM4MP6u-> .

    -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: [email protected] Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo
13. alimanfoo

    closed this as completedin #217on Jan 2, 2018