Skip to content

Suppressing ZipFile duplication warning #129

New issue
New issue
Closed
Closed
Suppressing ZipFile duplication warning#129

Description

@jakirkham
jakirkham
opened on Feb 24, 2017

Python's ZipFile allows writing duplicate files and does this by default. When it writes a duplicate file, it raises a UserWarning. This occurs for each file and is a bit noisy. As there doesn't seem to be a standard way of solving this, would recommend that we simply suppress this warning. Combining this with a resolution to issue ( https://github.com/alimanfoo/zarr/issues/128 ), would ensure that deduplication already occurs so the warning is no longer relevant.

Edit: Added link to Python bug after the fact.

Activity

  1. jakirkham commented on Feb 24, 2017

    @jakirkham
    jakirkham
    on Feb 24, 2017
    MemberAuthor

    Proposed fix in PR ( https://github.com/alimanfoo/zarr/pull/130 ).

  2. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    FWIW I think this needs some consideration. If duplicate files are being written into a zip file, and this is happening often, then it is likely that something rather sub-optimal is happening. In the pathological case, a user could be storing many multiples of the actual data for an array without realising, then wonder why the zip file is so large.

    Writing directly to a zip file is really only efficient if the array or arrays being stored in the zip store are written only once, and write operations can be perfectly aligned with chunk boundaries, in which case no duplicate chunk files will ever get created. This can be achieved if an array is created with zarr.array(data=data, ...) or with something like z = zarr.zeros(...); z[:] = data, as in either case zarr internally aligns the write operations with chunk boundaries.

    If the use case requires that data are written and then overwritten, and/or that write operations cannot be aligned with chunk boundaries, then a better approach is probably to initially store the data using DirectoryStore. Then when all writing has finished, the containing directory can be stored into a zip file using the standard command line zip utility. The resulting zip file can then be read directly by zarr without having to unpack. This should be at least as efficient in general as deduplicating a zip file on close by copying to a new zip file.

  3. jakirkham commented on Feb 24, 2017

    @jakirkham
    jakirkham
    on Feb 24, 2017
    MemberAuthor

    It's a fair point honestly. Though I really do like the idea of operating on a single file that includes all of the array data. Zip is nice as it is easy to inspect. Though I don't feel wedded to it given the issues I'm already experiencing by playing with it. Is there some other reasonable storage type that we could add to Zarr that wouldn't have these limitations?

  4. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    I don't know of anything better. Tar is worse apparently as it doesn't support random access. It would be possible (if a little twisted) to use an HDF5 file, however you'd lose the ability to do multi-threaded reads (which seem to work on a zip store surprisingly). cc @mrocklin.

  5. mrocklin commented on Feb 24, 2017

    @mrocklin
    mrocklin
    on Feb 24, 2017
    Contributor

    When I looked into this a long while ago I found that yes, there are other single-file compression formats out there that support random access, but none seemed common place. Generally speaking writing variable sized byte blocks into a single file is a hard problem.

    Another alternative would be an embedded key-value database. Zict has a MutableMapping for LMDB. https://github.com/dask/zict/blob/master/zict/lmdb.py This would be a single directory rather than a single file, but balances large writes and many small writes well.

  6. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    Maybe shelve is an option? It supports the MutableMapping interface so you could probably just use a Shelf as a store without needing to write any new code...

  7. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    Ha, @mrocklin you get much kudos for advocating the MutableMapping interface...

    Python 3.5.3 | packaged by conda-forge | (default, Jan 23 2017, 19:01:48) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zarr
>>> zarr.__version__
'2.1.4'
>>> import numpy as np
>>> import shelve
>>> store = shelve.open('shelf')
>>> z = zarr.array(data=np.arange(1000), store=store, chunks=100)
>>> np.all(z[:] == np.arange(1000))
True
>>> sorted(store)
['.zarray', '.zattrs', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

  8. mrocklin commented on Feb 24, 2017

    @mrocklin
    mrocklin
    on Feb 24, 2017
    Contributor

    Hooray standard interfaces!

  9. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    Looks like shelve supports multi-threaded reads...

    In [1]: import zarr

In [2]: import shelve

In [4]: store = shelve.open('shelf')

In [5]: z = zarr.zeros(10000000000, dtype='i4', chunks=1000000, store=store, overwrite
   ...: =True)

In [7]: %time z[:] = 42
CPU times: user 18.1 s, sys: 808 ms, total: 18.9 s
Wall time: 5.55 s

In [8]: !ls -lh shelf*
-rw-r--r-- 1 aliman aliman   25 Feb 24 23:29 shelf.bak
-rw-r--r-- 1 aliman aliman 199M Feb 24 23:29 shelf.dat
-rw-r--r-- 1 aliman aliman 259K Feb 24 23:29 shelf.dir

In [9]: z
Out[9]: 
Array((10000000000,), int32, chunks=(1000000,), order=C)
  nbytes: 37.3G; initialized: 10000/10000
  compressor: Blosc(cname='lz4', clevel=5, shuffle=1)
  store: DbfilenameShelf

In [10]: import dask.array as da

In [11]: d = da.from_array(z, chunks=z.chunks)

In [14]: %time d.mean().compute()
CPU times: user 1min 13s, sys: 1.25 s, total: 1min 14s
Wall time: 12.6 s
Out[14]: 42.0

  10. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    A BerkeleyDB hash table would probably be another option.

  11. jakirkham commented on Feb 24, 2017

    @jakirkham
    jakirkham
    on Feb 24, 2017
    MemberAuthor

    The fact that Zarr is using a MutableMapping seems like a very useful thing. Not that I have looked into this at all, but I wonder if there are any Key-Value Stores that would work well here.

  12. mrocklin commented on Feb 24, 2017

    @mrocklin
    mrocklin
    on Feb 24, 2017 via email
    Contributor

  13. alimanfoo commented on Feb 24, 2017

    @alimanfoo
    alimanfoo
    on Feb 24, 2017
    Member

    Yes, any key-value store should be an option.

  14. jakirkham commented on Feb 24, 2017

    @jakirkham
    jakirkham
    on Feb 24, 2017
    MemberAuthor

    Thanks for the feedback. I'll give this some more thought.

  15. alimanfoo commented on Feb 25, 2017

    @alimanfoo
    alimanfoo
    on Feb 25, 2017 via email
    Member

  16. jakirkham commented on Feb 27, 2017

    @jakirkham
    jakirkham
    on Feb 27, 2017
    MemberAuthor

    I added a little bit of Python code to zip up the directories after they are written to in such a way as to ensure Zarr can still load them. This is a good enough near term solution for my needs. Would be willing to contribute the utility function or perhaps add another store if there is interest.

  17. jakirkham
    closed this as completedon Feb 27, 2017

  18. alimanfoo commented on Feb 27, 2017

    @alimanfoo
    alimanfoo
    on Feb 27, 2017 via email
    Member

  19. jakirkham commented on Mar 1, 2017

    @jakirkham
    jakirkham
    on Mar 1, 2017
    MemberAuthor

    Opened issue ( https://github.com/alimanfoo/zarr/issues/137 ) to keep track of this idea.

  20. jakirkham commented on Oct 18, 2017

    @jakirkham
    jakirkham
    on Oct 18, 2017
    MemberAuthor

    Forgot to mention that create_group, create_dataset, and open_group will add an empty .zattrs entry to start with. Thus if the attributes need to be set or modified afterwards, this will create duplicate .zattrs entries in a Zip file. Have raised issue ( https://github.com/alimanfoo/zarr/issues/121 ) to allow attrs to be specified in these creation functions.

  21. alimanfoo commented on Oct 18, 2017

    @alimanfoo
    alimanfoo
    on Oct 18, 2017 via email
    Member
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions