Repository navigation
Add persistence examples using shelve/dbm, lmdb #131
Description
As discovered in #129, the Python standard shelve module can be used for persistent storage as it provides the MutableMapping interface. This is a good alternative to using the file system directly as it avoids lots of files. It's also a good alternative to using a Zip file which needs care when writing to avoid duplicate entries. Similar benefits probably apply to LMDB which has a MutableMapping interface provided by zict.
It would be good to add some examples using shelve and LMDB to the tutorial.
Activity
alimanfoo commentedon Feb 27, 2017
xref #133 - using dbm directly rather than shelve is probably a better example, as this would bypass the pickling/unpickling of values that shelve does, which in the case of Zarr is unnecessary as values are always bytes objects. However, dbm databases cannot currently be used directly as a store because they return keys as bytes objects.
alimanfoo commentedon Feb 27, 2017
Btw just as a reminder, LMDB supports returning buffer objects which provide direct access to it's own memory-mapped data, which should reduce the number of memory copies required. It would be interesting to benchmark this against other key-value stores to see if this provides an noticeable performance improvement (it may not if compression/decompression is the bottleneck, although previously I have found reducing memory copies can make a difference). Would require some tweak to zict's LMDB wrapper to enable the buffer support.
[-]Add persistence examples using shelve, lmdb[/-][+]Add persistence examples using shelve/dbm, lmdb[/+]jakirkham commentedon Feb 27, 2017
LMDB is definitely pretty slick from what I have read. Another kind of similar database to look at that might be nice is UnQLite.
Agree that pickling/unpickling done by shelve is unnecessary, but would be curious what the overhead is like. If it's not large, having something from the standard library that is easy to setup is nice.
alimanfoo commentedon Feb 27, 2017
alimanfoo commentedon Feb 27, 2017
jakirkham commentedon Feb 27, 2017
How big were your data and chunks?
As long as we stick to version 0, which is the default, we should be fine.
Agreed. Though a potential downside is sometimes it is tricky to get a fully built
dbmmodule. I think we (conda-forge) don't support the GNU stuff (gdbm). Perhaps we could try though. ( conda-forge/python-feedstock#125 )alimanfoo commentedon Feb 27, 2017
jakirkham commentedon Feb 27, 2017
Yep, sorry, I misspoke. So we could still force it to 0.
If it's helpful, we can explore as time allows. The main thing to notice is these might be tricky things to get working on Windows.
I would guess that LMDB is going to actually be a good mix of these. We already have a Python package for this for all platforms at conda-forge. They also build wheels for Windows that they have on PyPI. Building on Unix is straightforward.
It might be worthwhile to have shelve if we want something that works with the standard library. While dbm might be more performant, getting a fully featured build of this for all platforms may be more work and still be less performant than LMDB.