DBMStore - #186

Added tests against berkeleydb. These are run in travis (linux) only, I don't think it's worth trying to get bsddb3 built on appveyor.

OK, I think this is ready to go.

Here's an example using Berkeley DB B-tree:

In [1]: import zarr

In [2]: import bsddb3

In [3]: store = zarr.DBMStore('example.bdb', open=bsddb3.btopen)

In [4]: grp = zarr.group(store)

In [5]: z = grp.create_dataset('foo', shape=100000000, dtype='i8')

In [7]: import numpy as np

In [8]: z[:] = np.arange(z.shape[0])

In [9]: z[:]
Out[9]: array([       0,        1,        2, ..., 99999997, 99999998, 99999999])

In [10]: store.close()

cc @jeromekelleher, @jakirkham - this PR adds support for storing data in any DBM-style database, including Berkeley DB. Should provide an alternative to zip files, without the issues around replacing existing entries. I haven't figured out if/how this works under parallel reads or parallel writes to an array, I know Berkeley DB supports various concurrency options but I don't know which is enabled by default or which is most appropriate for use with zarr. In any case all the tests pass so I will probably merge this and add some caveats to the docs around unknowns for parallel usage. Would be interested if you do try it.

Sounds like a good idea. Don't have time to review it, but do like the idea of having this option.

Indeed. It is interesting. Thanks for the ping. :)

Thanks @alimanfoo, I'll have a play with this when I get a chance and let you know how it goes.

FYI, I'm trying this out instead of Zip containers. Working great so far!

Yeah, doing both concurrent reads and writes and it seems to work fine. Not sure how bsddb3 supports concurrency in BDB. It's pretty solid once it goes through the correct DBEnv object though as far as I know.

Good to know. FWIW it looks like if you use one of the shortcut functions like bsddb3.btopen then it uses a DBEnv with the locking subsystem initialized (DB_INIT_LOCK). I gather that means it is safe to attempt concurrent writes, as long as there's some way to detect deadlocks, which it looks like bsddb3 tries to do (_DeadlockWrap is used everywhere). The next interesting question is whether you do manage to get some concurrent throughput, i.e., you see multiple CPU utilisation while doing concurrent writes, or whether the database locking subsystem is preventing that at all.

It's hard to know in my case as I'm doing a lot of compression with concurrent writes, so that's dominating my CPU time. I have 4 cores doing compression, and one core feeding them and it's all looking like it should. As far as I can tell the locking is pretty fine grained and allowing everything to go ahead pretty nicely.