Repository navigation
Storing Sparse arrays to Zarr #222
Description
Periodically we have had users request some way to store sparse arrays with Zarr. TBH this is actually pretty doable generally today as demonstrated by this comment. Also this strategy works nicely with Sparse. Admittedly these examples are showing how this works with in-memory Zarr Arrays. Though this would work just as well with any MutableMapping derived store.
Given there seems to be a fair bit of interest in being able to work with more general N-D sparse arrays and having a flexible way to store them, am wondering if it makes sense to provide some functionality in Sparse to store data in Zarr Arrays. Happy to answer any questions. Also would be interested to hear thoughts on this proposal. :)
Activity
hameerabbasi commentedon Dec 15, 2018
Just so I understand here -- Are you proposing making this library a back-end for Zarr or Making Zarr a back-end for this library?
The issue with the latter is that we use Numba to perform a number of operations (including arithmetic and indexing), does Numba work with Zarr?
The other issue is that at this point in time,
DOKis mutable butCOOisn't.hameerabbasi commentedon Dec 15, 2018
It would be nice if you could enumerate what API or changes you would need in this library to make this possible -- I'm definitely willing to work with the Zarr team.
jakirkham commentedon Dec 16, 2018
Short-term having a way to load a Zarr Array into a Sparse array and store a Sparse array into a Zarr Array would be pretty good. These could be similar to the
from_numpyandtodensemethods. The latter case is pretty much solved. It would just benefit from having a convenience method. The former should be solvable any number of ways. As Zarr seems kind of similar toDOK, maybe that would be the easiest way to load it in.Long-term it would be interesting to have a Zarr-backed Sparse array. The main benefits here would be working with larger than memory sparse arrays and/or working with other storage backends. However this will take some more thought as you have noted.
hameerabbasi commentedon Mar 1, 2019
Okay, I just started thinking about this... Since the long-term goal of this project is to have SciPy depend on it, a dependency (even if optional) on Zarr wouldn't be so nice.
Of course, feel free to duck-patch
COOon import so thatsparse.COO.to_zarrandsparse.COO.from_zarrexist and do the right thing. 😄As long as it doesn't rely on fringe functionality of
sparseit should be fine.hameerabbasi commentedon Mar 1, 2019
Of course, I'd recommend patching
SparseArray.?instead, and then doing.toformat(COO)orcls(coo_arr)instead.daletovar commentedon Apr 3, 2019
I recently raised a zarr issue on this zarr-developers/zarr-python#424. I'm not sure what will come of it. Regardless, I like the idea of having the saving functionality live in zarr.