Chunkwise iteration over arrays. #399
Conversation
alimanfoo
left a comment
alimanfoo
left a comment
There was a problem hiding this comment.
Thanks @jeromekelleher, looks good.
Travis PY27 is failing because in PY2 it's itertools.izip_longest and PY3 it's itertools.zip_longest. Suggest to add something to the zarr.compat module to alias izip_longest to zip_longest under PY2.
Travis PY37 is failing with:
py37 runtests: commands[4] | flake8 zarr
zarr/storage.py:1452:25: E117 over-indented
...that's nothing to do with this PR, although I have no idea how that's crept in, but easiest thing to do would be to fix in this PR by dedenting that line.
Also please add an entry in the release notes (docs/release.rst).
zarr/core.py
Outdated
| chunk_size = self.chunks[0] | ||
| for j in range(self.shape[0]): | ||
| if j % chunk_size == 0: | ||
| chunk = self[j: j + chunk_size][:] |
There was a problem hiding this comment.
| chunk = self[j: j + chunk_size][:] | |
| chunk = self[j: j + chunk_size] |
Additional slice shouldn't be necessary.
There was a problem hiding this comment.
I thought you would need to take an extra copy to be sure the data was kept in a local numpy array. Changed now.
There was a problem hiding this comment.
Thanks, yeah just the single slice [j: j + chunk_size] is enough to bring data into a local numpy array. Adding the extra [:] is harmless but not necessary.
jeromekelleher
force-pushed
the
efficient-iter
branch
from
b716e47 to
1fa04e7
Compare
|
Thanks @alimanfoo, I've implemented those. Love the changelog format btw, I think I'll be copying that! |
2 participants
Closes #398.
Thanks to @alimanfoo for giving me the formula for unit tests! The only tricky thing is the zero-d case, which I've mirrored numpy's behaviour for.
TODO:
tox -e docs)