Repository navigation
keys/items yields two diferent results between v2 and v3 on ZipStore #2757
Description
Zarr version
Numcodecs version
0.15.0
Python Version
3.12
Operating System
Mac ARM
Installation
uv
Description
I have an admittedly old zarr zip store where the keys from one of the subgroups appears to be yielding itself in zarr v3 (erroneously as it errors out on access) but correctly in zarr v2. I can't reproduce this just by creating groups in v3 or v2, with either the current main branch or the final v2 commit.
It's possible the file is corrupted or something, tough to say since it is a zip store. The same problem does not happen when I unzip the store and either visually examine its contents or when I actually open the store using zarr.open and then run .keys()
Steps to reproduce
For the above file one should run in v2:
import zarr
g = zarr.open('adata.zarr.zip', mode="r")
list(g["obsm"].keys()) # ['array', 'df', 'sparse']
g["obsm"]["obsm"] # not there!and then
import zarr
from zarr.storage import ZipStore
g = zarr.open(ZipStore('/Users/ilangold/Projects/Theis/anndata/tests/data/archives/v0.7.0/adata.zarr.zip'), mode="r")
list(g["obsm"].keys()) # ['array', 'df', 'sparse', 'obsm']
g["obsm"]["obsm"] # also not there!Additional output
No response
Activity
d-v-b commentedon Jan 24, 2025
can you list the contents of the zip store, as zarr sees it? there's an async method on the
ZipStoreclass that should do this (I think if you calllist_prefix('')it should list everything), and you can callsyncaround it to invoke that method from non-async code. It wouldn't surprise me if this is due to a directory listing bugilan-gold commentedon Jan 24, 2025
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[16], line 1 ----> 1 [item async for item in store.list_prefix("")] File ~/Projects/Theis/zarr-python/src/zarr/storage/_zip.py:256, in ZipStore.list_prefix(self, prefix) 254 async def list_prefix(self, prefix: str) -> AsyncIterator[str]: 255 # docstring inherited --> 256 async for key in self.list(): 257 if key.startswith(prefix): 258 yield key File ~/Projects/Theis/zarr-python/src/zarr/storage/_zip.py:250, in ZipStore.list(self) 248 async def list(self) -> AsyncIterator[str]: 249 # docstring inherited --> 250 with self._lock: 251 for key in self._zf.namelist(): 252 yield key AttributeError: 'ZipStore' object has no attribute '_lock'This is from an
ipythonshell. Maybe I should try a lil test script?d-v-b commentedon Jan 24, 2025
zipstore is a little special, you might need to invoke a special method to completely create one (yes, this is cursed)
ilan-gold commentedon Jan 24, 2025
Just from
obsm(there are many subgroups), it does not list itselfobsmas a subgroup, askeysdoes:d-v-b commentedon Jan 24, 2025
so the stuff coming out of
keyscomes frommembers, which wraps some recursive functions that might not be the easiest to follow. I think the relevant line is here:zarr-python/src/zarr/core/group.py
Line 2809 in 0c154c3
that's where the to-be-recursed function lists the keys it can see at a given level of the hierarchy. this method, and the line I linked, are both async generators, which are kind of annoying to invoke in non-async code. you will need another helper function (
_collect_aiterator, sorry for the awful name), which you can also bring in fromsync.e.g., to replicate that
keysline yourself in non-async code, you would runsync(_collect_aiterator(group.store.list_dir(group.store.path))). See ifobsmappears there when it shouldntilan-gold commentedon Jan 24, 2025
@d-v-b For me, this old zip store is producing "groups" as keys from
g.store._zf.namelist()i.e.,obsmis returned as a key fromnamelisteven though it is only a "directory" within the zip. It's possible this file was created in some way that yielded this irregularity. I'm not entirely sure. It seems clear from the docs thatnamelistshould return only the file names.That being said, there is a check for this in the
ZipStorethat is wrong because these directory names end up containing a trailing slash that is not accounted for. I will open a PR and if you want, I can add this test file in (although it might be worth trying to create one from scratch even if I haven't yet figured out how).[-]`keys`/`items` yields two diferent results between v2 and v3[/-][+]`keys`/`items` yields two diferent results between v2 and v3 on `ZipStore`[/+]