Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't workingzarr-pythonRelevant to zarr-python upstreamRelevant to zarr-python upstream
Description
ZarrParser doesn't work when the zarr store path is a subdirectory of the object store root (e.g. a zarr store inside a cloud bucket).
import numpy as np
import xarray as xr
from obstore.store import LocalStore
from obspec_utils.registry import ObjectStoreRegistry
from virtualizarr.parsers.zarr import ZarrParser
# Zarr store nested inside the object store root
parent_dir = "/tmp/bucket_root"
storepath = f"{parent_dir}/subdir/data.zarr"
ds = xr.Dataset({"temp": (("x", "y"), np.arange(12, dtype="float32").reshape(3, 4))})
ds.to_zarr(storepath, consolidated=False, zarr_format=2)
store = LocalStore(prefix=parent_dir)
registry = ObjectStoreRegistry({f"file://{parent_dir}": store})
parser = ZarrParser()
parser(url=storepath, registry=registry) # raises GroupNotFoundErrorRoot cause:
Two issues:
- ZarrParser.call discards the store-relative path returned by registry.resolve(), so open_group_async doesn't know where to find the zarr group within the object store.
- The path used to construct chunk manifest entries is the full url rather than the store root URI, causing chunk paths to be doubled (e.g. subdir/data.zarr/subdir/data.zarr/temp/0.0).
Note: A full fix also requires zarr-developers/zarr-python#3657, which fixes ObjectStore.list_dir corrupting directory names when the store path shares characters with child names. That fix is on zarr-python main but not yet released.
FYI @jbusecke this is the full description of the (surprisingly nuanced) bug you found.
Reacted by Julius Busecke
Reacted by Raphael Hagen
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingzarr-pythonRelevant to zarr-python upstreamRelevant to zarr-python upstream
Activity