Skip to content

ZarrParser fails when zarr store is nested inside object store root #912

Closed

Description

@TomNicholas

ZarrParser doesn't work when the zarr store path is a subdirectory of the object store root (e.g. a zarr store inside a cloud bucket).

import numpy as np
import xarray as xr
from obstore.store import LocalStore
from obspec_utils.registry import ObjectStoreRegistry                                                                                                          
from virtualizarr.parsers.zarr import ZarrParser                                                                                                               
                                                                                                                                                                
# Zarr store nested inside the object store root                                                                                                               
parent_dir = "/tmp/bucket_root"
storepath = f"{parent_dir}/subdir/data.zarr"                                                                                                                    
                                                                                                                                                                
ds = xr.Dataset({"temp": (("x", "y"), np.arange(12, dtype="float32").reshape(3, 4))})                                                                          
ds.to_zarr(storepath, consolidated=False, zarr_format=2)                                                                                                        
                                                                                                                                                                
store = LocalStore(prefix=parent_dir)                                                                                                                          
registry = ObjectStoreRegistry({f"file://{parent_dir}": store})
                                                                                                                                                                
parser = ZarrParser()
parser(url=storepath, registry=registry)  # raises GroupNotFoundError

Root cause:

Two issues:

  1. ZarrParser.call discards the store-relative path returned by registry.resolve(), so open_group_async doesn't know where to find the zarr group within the object store.
  2. The path used to construct chunk manifest entries is the full url rather than the store root URI, causing chunk paths to be doubled (e.g. subdir/data.zarr/subdir/data.zarr/temp/0.0).

Note: A full fix also requires zarr-developers/zarr-python#3657, which fixes ObjectStore.list_dir corrupting directory names when the store path shares characters with child names. That fix is on zarr-python main but not yet released.

FYI @jbusecke this is the full description of the (surprisingly nuanced) bug you found.

Activity

  1. added
    bugSomething isn't working
    zarr-pythonRelevant to zarr-python upstream
    on Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingzarr-pythonRelevant to zarr-python upstream

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions