Skip to content

how to incorporate zarr's new open_consolidated method? #2558

Closed

Description

@rabernat

Zarr has a new feature called consolidated metadata. This feature will make it much faster to open certain zarr datasets, because all the metadata needed to construct the xarray dataset will live in a single .json file.

To use this new feature, the new function zarr.open_consolidated needs to be called. So it won't work with xarray out of the box. We need to decide how to add support for this at the xarray level.

I am seeking feedback on what API people would like to see before starting a PR. My proposal is to add a new keyword argument to xarray.open_zarr called consolidated (default = False). An alternative would be to automatically try open_consolidated and fall back on the standard open_group function if that fails.

I played around with this a bit and realized that zarr-developers/zarr-python#336 needs to be resolved before we can do the xarray side.

cc @martindurant, who might want to weigh on what would be most convenient for intake.

Activity

  1. shoyer commented on Nov 20, 2018

    @shoyer
    Member

    My proposal is to add a new keyword argument to xarray.open_zarr called consolidated (default = False)

    This sounds like a pretty safe approach to me. (We could imagine later changing this to be the default behavior, if we end up deciding that we usually want to write Zarr datasets with consolidated metadata.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions