@@ -778,9 +778,11 @@ chunk size, which will reduce the number of chunks and thus reduce the number of
778778round-trips required to retrieve data for an array (and thus reduce the impact of network
779779latency). Another option is to try to increase the compression ratio by changing
780780compression options or trying a different compressor (which will reduce the impact of
781- limited network bandwidth). As of version 2.2, Zarr also provides the
782- :class: `zarr.storage.LRUStoreCache ` which can be used to implement a local in-memory cache
783- layer over a remote store. E.g.::
781+ limited network bandwidth).
782+
783+ As of version 2.2, Zarr also provides the :class: `zarr.storage.LRUStoreCache `
784+ which can be used to implement a local in-memory cache layer over a remote
785+ store. E.g.::
784786
785787 >>> s3 = s3fs.S3FileSystem(anon=True, client_kwargs=dict(region_name='eu-west-2'))
786788 >>> store = s3fs.S3Map(root='zarr-demo/store', s3=s3, check=False)
@@ -828,6 +830,44 @@ interface to the storage.
828830
829831.. _tutorial_copy :
830832
833+ Consolidating metadata
834+ ~~~~~~~~~~~~~~~~~~~~~~
835+
836+ (This is an experimental feature.)
837+
838+ Since there is a significant overhead for every connection to a cloud object
839+ store such as S3, the pattern described in the previous section may incur
840+ significant latency while scanning the metadata of the dataset hierarchy, even
841+ though each individual metadata object is small. For cases such as these, once
842+ the data are static and can be regarded as read-only, at least for the
843+ metadata/structure of the dataset hierarchy, the many metadata objects can be
844+ consolidated into a single one via
845+ :func: `zarr.convenience.consolidate_metadata `. Doing this can greatly increase
846+ the speed of reading the dataset metadata, e.g.::
847+
848+ >>> zarr.consolidate_metadata(store) # doctest: +SKIP
849+
850+ This creates a special key with a copy of all of the metadata from all of the
851+ metadata objects in the store.
852+
853+ Later, to open a Zarr store with consolidated metadata, use
854+ :func: `zarr.convenience.open_consolidated `, e.g.::
855+
856+ >>> root = zarr.open_consolidated(store) # doctest: +SKIP
857+
858+ This uses the special key to read all of the metadata in a single call to the
859+ backend storage.
860+
861+ Note that, the hierarchy could still be opened in the normal way and altered,
862+ causing the consolidated metadata to become out of sync with the real state of
863+ the dataset hierarchy. In this case,
864+ :func: `zarr.convenience.consolidate_metadata ` would need to be called again.
865+
866+ To protect against consolidated metadata accidentally getting out of sync, the
867+ root group returned by :func: `zarr.convenience.open_consolidated ` is read-only
868+ for the metadata, meaning that no new groups or arrays can be created, and
869+ arrays cannot be resized. However, data values with arrays can still be updated.
870+
831871Copying/migrating data
832872----------------------
833873
0 commit comments