Commit 9783489

The documentation now has a short landing page that points each kind of reader to a guide page, instead of one long page. New guide pages cover models and readers, the rules the readers check, consolidated metadata, repairing known writer bugs, writing an extension, JSON Schema and Pydantic. Each claim the pages make about a Zarr specification links to the section of the spec that makes it, and the consolidated metadata page says what the specs do and do not settle, with links to the zarr-specs pull requests and issues and to the original zarr-python implementation. The pages say "context" for a `Context` where some said "scope".

document and the context it was read in), the readers that return

`zarr_metadata.v2.definition`: the v2 kinds of metadata, the context

context type `Context`. Its module docstring is the guide

pipeline types, and the v3 contexts `CORE_CONTEXT`,

`CORE_AND_EXTENSIONS_CONTEXT` and `ZARR_PYTHON_CONTEXT`

Consolidated metadata is a copy of every metadata document below a group,

kept in one place. A reader can then open a whole hierarchy with one read.

Consolidated metadata is a convention of zarr-python, the reference

implementation. Neither Zarr specification defines it in full. This page

first says what is settled, then describes the API.

**Zarr v2.** The v2 spec has no consolidated metadata.

- The `.zmetadata` file was designed in zarr-python

[PR 268](https://github.com/zarr-developers/zarr-python/pull/268). It

merged in November 2018 and shipped in zarr-python 2.3.0.

- The

[zarr-python 2 tutorial](https://zarr.readthedocs.io/en/v2.18.3/tutorial.html#consolidating-metadata)

describes it.

- zarr-specs [issue 113](https://github.com/zarr-developers/zarr-specs/issues/113),

on standardizing the format, is open.

**Zarr v3.** The core spec names the member but does not define its

entries.

- zarr-specs [PR 309](https://github.com/zarr-developers/zarr-specs/pull/309)

proposes the `consolidated_metadata` member, with the `kind: "inline"`

envelope that zarr-python 3 writes. It follows

[issue 136](https://github.com/zarr-developers/zarr-specs/issues/136). It

has been open since August 2024 and is not merged.

- [PR 373](https://github.com/zarr-developers/zarr-specs/pull/373) merged

in July 2026. Since then, the core spec names the member "for historical

reasons" and fixes its envelope

([spec text](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816)).

The envelope is an object with:

- `must_understand: false`,

- `kind: "inline"`,

- a `metadata` object.

- PR 373 closed

[issue 371](https://github.com/zarr-developers/zarr-specs/issues/371).

That issue found a problem with the v3.1.0 extension rules. Read

strictly, they put every consolidated group zarr-python 3 had written out

of spec, because the envelope has no `name`.

This package reads the envelope the spec fixes. It reads the entries as

zarr-python writes them.

Consolidated metadata holds the hierarchy below its group, with the group

as the root.

- The document of the node at `/a/b` is stored under the key `a/b`. The

[stored representation](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#stored-representation)

strips the leading `/` of a path.

- The documents and the group must form a tree. Only groups have child

nodes, and the parent of every node is present.

`ZarrV3ConsolidatedMetadata` checks this when it reads the documents.

`NodeName` and `NodePath`, in `zarr_metadata.v3`, are string types. They

satisfy the spec's rules for

[node names](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#node-names)

and paths. They are modeled on the zarrs types with the same names. These

functions check a string against those rules:

- `validate_node_name_v3`, `is_node_name_v3` and `parse_node_name_v3`,

- the matching `node_path` functions.

A group does not read the documents its `consolidated_metadata` lists.

Consolidated metadata is a convention of the reference implementation, not

core metadata. So the group checks the

[envelope that the core spec fixes](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816),

then holds the member as written. `CORE_CONTEXT` holds

`CONSOLIDATED_METADATA_EXTRA_FIELD`, the definition that reads this

envelope.

A reader can use the member as a source of child documents, in place of

the store. There are two ways:

- Read one document with `read_node_metadata_v3` when that node is opened.

- Read all of them at once, with one of:

- the group's `consolidated()`,

- `ZarrV3ConsolidatedMetadata`,

- `read_consolidated_metadata_v3`, which returns problems instead of

raising them.

Either way, a document with a problem fails only when it is read.

Which context the documents are read in:

- `consolidated()` uses the group's own context.

- The other three take a `context`. Pass `context=group.context` to use the

group's own context. With no `context`, they use

`CORE_AND_EXTENSIONS_CONTEXT`.

To write consolidated metadata:

1. Build a `ZarrV3ConsolidatedMetadata`.

2. Pass it to the group's `update`.

The group reads the listed models in its own context. It refuses a model

that its context reads differently. Then it writes the document.

`ZarrV3ConsolidatedMetadata` reads every document strictly. It also

accepts node models as entries. It accepts such an entry when the entry's

claims refine into the model's context. Otherwise it refuses the entry at

its path. To consolidate children read in several contexts, consolidate

them in the `union` of those contexts.

A v2 hierarchy's consolidated metadata is the `.zmetadata` document at the

root of the store. zarr-python 2.3 and later write it. It is an object

with:

- `zarr_consolidated_format: 1`,

- a `metadata` object. This maps each metadata file's path to that file's

contents, under keys such as `a/.zarray`, `a/.zattrs` and `a/b/.zgroup`.

The format comes from zarr-python

[PR 268](https://github.com/zarr-developers/zarr-python/pull/268). No

specification defines it.

`ZarrV2ConsolidatedMetadata` reads such a document.

`read_consolidated_metadata_v2` does the same, but returns problems instead

of raising them. Both take a `context`. The default is `CORE_V2_CONTEXT`.

How the entries are read:

- Each `.zarray` or `.zgroup` entry is read as a node, merged with its

sibling `.zattrs`.

- A `.zattrs` with no sibling is kept, but forms no node.

- Any other entry must be JSON, and is kept.

`to_json` returns the entries keyed by file, as written.

The `.zmetadata` that zarr-python 3 writes for a v2 hierarchy has a known

bug. `read_repaired_consolidated_metadata_v2` undoes it. See

[Repairing known writer bugs](repair.md).

The Zarr V3 spec defines the JSON structure of `zarr.json` for arrays and groups exactly. It defines the members that must be present and the JSON types they must have. But the __interpretation__ of those members as actual data types, codecs, chunk grids, etc is not strictly defined by the spec. Relating a JSON value like `"bytes"` to the `bytes` codec requires a mapping from names to codec types.

In the above code snippet we define a `zarr.json` document with three codec declarations. (We will just call them codecs for brevity.) They are named `bytes`, `gzip` and `unknown`. The `bytes` codec here is correctly configured according to the `bytes` codec spec. The `gzip` declaration here violates the `gzip` codec spec because the `level` configuration member is too high. And the `unknown` codec is unknown, it has no specification and cannot be resolved.

- The `unknown` codec is parsed to an `UnclaimedDeclaration`. Nothing in the context claims the name, so it is kept as written and not checked. This is not a problem: the spec [lets a document name codecs](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-codecs) defined outside it, which a reader may not know.

A v3 document may name a codec, data type, chunk grid, chunk key encoding

or storage transformer that this package does not define. Each of these is

an

[extension point](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-points)

of the v3 spec.

The readers leave such a declaration unclaimed. To check it, a program

adds a **definition** to the context it reads in.

The [`zarr_metadata.definition`](api/definition.md) module docstring is the

full reference. This page is the short version.

A definition is a value, not a class to subclass. It has:

- a `name`,

- a `TypedDict` for the configuration's JSON,

- a `rules` function, when the type cannot express every rule. The function

yields the spec violations in a configuration.

A v3 codec definition also says:

- what kind of codec it is,

- whether its output size is fixed by its input size.

```python test="true" session="extensions"

from collections.abc import Iterator

from typing import Annotated, NotRequired

from annotated_types import Ge

from typing_extensions import TypedDict

from zarr_metadata import ValidationProblem

from zarr_metadata.definition import Nested

from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context

class AcmeLz4Configuration(TypedDict, closed=True):

acceleration: Annotated[int, Ge(1)]

dictionary: NotRequired[str]

dictionary_size: NotRequired[Annotated[int, Ge(1)]]

def acme_lz4_rules(

configuration: AcmeLz4Configuration, nested: Nested

) -> Iterator[ValidationProblem]:

if "dictionary" in configuration and "dictionary_size" not in configuration:

yield ValidationProblem(

("dictionary_size",), "a dictionary needs its size", "missing_key"

)

ACME_LZ4 = CodecDefinition(

name="acme.lz4",

configuration=AcmeLz4Configuration,

kind="bytes_bytes",

size="dynamic",

rules=acme_lz4_rules,

)

CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4]))

Pass `context=CONTEXT` to any reader or model constructor. A document that

names `acme.lz4` is then checked against this definition.

**The `TypedDict`**

- It must be a `typing_extensions.TypedDict`.

- It must say how undeclared keys are handled: `closed=True`,

`extra_items=`, or `closed=False`. An open-by-default `TypedDict` is

refused, because it would accept a misspelled key silently.

**Its members** must be JSON shapes:

- `int`, `float`, `bool`, `str`, `None` or `JSONValue`,

- a `Literal`,

- `tuple[T, ...]` or `tuple[T1, T2]`,

- a union,

- a `TypedDict` or `Mapping[str, V]`,

- a `NewType` or a type alias.

A number may carry bounds from annotated-types: `Gt`, `Ge`, `Lt`, `Le` or

`Interval`. Any other `Annotated` metadata is refused when the definition

is built.

**The rules function**

- It runs only on a configuration within its bounds, so it can rely on

them.

- It reports the location of each problem. The value there becomes the

problem's `input`.

- Show values in messages with `zarr_metadata.problems.shown`. Then they

read like the package's own messages.

**Pickling.** Define each function at module top level. A model holds the

definitions that read it, so the model can be pickled only if they can.

Other kinds declare more:

- A data type declares its fill value (`fill_value`, `fill_value_rules`,

`fill_value_canonical`). It also declares how its values are stored

(`storage`).

- A chunk grid declares which shapes it fits (`shape_rules`) and its chunk

lengths (`chunk_lengths`).

- A codec declares:

- what it checks about the chunk it receives (`chunk_rules`),

- what it hands on (`transition`),

- any pipelines it contains (`pipelines`).

The [`zarr_metadata.v3.definition`](api/v3/definition.md) docstring

describes each of these.