Status: design approved in conversation 2026-10-06; spec awaiting review. Builds on the 370→382 stack and the repair layer (d-v-b/zarr-python#391).
A v3 metadata model is read in a scope (Context): the definitions that
decide what each extension name means. Today the model does not keep its
scope. Each field (Read/Unclaimed) keeps the definition that read it, but
the model as a whole forgets the scope it was read in. Consequences:
update(*, context, **members)must be told a scope again, or it would read new members in a different scope than the old ones.- A model can end up holding fields read by different scopes (an
updatein another scope keeps the held fields and reads the new ones there). - A "held fields" mechanism exists only so a model can check itself without
re-reading its own fields:
held=,own_fields,held_document,_as_read, theNO_SCOPEre-read in__post_init__, andconstruct. to_jsonwrites fields as readers take them, normalizing spelling, so rewriting a foreign document reformats it.Contextis not hashable (MappingProxyTypetables), and its equality works only by accident.
-
A v3 model is the pair
(document, context). The document is the JSON as written, refined (string keys, arrays as tuples). The context is the scope it was read in. The reading is computed from the pair at construction and cached; it is never set independently. -
Equality is by interpretation, not by document text or by context. Two models are equal when every field is read by the same definition with the same canonical configuration, an unclaimed field comparing by name and JSON, the fill value is equal in the canonical spelling its data type's definition gives it (as JSON text when the data type is unclaimed), and every other member is equal as JSON text. This is today's
field_keyequality. The context is excluded: two scopes that differ only in definitions the document does not use give equal models. Motivating example: a document usingbytesread in a scope wherebytesis a privately registered codec with other semantics is byte-identical JSON but a different (unequal) model. -
Compatibility is an information order, not equality. Per field position:
Unclaimed(name, json)⊑Read(d, …)whendclaimsnameand reads that JSON (a gain).Read(d, c)⊑Read(d, c')only whendis the same definition andc,c'are the same canonical configuration.Read(d, …)andRead(d', …)withd ≠ d'are incomparable (a conflict).Read→Unclaimedis not ⊑ (a loss).
Model
m ⊑ nwhen every position, nested ones included, holds.m == nis exactly mutual refinement. -
Scopes share the order.
c ⊑ c'when everythingcclaims,c'claims identically.Context.joinedis the lattice join: the least scope above all inputs, or a conflict. Unclaimed is the bottom element for each name.extended_withstays an explicit override and is deliberately outside the order. -
Changing scope has two operations.
with_context(c)re-reads the document inc, allowing gains, losses and conflicts.refined_in(c)only moves up the order and raises on any conflict or loss. -
One tree, one scope. Documents inside consolidated metadata are views of the group's pair and share its context.
A plain immutable class with private slots _document, _context,
_reading and _key (the cached equality key). It is no longer a dataclass:
a dataclass promises its fields are its state, which they are not.
Construction (only by reading):
ZarrV3ArrayMetadata(document, context=None)readsdocumentincontext(NonemeansCORE_AND_EXTENSIONS) and raisesMetadataValidationErrorwith every problem.self.contextis always aContext, neverNone.from_json(document, *, context=None)andfrom_key_value(mapping, *, context=None)stay as named constructors.create_default(*, context=None, **members)is unchanged in behavior.read_array_metadata_v3(document, *, context=None)still returns the full reading, with.metadataset when there is no problem.
There is no constructor from typed fields (a deliberate break).
Read-only properties:
context.data_type,chunk_grid,chunk_key_encoding(Read | Unclaimed),codecs,storage_transformers(tuples of those),shape,fill_value,dimension_names,attributesandextra_fields(read-only mappings),must_understand_fields,zarr_format,node_type.reading: the fullZarrV3ArrayMetadataReading(pipeline, chunk each codec is handed).claims:Mapping[tuple[type[Definition], str], Definition | None], the definition each(kind, name)the document writes resolved to,Nonewhere nothing claimed it. It covers nested fields (a shard's inner codecs, a struct's field types) and keys raw bits by their filed name (r16→r*).
Changing:
update(**members): merge JSON members into the document (UNSETremoves one) and read the result inself.context. No scope argument.with_context(context): read the same document in another scope; raisesMetadataValidationErrorif it has problems there. Reuses the cached reading whencontext.disagreements(self.claims)is empty.refined_in(context): aswith_context, but raisesScopeConflictErroron any conflict or loss.refines(other) -> bool: whetherother ⊑ self. Not<=: Python readers assume a total order, andsortedwould misbehave.
Writing:
to_json(): a copy of the document as written; spelling is preserved.to_key_value(*, indent=None).
Identity:
__eq__/__hash__by the cached interpretation key; context excluded.- Pickle reduces to
(document, context)and reads again on load, so a model's reading can never disagree with its document.
Round-trip property: from_json(m.to_json(), context=m.context) == m.
JSON alone does not determine a model.
Same shape: ZarrV3GroupMetadata(document, context=None); properties
context, attributes, extra_fields, must_understand_fields,
consolidated_metadata (ZarrV3ConsolidatedMetadata | UNSET), claims;
update, with_context, refined_in, refines, to_json,
to_key_value.
A view inside the group's pair. metadata: Mapping[str, ZarrV3NodeMetadata]
maps each path to a node model built from the group's reading (not read
again), each with the group's context. A nested model's to_json() is the
nested document as written. The hierarchy checks (paths, nesting, each
group's own listing) stay in the group's read.
ZarrV3ConsolidatedMetadata(member, context=None) reads the
consolidated_metadata member on its own.
update(consolidated_metadata=...) and the group constructor accept node
models as entries of metadata, as well as JSON. A model given at path p
is accepted when its claims ⊑ the group's scope (no conflict, no loss;
gains allowed). The group holds the child re-read in its own scope when
there are gains, and the child's existing reading otherwise. A gain can
surface a problem (a newly claimed zstd refusing a level of 30), which is
reported as a ValidationProblem at that path. A conflict or loss is also a
ValidationProblem at that path, naming the (kind, name) and both
definitions. Consolidating children read in different scopes uses
Context.joined(*(child.context for child in children)) to build the
parent's scope.
claimant(kind, name)(exists).extended_with(*definitions)(exists): override.disagreements(claims) -> Disagreements: per(kind, name)inclaims, whether this scope reads it identically, as a gain, as a conflict, or as a loss.Disagreementsexposes.conflicts(conflicts and losses) and.gains.joined(*contexts) -> Context(classmethod): the join; raisesScopeConflictErrorkeyed by(kind, name)when two inputs file different definitions under one name.__eq__/__hash__by the set of(kind, name, definition)filed.
- A document with problems:
MetadataValidationErrorfrom the constructor, as today. refined_inwith a conflict or loss:ScopeConflictError(ValueError), listing each conflict with its location in the document, its(kind, name), and both definitions (orNone).Context.joinedwith a conflict:ScopeConflictError, keyed by(kind, name).- A consolidated child that conflicts with the group's scope: a
ValidationProblemat the child's path, inside the group'sMetadataValidationError.
Every function that takes a scope takes context: Context | None = None
(keyword-only where it is today), None meaning CORE_AND_EXTENSIONS:
from_json, from_key_value, create_default, read_*, validate_*,
is_*, parse_*, node_metadata_*, read_repaired_node_metadata_v3,
node_metadata_json_schema_v3. The field-level functions (resolve,
canonicalize, field_json_schema) keep a required positional scope.
- pydantic: the validator builds the pair with the scope from pydantic's
validation context (or the default); the serializer writes
to_json(). A dump → validate round trip needs the same scope passed again. - Repair layer: unchanged; it works on documents before a pair exists.
- JSON Schema: unchanged.
- v2: unchanged in these three PRs, but not because it has no scope. A v2
document is read in one too, less formally: the dtype strings
zarr-python 2.x understood and the codec ids numcodecs 0.16.x decoded
form a scope, and the package today interprets neither (a
dtypeis a string, a codec a mapping with anid). Modeling them as definitions is PR 4; until then the v2 models stay as they are.
construct, the held-field path (held=, own_fields, held_document,
_as_read), the NO_SCOPE re-read, the required context on update,
dataclasses.replace / dataclasses.fields support on v3 models, and
constructing a v3 model from typed fields.
One parametrized test per operation over the cases that work; one named test per failure mode. Each test has a docstring stating the behavior.
- Round trip:
from_json(m.to_json(), context=m.context) == m, over the existing document fixtures. - Order (Hypothesis over field strategies):
refinesis reflexive and transitive, and mutual refinement is exactly==. - Equality: the private-
bytesexample is unequal; two scopes differing only in an unused definition give equal models;"bytes"and{"name": "bytes"}read by the core codec are equal. updatereads in the held scope (a private-bytesmodel stays private afterupdate(attributes=...)).with_context: one test per outcome (gain, loss, conflict all allowed).refined_in: accepts gains; raises on conflict; raises on loss (two tests).Context.joined: the join; raises on conflict.Contextequality and hashing by definitions.- Consolidated rule b: accepted with gains (held re-read); refused on conflict; refused on loss; a gain surfacing a problem is reported at the child's path.
- Scope threading: one parametrized test over every public entry point, refusing a document in the default scope and accepting it in an empty one.
- Pickle: a model round-trips and its reading matches its document.
Three PRs on top of #391, each on the user's fork:
- Scope algebra (additive).
Context.__eq__/__hash__,disagreements,joined;claimsand the refinement order on fields and readings;ScopeConflictError. - The model becomes the pair. Array, group and consolidated models;
properties;
updatewithout a scope;with_context,refined_in,refines; cached key; pickle; removal of the held-field machinery and dataclass-ness. - Consolidating from models and uniform signatures. Rule b;
context: Context | None = Noneeverywhere. - v2 scopes. See "PR 4: v2 scopes" below: two PRs, 4a (kinds,
definitions,
CORE_V2, field readers) and 4b (v2 models as pairs).
Decided 2026-10-06. v2 dtypes and codecs are kinds of their own, separate
from v3's; unifying a v2 <f4 with a v3 float32 is a higher-level
routine, outside this package's model layer.
Kinds. ZarrV2DataTypeDefinition and ZarrV2CodecDefinition, in
zarr_metadata/v2/_definition.py, subclass Definition. To admit them,
Definition gains three classmethods with v3 defaults, which the v2 kinds
override: well_named(name) (v3: the extension-name rule; v2 dtype: the
numpy dtype grammar; v2 codec: a numcodecs id), spelled(name) (the filed
name and the configuration a name carries; v3: identity but for raw bits;
v2 dtype: the family and {"byteorder", "itemsize"[, "unit"]}), and
envelope_json(name, configuration) (v3: {"name", "configuration"} or
a bare data-type name; v2 codec: {"id": name, **configuration}; v2
dtype: the string). KINDS becomes a registry the v2 module extends.
Context, Read/Unclaimed/Refused, field_key, refines,
claims_of, disagreements and joined are kind-agnostic and work
unchanged.
Data types. One definition per numpy family, filed under the family:
bool (|b1), int (i1..i8), uint, float (f2,f4,f8), complex
(c8,c16), bytes (|S<n>), str (<U<n>), void (|V<n>),
datetime64 and timedelta64 (<M8[unit], <m8[unit]), object
(|O), and struct for a tuple of field records, whose field dtypes are
read as nested fields, as v3's struct reads its field types. The rules
say which item sizes, byte orders and units each family takes. Each
definition declares its fill value's JSON shape and rules as the v3 data
types do: numbers, "NaN"/"Infinity"/"-Infinity" for floats, base64
strings for bytes, void and struct, null anywhere; and a canonical
spelling, so two fill values are one value exactly when they spell alike.
Codecs. One definition per numcodecs id the package models, with a
TypedDict of its parameters and rules: zlib, gzip, bz2, lzma,
blosc, zstd, lz4, shuffle, delta, fixedscaleoffset,
quantize, bitround, astype, packbits, vlen-utf8, vlen-bytes,
vlen-array, crc32, crc32c, adler32, fletcher32. A dtype string
inside a codec's parameters (astype.encode_dtype, delta.dtype) is a
string the v2 dtype grammar accepts, judged by a rule, not a nested field.
Any other id (categorize, pickle, msgpack2, json2, base64,
jenkins_lookup3) is Unclaimed.
Scope. CORE_V2 = Context.of(*V2_DATA_TYPES, *V2_CODECS) in
zarr_metadata.v2.definition. One Context class holds kinds of both
formats, since its tables are keyed by kind; the default scopes stay
apart: v2 readers default to CORE_V2, v3 readers to
CORE_AND_EXTENSIONS.
Readers. resolve_dtype_v2(value, context) and
resolve_codec_v2(value, context) read one field each, giving Read,
Unclaimed or Refused with problems, as resolve does for v3. The v2
array validator reads dtype, compressor and filters through them,
and judges fill_value against the dtype as the v3 validator judges it
against data_type.
Models (PR 4b). ZarrV2ArrayMetadata(document, context=None) becomes
a pair as the v3 models are: dtype a Read | Unclaimed, compressor a
Read | Unclaimed | None, filters a tuple of those or None; update,
with_context, refined_in, refines, claims; equality by
interpretation, as v3; construct is removed. ZarrV2GroupMetadata and
ZarrV2ConsolidatedMetadata follow, the consolidated model's .zarray
and .zgroup entries as models of its scope. Every v2 reader takes
context: Context | None = None.
Staging. PR 4a: the Definition classmethods and registry, the two
kinds, the definitions, CORE_V2, the two field readers, and the v2
validator reading through them (additive: a valid v2 document stays
valid; invalid dtypes and codec parameters that were unjudged become
problems). PR 4b: the models as pairs and the reader signatures
(breaking).
The repair layer (#391) models the broken documents it accepts as
TypedDicts, one per writer bug, and repairs JSON before a strict read. With
models as (document, context), the space of accepted broken documents can
itself be a scope:
REPAIRABLE = CORE_AND_EXTENSIONS.extended_with(LENIENT_REGULAR_GRID, ...), whereLENIENT_REGULAR_GRIDadmits a chunk length of 0 along a dimension of length 0 (its shape rules see the shape). A document zarr-python 3.0 or 3.1 wrote reads without problems inREPAIRABLE, so "the documents we accept" is exactly "the documents that read inREPAIRABLE".- A lenient definition carries a
repairedtransform to the strict definition it stands in for. Repair is a named operation on the pair,m.repaired_in(strict_context), mapping each field read by a lenient definition through its transform and reading the result in the strict scope. The repairs made are listed as today. - The order stays honest: a lenient and a strict
regularare different definitions under one name, so they conflict, andrefined_inrefuses to move between them. Repair is an explicit transform, not a refinement.
Limit: a scope holds extension definitions, so only bugs inside a field
can be modeled this way. A document-level bug, such as
consolidated_metadata: null, is outside any field. Either the scope grows
to cover the document grammar, or document-level repairs stay as JSON
transforms before the pair exists. Decide when a second document-level
bug appears. Not part of the three PRs above.
- Whether a canonical-document writer (the whole document in canonical
spelling) is wanted; deferred until a caller needs it.
canonical_ofalready spells one field. - Whether
with_contextshould warn on losses; deferred until use shows whether silent losses surprise people.