Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save d-v-b/b909a2c2f93eaa3912ef1ac1e38d18f9 to your computer and use it in GitHub Desktop.

Select an option

Save d-v-b/b909a2c2f93eaa3912ef1ac1e38d18f9 to your computer and use it in GitHub Desktop.
zarr-metadata: v3 metadata as (document, context) — design spec

zarr-metadata: v3 metadata as (document, context)

Status: design approved in conversation 2026-10-06; spec awaiting review. Builds on the 370→382 stack and the repair layer (d-v-b/zarr-python#391).

Problem

A v3 metadata model is read in a scope (Context): the definitions that decide what each extension name means. Today the model does not keep its scope. Each field (Read/Unclaimed) keeps the definition that read it, but the model as a whole forgets the scope it was read in. Consequences:

  • update(*, context, **members) must be told a scope again, or it would read new members in a different scope than the old ones.
  • A model can end up holding fields read by different scopes (an update in another scope keeps the held fields and reads the new ones there).
  • A "held fields" mechanism exists only so a model can check itself without re-reading its own fields: held=, own_fields, held_document, _as_read, the NO_SCOPE re-read in __post_init__, and construct.
  • to_json writes fields as readers take them, normalizing spelling, so rewriting a foreign document reformats it.
  • Context is not hashable (MappingProxyType tables), and its equality works only by accident.

Decisions

  1. A v3 model is the pair (document, context). The document is the JSON as written, refined (string keys, arrays as tuples). The context is the scope it was read in. The reading is computed from the pair at construction and cached; it is never set independently.

  2. Equality is by interpretation, not by document text or by context. Two models are equal when every field is read by the same definition with the same canonical configuration, an unclaimed field comparing by name and JSON, the fill value is equal in the canonical spelling its data type's definition gives it (as JSON text when the data type is unclaimed), and every other member is equal as JSON text. This is today's field_key equality. The context is excluded: two scopes that differ only in definitions the document does not use give equal models. Motivating example: a document using bytes read in a scope where bytes is a privately registered codec with other semantics is byte-identical JSON but a different (unequal) model.

  3. Compatibility is an information order, not equality. Per field position:

    • Unclaimed(name, json) ⊑ Read(d, …) when d claims name and reads that JSON (a gain).
    • Read(d, c) ⊑ Read(d, c') only when d is the same definition and c, c' are the same canonical configuration.
    • Read(d, …) and Read(d', …) with d ≠ d' are incomparable (a conflict).
    • Read → Unclaimed is not ⊑ (a loss).

    Model m ⊑ n when every position, nested ones included, holds. m == n is exactly mutual refinement.

  4. Scopes share the order. c ⊑ c' when everything c claims, c' claims identically. Context.joined is the lattice join: the least scope above all inputs, or a conflict. Unclaimed is the bottom element for each name. extended_with stays an explicit override and is deliberately outside the order.

  5. Changing scope has two operations. with_context(c) re-reads the document in c, allowing gains, losses and conflicts. refined_in(c) only moves up the order and raises on any conflict or loss.

  6. One tree, one scope. Documents inside consolidated metadata are views of the group's pair and share its context.

Model API (v3)

ZarrV3ArrayMetadata

A plain immutable class with private slots _document, _context, _reading and _key (the cached equality key). It is no longer a dataclass: a dataclass promises its fields are its state, which they are not.

Construction (only by reading):

  • ZarrV3ArrayMetadata(document, context=None) reads document in context (None means CORE_AND_EXTENSIONS) and raises MetadataValidationError with every problem. self.context is always a Context, never None.
  • from_json(document, *, context=None) and from_key_value(mapping, *, context=None) stay as named constructors.
  • create_default(*, context=None, **members) is unchanged in behavior.
  • read_array_metadata_v3(document, *, context=None) still returns the full reading, with .metadata set when there is no problem.

There is no constructor from typed fields (a deliberate break).

Read-only properties:

  • context.
  • data_type, chunk_grid, chunk_key_encoding (Read | Unclaimed), codecs, storage_transformers (tuples of those), shape, fill_value, dimension_names, attributes and extra_fields (read-only mappings), must_understand_fields, zarr_format, node_type.
  • reading: the full ZarrV3ArrayMetadataReading (pipeline, chunk each codec is handed).
  • claims: Mapping[tuple[type[Definition], str], Definition | None], the definition each (kind, name) the document writes resolved to, None where nothing claimed it. It covers nested fields (a shard's inner codecs, a struct's field types) and keys raw bits by their filed name (r16 → r*).

Changing:

  • update(**members): merge JSON members into the document (UNSET removes one) and read the result in self.context. No scope argument.
  • with_context(context): read the same document in another scope; raises MetadataValidationError if it has problems there. Reuses the cached reading when context.disagreements(self.claims) is empty.
  • refined_in(context): as with_context, but raises ScopeConflictError on any conflict or loss.
  • refines(other) -> bool: whether other ⊑ self. Not <=: Python readers assume a total order, and sorted would misbehave.

Writing:

  • to_json(): a copy of the document as written; spelling is preserved.
  • to_key_value(*, indent=None).

Identity:

  • __eq__ / __hash__ by the cached interpretation key; context excluded.
  • Pickle reduces to (document, context) and reads again on load, so a model's reading can never disagree with its document.

Round-trip property: from_json(m.to_json(), context=m.context) == m. JSON alone does not determine a model.

ZarrV3GroupMetadata

Same shape: ZarrV3GroupMetadata(document, context=None); properties context, attributes, extra_fields, must_understand_fields, consolidated_metadata (ZarrV3ConsolidatedMetadata | UNSET), claims; update, with_context, refined_in, refines, to_json, to_key_value.

ZarrV3ConsolidatedMetadata

A view inside the group's pair. metadata: Mapping[str, ZarrV3NodeMetadata] maps each path to a node model built from the group's reading (not read again), each with the group's context. A nested model's to_json() is the nested document as written. The hierarchy checks (paths, nesting, each group's own listing) stay in the group's read. ZarrV3ConsolidatedMetadata(member, context=None) reads the consolidated_metadata member on its own.

Consolidating from models (rule b)

update(consolidated_metadata=...) and the group constructor accept node models as entries of metadata, as well as JSON. A model given at path p is accepted when its claims ⊑ the group's scope (no conflict, no loss; gains allowed). The group holds the child re-read in its own scope when there are gains, and the child's existing reading otherwise. A gain can surface a problem (a newly claimed zstd refusing a level of 30), which is reported as a ValidationProblem at that path. A conflict or loss is also a ValidationProblem at that path, naming the (kind, name) and both definitions. Consolidating children read in different scopes uses Context.joined(*(child.context for child in children)) to build the parent's scope.

Context API

  • claimant(kind, name) (exists).
  • extended_with(*definitions) (exists): override.
  • disagreements(claims) -> Disagreements: per (kind, name) in claims, whether this scope reads it identically, as a gain, as a conflict, or as a loss. Disagreements exposes .conflicts (conflicts and losses) and .gains.
  • joined(*contexts) -> Context (classmethod): the join; raises ScopeConflictError keyed by (kind, name) when two inputs file different definitions under one name.
  • __eq__ / __hash__ by the set of (kind, name, definition) filed.

Errors

  • A document with problems: MetadataValidationError from the constructor, as today.
  • refined_in with a conflict or loss: ScopeConflictError(ValueError), listing each conflict with its location in the document, its (kind, name), and both definitions (or None).
  • Context.joined with a conflict: ScopeConflictError, keyed by (kind, name).
  • A consolidated child that conflicts with the group's scope: a ValidationProblem at the child's path, inside the group's MetadataValidationError.

Signatures

Every function that takes a scope takes context: Context | None = None (keyword-only where it is today), None meaning CORE_AND_EXTENSIONS: from_json, from_key_value, create_default, read_*, validate_*, is_*, parse_*, node_metadata_*, read_repaired_node_metadata_v3, node_metadata_json_schema_v3. The field-level functions (resolve, canonicalize, field_json_schema) keep a required positional scope.

Integrations

  • pydantic: the validator builds the pair with the scope from pydantic's validation context (or the default); the serializer writes to_json(). A dump → validate round trip needs the same scope passed again.
  • Repair layer: unchanged; it works on documents before a pair exists.
  • JSON Schema: unchanged.
  • v2: unchanged in these three PRs, but not because it has no scope. A v2 document is read in one too, less formally: the dtype strings zarr-python 2.x understood and the codec ids numcodecs 0.16.x decoded form a scope, and the package today interprets neither (a dtype is a string, a codec a mapping with an id). Modeling them as definitions is PR 4; until then the v2 models stay as they are.

Removed

construct, the held-field path (held=, own_fields, held_document, _as_read), the NO_SCOPE re-read, the required context on update, dataclasses.replace / dataclasses.fields support on v3 models, and constructing a v3 model from typed fields.

Testing

One parametrized test per operation over the cases that work; one named test per failure mode. Each test has a docstring stating the behavior.

  • Round trip: from_json(m.to_json(), context=m.context) == m, over the existing document fixtures.
  • Order (Hypothesis over field strategies): refines is reflexive and transitive, and mutual refinement is exactly ==.
  • Equality: the private-bytes example is unequal; two scopes differing only in an unused definition give equal models; "bytes" and {"name": "bytes"} read by the core codec are equal.
  • update reads in the held scope (a private-bytes model stays private after update(attributes=...)).
  • with_context: one test per outcome (gain, loss, conflict all allowed).
  • refined_in: accepts gains; raises on conflict; raises on loss (two tests).
  • Context.joined: the join; raises on conflict.
  • Context equality and hashing by definitions.
  • Consolidated rule b: accepted with gains (held re-read); refused on conflict; refused on loss; a gain surfacing a problem is reported at the child's path.
  • Scope threading: one parametrized test over every public entry point, refusing a document in the default scope and accepting it in an empty one.
  • Pickle: a model round-trips and its reading matches its document.

Staging

Three PRs on top of #391, each on the user's fork:

  1. Scope algebra (additive). Context.__eq__/__hash__, disagreements, joined; claims and the refinement order on fields and readings; ScopeConflictError.
  2. The model becomes the pair. Array, group and consolidated models; properties; update without a scope; with_context, refined_in, refines; cached key; pickle; removal of the held-field machinery and dataclass-ness.
  3. Consolidating from models and uniform signatures. Rule b; context: Context | None = None everywhere.
  4. v2 scopes. See "PR 4: v2 scopes" below: two PRs, 4a (kinds, definitions, CORE_V2, field readers) and 4b (v2 models as pairs).

PR 4: v2 scopes

Decided 2026-10-06. v2 dtypes and codecs are kinds of their own, separate from v3's; unifying a v2 <f4 with a v3 float32 is a higher-level routine, outside this package's model layer.

Kinds. ZarrV2DataTypeDefinition and ZarrV2CodecDefinition, in zarr_metadata/v2/_definition.py, subclass Definition. To admit them, Definition gains three classmethods with v3 defaults, which the v2 kinds override: well_named(name) (v3: the extension-name rule; v2 dtype: the numpy dtype grammar; v2 codec: a numcodecs id), spelled(name) (the filed name and the configuration a name carries; v3: identity but for raw bits; v2 dtype: the family and {"byteorder", "itemsize"[, "unit"]}), and envelope_json(name, configuration) (v3: {"name", "configuration"} or a bare data-type name; v2 codec: {"id": name, **configuration}; v2 dtype: the string). KINDS becomes a registry the v2 module extends. Context, Read/Unclaimed/Refused, field_key, refines, claims_of, disagreements and joined are kind-agnostic and work unchanged.

Data types. One definition per numpy family, filed under the family: bool (|b1), int (i1..i8), uint, float (f2,f4,f8), complex (c8,c16), bytes (|S<n>), str (<U<n>), void (|V<n>), datetime64 and timedelta64 (<M8[unit], <m8[unit]), object (|O), and struct for a tuple of field records, whose field dtypes are read as nested fields, as v3's struct reads its field types. The rules say which item sizes, byte orders and units each family takes. Each definition declares its fill value's JSON shape and rules as the v3 data types do: numbers, "NaN"/"Infinity"/"-Infinity" for floats, base64 strings for bytes, void and struct, null anywhere; and a canonical spelling, so two fill values are one value exactly when they spell alike.

Codecs. One definition per numcodecs id the package models, with a TypedDict of its parameters and rules: zlib, gzip, bz2, lzma, blosc, zstd, lz4, shuffle, delta, fixedscaleoffset, quantize, bitround, astype, packbits, vlen-utf8, vlen-bytes, vlen-array, crc32, crc32c, adler32, fletcher32. A dtype string inside a codec's parameters (astype.encode_dtype, delta.dtype) is a string the v2 dtype grammar accepts, judged by a rule, not a nested field. Any other id (categorize, pickle, msgpack2, json2, base64, jenkins_lookup3) is Unclaimed.

Scope. CORE_V2 = Context.of(*V2_DATA_TYPES, *V2_CODECS) in zarr_metadata.v2.definition. One Context class holds kinds of both formats, since its tables are keyed by kind; the default scopes stay apart: v2 readers default to CORE_V2, v3 readers to CORE_AND_EXTENSIONS.

Readers. resolve_dtype_v2(value, context) and resolve_codec_v2(value, context) read one field each, giving Read, Unclaimed or Refused with problems, as resolve does for v3. The v2 array validator reads dtype, compressor and filters through them, and judges fill_value against the dtype as the v3 validator judges it against data_type.

Models (PR 4b). ZarrV2ArrayMetadata(document, context=None) becomes a pair as the v3 models are: dtype a Read | Unclaimed, compressor a Read | Unclaimed | None, filters a tuple of those or None; update, with_context, refined_in, refines, claims; equality by interpretation, as v3; construct is removed. ZarrV2GroupMetadata and ZarrV2ConsolidatedMetadata follow, the consolidated model's .zarray and .zgroup entries as models of its scope. Every v2 reader takes context: Context | None = None.

Staging. PR 4a: the Definition classmethods and registry, the two kinds, the definitions, CORE_V2, the two field readers, and the v2 validator reading through them (additive: a valid v2 document stays valid; invalid dtypes and codec parameters that were unjudged become problems). PR 4b: the models as pairs and the reader signatures (breaking).

Future direction: broken documents as a scope

The repair layer (#391) models the broken documents it accepts as TypedDicts, one per writer bug, and repairs JSON before a strict read. With models as (document, context), the space of accepted broken documents can itself be a scope:

  • REPAIRABLE = CORE_AND_EXTENSIONS.extended_with(LENIENT_REGULAR_GRID, ...), where LENIENT_REGULAR_GRID admits a chunk length of 0 along a dimension of length 0 (its shape rules see the shape). A document zarr-python 3.0 or 3.1 wrote reads without problems in REPAIRABLE, so "the documents we accept" is exactly "the documents that read in REPAIRABLE".
  • A lenient definition carries a repaired transform to the strict definition it stands in for. Repair is a named operation on the pair, m.repaired_in(strict_context), mapping each field read by a lenient definition through its transform and reading the result in the strict scope. The repairs made are listed as today.
  • The order stays honest: a lenient and a strict regular are different definitions under one name, so they conflict, and refined_in refuses to move between them. Repair is an explicit transform, not a refinement.

Limit: a scope holds extension definitions, so only bugs inside a field can be modeled this way. A document-level bug, such as consolidated_metadata: null, is outside any field. Either the scope grows to cover the document grammar, or document-level repairs stay as JSON transforms before the pair exists. Decide when a second document-level bug appears. Not part of the three PRs above.

Open questions

  • Whether a canonical-document writer (the whole document in canonical spelling) is wanted; deferred until a caller needs it. canonical_of already spells one field.
  • Whether with_context should warn on losses; deferred until use shows whether silent losses surprise people.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment