Skip to content

Commit 8dc50b8

Browse files
d-v-bclaude
andauthored
feat(zarr-metadata): metadata fields read against their definitions (#4434)
* feat(zarr-metadata): must_understand false refused at every extension point Ignoring a codec gives wrong bytes as surely as ignoring a data type gives wrong values, so no extension point may be declared ignorable: `validate_array_metadata_v3` now refuses `must_understand: false` on a codec or a storage transformer, as it already did on a data type, a chunk grid and a chunk key encoding. The spec names only those three, which this package reads as an oversight rather than a licence. `must_understand: false` keeps its meaning where it has one: an unknown top-level extension field, which a reader really can skip. It changes verdicts and nothing else depends on it, so it stands on its own as a reading of the spec. The definition layer above reads every metadata field under the same policy. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): the fragment for must_understand false refused everywhere Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): the array schema refuses must_understand false where the validator does The JSON schema the pydantic integration generates typed a codec and a storage transformer with the envelope that allows `must_understand: false`, so it accepted an array document the validator refuses. Every extension point of an array document now has the mandatory envelope, and the test that holds schema and runtime together covers all five. A metadata field read on its own keeps the general envelope, as `parse_metadata_field_v3` does. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): cite the spec lines that name the three points The comment linked the section's opening rules, not the sentence that limits `must_understand=False` to three extension points. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): the fragment takes this PR's number and says what breaks Assisted-by: ClaudeCode:claude-opus-5-5 * feat(zarr-metadata): one metadata field read against its definition The one-field layer, JSON first: the type check stands alone and feeds the rules, and nothing is built from a configuration to judge it. A definition is a value: the name the metadata carries, the TypedDict its configuration is -- the one declaration of that JSON, which the checker is compiled from -- and plain functions over that TypedDict for the rules. Its kind is its type (`CodecDefinition`, `ChunkGridDefinition`, ...), and a generic frozen dataclass checks itself in `__post_init__`: no `__init_subclass__`, no metaclass, no abstract base. A scope holds definitions by kind. Three steps, each usable on its own by a caller holding JSON: `check` (JSON against a TypedDict, no scope: #347's, re-exported), `Definition.judge` (the check, each nested field's envelope judged -- a member annotated `CodecField` holds one -- then the rules), and `resolve` (a whole field in a scope, returning `Resolved` with a `Resolution` and every problem located). gzip, bytes, crc32c and the regular grid are defined. What the rules are handed is what the type says: a configuration holding the keys its TypedDict admits and nothing else, whose nested fields' envelopes are sound, so a rule may index a table by key or read a nested field's name; a rule's problems land in the configuration in `judge` and `resolve` alike. The checker hands nested fields back in the typed value, so the branch of a union that did not match reads none. A definition refuses at build what it could not read with -- a TypedDict that says nothing of the keys it does not declare, a member typed `ZarrV3MetadataFieldJSON`, an annotation that does not resolve, a name that is not a string, a codec kind outside the three, rules or `names` that are not functions -- and a field is read as one of the five kinds, with or without type arguments, anything else a `TypeError`. `configuration_of` types a read configuration as its definition's TypedDict, and the configuration's type parameter has a default, so `resolve(field, CodecDefinition, scope)` infers nothing unknown. The definitions read JSON and its problems from `zarr_metadata._json`, and a metadata field's own validators from `zarr_metadata.v3._common`, where they move from the model, so nothing this layer adds imports the model it supports. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(zarr-metadata): every extension as a definition Layer 2b interpolates 2a's single field across the package: each codec, data type, chunk grid and chunk key encoding is a definition -- its name, the TypedDict main already declares for its configuration, and its rules as a function over that TypedDict. - Codecs: blosc, zstd, transpose, scale_offset, sharding_indexed and cast_value join gzip, bytes and crc32c. A shard's two pipelines and a cast's target type are nested fields: their TypedDicts annotate them `CodecField` and `DataTypeField`, statically the same envelope type, and they are read in the scope the containing field is read in. - Data types: the sixteen with nothing to configure take `EmptyConfiguration`; the numpy time types share their scale-factor rule; struct judges its field names; the raw-bytes family claims every `r<N>` and judges its width. - The rectilinear grid, and the default and v2 chunk key encodings. - `canonical` per definition, and `canonicalize` for a field in scope: blosc drops a `typesize` that `noshuffle` ignores, rectilinear run-length encodes, and nested fields and envelopes simplify without code of their own. The configuration it starts from holds what its TypedDict admits at every depth, and what `canonical` gives is judged again: one that does not hold is a `ValueError`, the definition's fault rather than the field's. `CORE` holds the 24 the specification defines and `CORE_AND_EXTENSIONS` the 9 more zarr-extensions registers, as on #4379. Against #4379's entity layer, over every metadata field in the 40k-document corpus (157,254): the same resolution for every field, and the same canonical spelling for all 114,445 that read. Problems and messages are the same for all but 9, sharding codecs with a stray member in an inner codec's envelope: the shard's own rule is not asked of a configuration whose nested envelopes are not sound, as `judge` would not ask it, so its problem waits until the envelope is fixed. Deferred to the layer that reads types together: whether a struct field's type is fixed-size. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): the fragments for metadata fields read against their definitions 2a and 2b land together, here: alone, 2a's scopes hold four definitions and read a core data type as out of scope. So 2a's fragment is named for this PR, beside 2b's. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): a field with a problem has no simplest spelling `canonicalize` gave a spelling for a field with an unknown key, a stray envelope member or a `must_understand` of `false`, reporting the problem beside a form that had quietly dropped what the author wrote: `{"name": "bytes", "configuration": {"endain": "big"}}` came back as `"bytes"`. Now a field with any problem comes back None, with its problems. A field with an unknown key is still read, and `resolve` still hands back its configuration, so a consumer that tolerates one filters by kind; the guide says the field is valid only when there is no problem at all. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(zarr-metadata): resolve refuses must_understand false wherever the model does The definition layer judges a field's envelope with the same refusal the model's validator applies at each extension point. One test now holds the two together at all five points, so a change to the reading that reaches one and not the other fails here. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): one fragment for metadata fields read against their definitions The two fragments were 2a's and 2b's, and the first still said the scopes hold four definitions. #350 lands both, so one fragment says what it lands. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): null is not a verdict for a field or a configuration `resolve(None, ...)` answered "invalid" with no problem, and `Definition.judge(None)` and `canonicalize(None, ...)` answered with no value and no problem. `refine_json` gives JSON's `null` as None with no problem, and each read a None refinement as "not JSON, and its problems say why". They now ask the problems whether the value is JSON, so `null` is read like any other value, and refused as a value of the wrong type. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): a reader's definition takes a name over from a family `CORE_AND_EXTENSIONS.extended_with(my_r16)` still read `r16` with the raw-bytes family, since `claimant` asked the definitions in the order they were filed and the family came first, so a reader could not substitute its own reading of one name a family claims. It now asks the one filed last first, as `Context.of` and `extended_with` say: a definition passed later takes a name over, whether the earlier one was filed under it or claims it as a family. No two definitions the package files claim the same name, so its own scopes read as before. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): a regular grid's chunk extent may be 0 The core spec: "The chunk shape elements are non-zero when the corresponding dimensions of the arrays have non-zero length." So 0 is right on a dimension of length 0 -- zarr-python 3.0 and 3.1 wrote it, and this package's own `create_default(shape=(0, 4))` does -- and a grid alone cannot tell that dimension from one of non-zero length. The field now refuses only a negative extent; an extent of 0 on a dimension that is not empty is for the layer that reads the grid beside the shape. Needed before the model reads grids through their definitions: without it the model would refuse to read back a zero-length array it wrote. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(zarr-metadata): no re-check of a TypedDict a definition vetted `_configuration_checked` refused a shape that is not a TypedDict, but its one caller hands it a definition's configuration, which the definition vetted when it was built. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): a definition's rules are checked where they are called The rules are an extension author's code, and a slip there surfaced far from it: a rule yielding a string failed on `.loc`, a plain function that forgot to yield failed as `'NoneType' object is not iterable` on valid documents, and a rule that raised let a bare `KeyError` out of a validator with nothing to say whose it was. What the rules yield is now checked to be what they declare, naming the definition, and an error one raises carries a note naming the definition and where it was reading. The refusal of an open TypedDict, and the guide, name `typing_extensions.TypedDict`, since `typing.TypedDict` takes no `closed` on the versions the package supports. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): a family's name rules are checked as its rules are A family's `name_rules` is the extension author's code too, and a name rule yielding a string failed on `.loc`, as a rule did before `_ruled`. `_ruled` now takes the call, and reads both through the same checks: what they yield is checked, and an error one raises carries a note naming the definition. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): a definition filed again is the one filed last `claimant` asks the definition filed last first, but a name filed again kept its first place in its table, so extending a scope with a family it already held did not let the family take a name back from a definition filed after it. A name filed again now moves to the end. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(zarr-metadata): codecs say whether their size is static; a shard's index takes static codecs The sharding spec says codecs that produce a variable-sized encoded representation, such as compressors, MUST NOT be used for index codecs, and nothing checked it: a shard with a gzip in `index_codecs` read. A `CodecDefinition` now says its `size`, `"static"` or `"dynamic"`: whether the size of what it gives out is fixed by the size of what it is handed. It is required, and checked when the definition is built, as `kind` is. The package's codecs say what zarr-python's `is_fixed_size` says: `bytes`, `crc32c`, `transpose`, `cast_value` and `scale_offset` are static; `blosc`, `gzip`, `zstd` and `sharding_indexed`, whose empty inner chunks take no bytes, are dynamic. A member annotated `StaticCodecField` takes codecs of static size only. The shard's TypedDict declares `index_codecs: tuple[StaticCodecField, ...]`, so the constraint lives in the one declaration of that JSON, and any extension's container codec can say the same. Where a nested field is read in a scope, a codec of dynamic size in such a member is an `invalid_value` at its place. A name nothing in scope claims is left unjudged, its size unknown with the rest of it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(zarr-metadata): the every-definition tests say what they read The module docstring named the fork's layer, which a reader of this package has no reason to know. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): the definitions' fragment takes this PR's number Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(zarr-metadata): raw bits read as r* with the size their name carries `r16` reads as the definition filed under `r*`, the name the spec's data types table gives raw bits, with `{"bits": 16}` as its configuration; a size that is not a positive multiple of 8 is a rule over that configuration, reported at the field. Every name a scope looks up is now exact, so `Context.claimant` is a dict lookup, and `Definition` loses `names` and `name_rules`, which only raw bits used. A reader reads raw bits its own way by defining `r*`; a data type named `r16` is refused, since that name reads as `r*`. `canonicalize` writes the size back into the name, in decimal, so `r008` is `r8`. Assisted-by: ClaudeCode:claude-opus-5-5 * perf(zarr-metadata): resolve judges the envelope without walking the JSON it refined `resolve` refines the whole field before reading it, yet its envelope check walked each configuration member as JSON again, and each nested field's configuration once more. `envelope_problems` judges the envelope alone, and `resolve` asks it of the field and of each nested field; `validate_metadata_field_v3`, for a value nothing has refined, is it plus the JSON check of the configuration's members. For a shard holding four codecs, a resolve makes 766 Python calls instead of 1,070. Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent c383b22 commit 8dc50b8

53 files changed

Lines changed: 3472 additions & 123 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
**Breaking:** `must_understand: false` is refused at every extension point
2+
of an array document, codecs and storage transformers included, not only
3+
at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec
4+
gives wrong bytes as surely as ignoring a data type gives wrong values,
5+
and the spec naming only three points is read as an oversight. A document
6+
that declares a codec or a storage transformer ignorable now has a problem
7+
at `codecs.N.must_understand` or `storage_transformers.N.must_understand`,
8+
so the readers refuse it, and `to_key_value` refuses to write a model that
9+
holds one. The JSON schema `zarr_metadata.pydantic` generates for an array
10+
document says the same. A metadata field read on its own still takes
11+
`must_understand: false`.
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
Each extension's metadata is a definition: a value holding the name the
2+
metadata carries, the TypedDict its configuration is -- the one
3+
declaration of that JSON -- and plain functions over that TypedDict for
4+
the rules. `zarr_metadata.v3.definition` is the door, and its module
5+
docstring the guide. Every extension the package models is one: the
6+
codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`,
7+
`sharding_indexed`, `transpose` and `zstd`; every data type, raw bits
8+
included (`r16` reads as `r*` with `{"bits": 16}`, the size its name
9+
carries); the `regular` and `rectilinear` chunk grids; and the `default`
10+
and `v2` chunk key encodings. Each is a constant in its own module --
11+
`GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the
12+
specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions`
13+
registers besides. A third party writes a TypedDict, a rule function and
14+
a definition, and extends a scope with it; a definition checks itself
15+
when built, and nothing happens at class creation.
16+
17+
Reading a field is three steps, each usable on its own by a caller
18+
holding nothing but JSON. `check(value, SomeTypedDict)`, from
19+
`zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs
20+
nothing else; `definition.judge(configuration)` is the check, each nested
21+
field's envelope judged, and then the rules; `resolve(field,
22+
CodecDefinition, scope)` reads a whole field in a scope -- its envelope
23+
judged, its name related to a definition, its configuration judged, each
24+
nested field read the same way -- and returns what the scope made of it,
25+
`Resolved`, with every problem located. A name nothing in scope claims is
26+
`out_of_scope` and left unjudged. A shard's pipelines, a cast's target
27+
type and a struct's field types are nested fields, annotated `CodecField`
28+
and `DataTypeField`, and read in the scope their field is read in.
29+
`configuration_of(resolved, GZIP_CODEC)` is a read configuration typed as
30+
its definition's TypedDict. A codec definition also says whether the size
31+
of what it gives out is `static` or `dynamic`, and a member annotated
32+
`StaticCodecField` takes codecs of static size only: a shard's
33+
`index_codecs`, where the spec forbids a compressor, so a `gzip` there is a
34+
problem at its place.
35+
36+
The TypedDict reads as the typing spec defines it -- `total`, `Required`,
37+
`NotRequired`, `closed` and `extra_items` mean what they mean to a type
38+
checker, whether or not its module postpones annotations. A key a closed
39+
TypedDict does not declare is reported, as `unknown_key`, and left out of
40+
what the rules are handed. A definition refuses a TypedDict that says
41+
nothing of the keys it does not declare, and a member typed
42+
`ZarrV3MetadataFieldJSON`, which would check as plain JSON.
43+
44+
`canonicalize(field, kind, scope)` gives a field without problems in its
45+
simplest equivalent spelling: each nested field in its own simplest
46+
spelling, then its definition's own `canonical`, and the envelope in the
47+
fewest words -- the bare name when nothing is configured, and no
48+
`must_understand`. A field with any problem, an unknown key included, has
49+
none, since a simpler spelling would erase what its author wrote. What
50+
`canonical` gives is judged again, and one that does not hold is a
51+
`ValueError`, a fault in the definition.

‎packages/zarr-metadata/docs/api/index.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ The package is organized to mirror the structure of the Zarr specifications:
1919
documents, with subpackages for [chunk grids](v3/chunk_grid.md),
2020
[chunk key encodings](v3/chunk_key_encoding.md), [codecs](v3/codec.md),
2121
and [data types](v3/data_type.md)
22+
- [`zarr_metadata.v3.definition`](v3/definition.md) — each extension's
23+
metadata as a definition: the TypedDict its configuration is, and the
24+
rules on it; check JSON against a TypedDict, judge a configuration, or
25+
read a whole field in a scope. Its module docstring is the guide
2226

2327
The document types, models, and spec vocabulary — including the store keys —
2428
are re-exported at the top level, so
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: definition
3+
---
4+
5+
::: zarr_metadata.v3.definition
6+
options:
7+
inherited_members: false
8+
show_if_no_docstring: false

‎packages/zarr-metadata/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ nav:
2424
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3.chunk_key_encoding</code>': api/v3/chunk_key_encoding.md
2525
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3.codec</code>': api/v3/codec.md
2626
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3.data_type</code>': api/v3/data_type.md
27+
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3.definition</code>': api/v3/definition.md
2728
- Release notes: release-notes.md
2829
# This site is a Read the Docs subproject of zarr-python; give readers a way
2930
# back to the parent docs, which list every companion package.

‎packages/zarr-metadata/src/zarr_metadata/_pydantic_schema.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,15 @@
2020

2121

2222
class ZarrV3NamedConfigJSON(TypedDict, closed=True):
23-
"""Closed v3 named configuration accepted at optional extension points."""
23+
"""Closed v3 named configuration read on its own, outside a document, where `must_understand` may be `false`."""
2424

2525
name: str
2626
configuration: NotRequired[Mapping[str, JSONValue]]
2727
must_understand: NotRequired[bool]
2828

2929

3030
class ZarrV3MandatoryNamedConfigJSON(TypedDict, closed=True):
31-
"""Closed named configuration accepted where understanding is mandatory."""
31+
"""Closed named configuration at an extension point of a document, where understanding is mandatory."""
3232

3333
name: str
3434
configuration: NotRequired[Mapping[str, JSONValue]]
@@ -37,7 +37,9 @@ class ZarrV3MandatoryNamedConfigJSON(TypedDict, closed=True):
3737

3838
ZarrV3MetadataFieldJSON = str | ZarrV3NamedConfigJSON
3939
ZarrV3MandatoryMetadataFieldJSON = str | ZarrV3MandatoryNamedConfigJSON
40-
ZarrV3CodecPipelineJSON = Annotated[tuple[ZarrV3MetadataFieldJSON, ...], Field(min_length=1)]
40+
ZarrV3CodecPipelineJSON = Annotated[
41+
tuple[ZarrV3MandatoryMetadataFieldJSON, ...], Field(min_length=1)
42+
]
4143
ZarrV2FilterPipelineJSON = tuple[ZarrV2CodecMetadata, ...]
4244

4345

@@ -53,7 +55,7 @@ class ZarrV3ArrayMetadataJSON(TypedDict, extra_items=JSONValue):
5355
fill_value: JSONValue
5456
codecs: ZarrV3CodecPipelineJSON
5557
attributes: NotRequired[Mapping[str, JSONValue]]
56-
storage_transformers: NotRequired[tuple[ZarrV3MetadataFieldJSON, ...]]
58+
storage_transformers: NotRequired[tuple[ZarrV3MandatoryMetadataFieldJSON, ...]]
5759
dimension_names: NotRequired[tuple[str | None, ...]]
5860

5961

‎packages/zarr-metadata/src/zarr_metadata/model/__init__.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,17 +49,14 @@
4949
is_array_metadata_v3,
5050
is_group_metadata_v2,
5151
is_group_metadata_v3,
52-
is_metadata_field_v3,
5352
parse_array_metadata_v2,
5453
parse_array_metadata_v3,
5554
parse_group_metadata_v2,
5655
parse_group_metadata_v3,
57-
parse_metadata_field_v3,
5856
validate_array_metadata_v2,
5957
validate_array_metadata_v3,
6058
validate_group_metadata_v2,
6159
validate_group_metadata_v3,
62-
validate_metadata_field_v3,
6360
)
6461

6562
# Store keys are facts about the on-disk specs, so they are defined in the
@@ -81,6 +78,11 @@
8178
ZARR_V2_GROUP_METADATA_STORE_KEY,
8279
ZarrV2GroupMetadataStoreKey,
8380
)
81+
from zarr_metadata.v3._common import (
82+
is_metadata_field_v3,
83+
parse_metadata_field_v3,
84+
validate_metadata_field_v3,
85+
)
8486
from zarr_metadata.v3.array import (
8587
ZARR_V3_ARRAY_METADATA_STORE_KEY,
8688
ZarrV3ArrayMetadataStoreKey,

‎packages/zarr-metadata/src/zarr_metadata/model/_array.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@
2222
load_store_json,
2323
parse_array_metadata_v2,
2424
parse_array_metadata_v3,
25-
parse_metadata_field_v3,
2625
)
2726
from zarr_metadata.v2.array import ZARR_V2_ARRAY_METADATA_STORE_KEY
2827
from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY
28+
from zarr_metadata.v3._common import parse_metadata_field_v3
2929
from zarr_metadata.v3.array import ZARR_V3_ARRAY_METADATA_STORE_KEY
3030

3131
if TYPE_CHECKING:

‎packages/zarr-metadata/src/zarr_metadata/model/_validation.py‎

Lines changed: 21 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,9 @@
3131
from zarr_metadata._json import prefixed as _prefix
3232
from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON
3333
from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON
34-
from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON
34+
from zarr_metadata.v3._common import (
35+
validate_metadata_field_v3,
36+
)
3537
from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON
3638
from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON
3739

@@ -144,87 +146,6 @@ def _validate_other_members(
144146
return tuple(problems)
145147

146148

147-
def validate_metadata_field_v3(
148-
value: object, *, allow_must_understand_false: bool = True
149-
) -> tuple[ValidationProblem, ...]:
150-
"""Return every reason `value` is not a v3 metadata field.
151-
152-
A metadata field is a bare name string or a mapping containing `name` and
153-
optional `configuration` and `must_understand` members.
154-
"""
155-
if isinstance(value, str):
156-
return ()
157-
if not isinstance(value, Mapping):
158-
return (
159-
ValidationProblem(
160-
(),
161-
"expected a metadata field (string or extension object)",
162-
"invalid_type",
163-
),
164-
)
165-
field = cast("Mapping[object, object]", value)
166-
problems: list[ValidationProblem] = []
167-
allowed_keys = frozenset({"name", "configuration", "must_understand"})
168-
for key in field:
169-
if not isinstance(key, str):
170-
problems.append(
171-
ValidationProblem((), f"non-string metadata field key {key!r}", "invalid_type")
172-
)
173-
elif key not in allowed_keys:
174-
problems.append(
175-
ValidationProblem((key,), "unexpected metadata field member", "invalid_value")
176-
)
177-
if not isinstance(field.get("name"), str):
178-
problems.append(ValidationProblem(("name",), "expected a string name", "invalid_type"))
179-
if "configuration" in field:
180-
configuration = field["configuration"]
181-
if not isinstance(configuration, Mapping):
182-
problems.append(
183-
ValidationProblem(("configuration",), "expected a mapping", "invalid_type")
184-
)
185-
elif not all(isinstance(k, str) for k in cast("Mapping[object, object]", configuration)):
186-
problems.append(
187-
ValidationProblem(("configuration",), "expected string keys", "invalid_type")
188-
)
189-
else:
190-
for key, item in cast("Mapping[str, object]", configuration).items():
191-
problems.extend(_prefix("configuration", _prefix(key, validate_json(item))))
192-
if "must_understand" in field:
193-
must_understand = field["must_understand"]
194-
if not isinstance(must_understand, bool):
195-
problems.append(
196-
ValidationProblem(("must_understand",), "expected a boolean", "invalid_type")
197-
)
198-
elif not allow_must_understand_false and not must_understand:
199-
problems.append(
200-
ValidationProblem(
201-
("must_understand",),
202-
"false is not supported at this extension point",
203-
"invalid_value",
204-
)
205-
)
206-
return tuple(problems)
207-
208-
209-
def is_metadata_field_v3(value: object) -> TypeGuard[ZarrV3MetadataFieldJSON]:
210-
"""Whether `value` is a v3 metadata field: a bare name or a named config."""
211-
if isinstance(value, str):
212-
return True
213-
if not isinstance(value, dict):
214-
return False
215-
field = cast("dict[object, object]", value)
216-
return _is_canonical_json(field) and not validate_metadata_field_v3(field)
217-
218-
219-
def parse_metadata_field_v3(value: object) -> ZarrV3MetadataFieldJSON:
220-
"""Return `value` narrowed to `ZarrV3MetadataFieldJSON`, or raise `MetadataValidationError`."""
221-
normalized = arrays_to_tuples(value)
222-
problems = validate_metadata_field_v3(normalized)
223-
if len(problems) != 0:
224-
raise MetadataValidationError(problems)
225-
return cast(ZarrV3MetadataFieldJSON, normalized)
226-
227-
228149
def _is_array(value: object) -> TypeGuard[Sequence[object]]:
229150
"""Whether `value` reads as a JSON array: a sequence that is not a string or bytes.
230151
@@ -410,6 +331,13 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]:
410331
problems.extend(_validate_dim_sequence(doc, "shape"))
411332
if "fill_value" in doc:
412333
problems.extend(_prefix("fill_value", validate_json(doc["fill_value"])))
334+
# Every extension *point* must be understood: ignoring a codec gives
335+
# wrong bytes just as surely as ignoring a data type gives wrong
336+
# values. The spec names only the first three
337+
# (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1580-L1581),
338+
# which this package reads as an oversight rather than a licence.
339+
# `must_understand: false` keeps its meaning where it has one: an
340+
# unknown top-level extension *field*, which a reader really can skip.
413341
for key in ("data_type", "chunk_grid", "chunk_key_encoding"):
414342
if key in doc:
415343
problems.extend(
@@ -431,7 +359,17 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]:
431359
)
432360
)
433361
for index, entry in enumerate(entries):
434-
problems.extend(_prefix(key, _prefix(index, validate_metadata_field_v3(entry))))
362+
problems.extend(
363+
_prefix(
364+
key,
365+
_prefix(
366+
index,
367+
validate_metadata_field_v3(
368+
entry, allow_must_understand_false=False
369+
),
370+
),
371+
)
372+
)
435373
if "attributes" in doc:
436374
problems.extend(_validate_attributes(doc["attributes"]))
437375
if "dimension_names" in doc:

0 commit comments

Comments
 (0)