Skip to content

Commit 0401a7f

Browse files
d-v-bclaude
andauthored
feat(zarr-metadata)!: the model reads each extension point through its definition (#4436)
* feat(zarr-metadata): the model reads each extension point through its definition `validate_array_metadata_v3` judged an array document's structure and left every configuration unread: a gzip `level` of 99, or a key a codec does not declare, passed. It now reads each extension point -- the data type, the chunk grid, the chunk key encoding, each codec and each storage transformer -- with `resolve`, in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: the envelope judged, `must_understand: false` refused at each, and the configuration read against the definition that claims its name. A name nothing in scope claims is left unjudged, which keeps the format open. So do the `is_*` and `parse_*` beside it and the v3 group validators, with each array an inline `consolidated_metadata` holds, and the v3 model classes' `from_json` and `from_key_value` take the same `context`, so a reader that substitutes its own reading of a codec builds the model from a document its scope accepts. The pydantic types read in `CORE_AND_EXTENSIONS`, since a field type holds no scope; the docstrings and the definition guide say so, and say that no rule here reads one field against another. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): the validation boundary says what the validators read The README and the docs index said the model validators do not interpret extension names or configurations. In a v3 document they now read each extension point through its definition in a scope, refuse what the definition refuses and report an undeclared key as `unknown_key`; they leave names nothing claims unjudged, and do not judge fields against each other. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): the fragment for the model reading extension points through definitions Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(zarr-metadata): a v3 model writes in the scope it reads in `to_key_value` validates what it writes with its reader's `parse_*`, and at this layer the reader reads each extension point in a scope. The v3 array and group `to_key_value` take the same `context` as `from_json` and `from_key_value`, so a model read with a reader's own definitions is written with them, and the default scope still refuses to write what it would refuse to read. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): the fragment takes this PR's number and says what breaks Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): the must_understand fragment names the PR that merged it The change merged with #4434, which carried #4433's commits; towncrier links a fragment to the pull request its name gives. Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 8dc50b8 commit 0401a7f

13 files changed

Lines changed: 439 additions & 115 deletions

File tree

‎packages/zarr-metadata/README.md‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Two layers and an optional integration:
1414
specifications, plus types for [`zarr-extensions`](https://github.com/zarr-developers/zarr-extensions/)
1515
and a few widely-used-but-unspecified entities (e.g. consolidated metadata).
1616
- **Document models** (`zarr_metadata.model`): canonical frozen-dataclass
17-
models of whole metadata documents, with structural validators, loc-aware
17+
models of whole metadata documents, with validators, loc-aware
1818
parsers, and store-key (de)serialization. A document produced by `to_json`
1919
shares no mutable state with the model that produced it.
2020
- **Optional Pydantic integration** (`zarr_metadata.pydantic`, requires
@@ -57,10 +57,16 @@ members that the strict model parser rejects.
5757
The model validators enforce the declared document structure and a small set
5858
of context-free consistency rules, including fixed format literals, finite
5959
JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
60-
`dimension_names` entry per array dimension. They do not interpret extension
61-
names or configurations, resolve codec pipelines, or decide whether a data
62-
type, chunk grid, codec, or storage transformer is supported. Those decisions
63-
belong to consumer implementations.
60+
`dimension_names` entry per array dimension. In a v3 document they also read
61+
each extension point -- the data type, chunk grid, chunk key encoding, each
62+
codec and each storage transformer -- through the definition that claims its
63+
name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a
64+
configuration its definition refuses is refused, and a key it does not
65+
declare is reported as `unknown_key`. A name nothing in the scope claims is
66+
left unjudged, and whether to support it is the consumer's decision. The
67+
validators do not judge fields against each other: a fill value against its
68+
data type, a codec against the array it is handed, a chunk grid against the
69+
shape.
6470

6571
The Pydantic integration's generated JSON Schemas express independently
6672
checkable document structure and field constraints, but they are not a
@@ -75,7 +81,7 @@ versus `shape`. Consumers should run the model parser after schema validation.
7581
At minimum, this library supports what Zarr-Python needs: the complete
7682
Zarr v2 and v3 specs, consolidated metadata, and a subset of the metadata
7783
defined in `zarr-extensions`. We are generally open to contributions that
78-
add types, models, or structural validation for Zarr metadata with a
84+
add types, models, or validation for Zarr metadata with a
7985
published spec.
8086

8187
Runtime array behavior is out of scope: nothing here encodes or decodes
File renamed without changes.
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
**Breaking:** the v3 validators read each extension point through its
2+
definition: `validate_array_metadata_v3` judges the data type, the chunk
3+
grid, the chunk key encoding, each codec and each storage transformer
4+
against the definition that claims its name in a scope,
5+
`CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of
6+
99, or a key a codec does not declare, is a problem located in the
7+
configuration, and a document holding one, accepted before, is refused.
8+
So do the `is_*` and `parse_*` beside it and the v3 group validators,
9+
with each array an inline `consolidated_metadata` holds, and the v3
10+
model classes' `from_json`, `from_key_value` and `to_key_value` take the
11+
same `context`, so a model read in a scope is written in it. The
12+
pydantic types read in `CORE_AND_EXTENSIONS`. A name nothing in scope
13+
claims is left unjudged.

‎packages/zarr-metadata/docs/api/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ title: API reference
77
The package is organized to mirror the structure of the Zarr specifications:
88

99
- [`zarr_metadata.model`](model.md) — frozen-dataclass document models,
10-
structural validators, loc-aware parsers, and the `UNSET` sentinel
10+
validators, loc-aware parsers, and the `UNSET` sentinel
1111
- [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types
1212
over the models
1313
- [`zarr_metadata.typed_json`](typed_json.md) — `check`, which type-checks

‎packages/zarr-metadata/docs/index.md‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ closely model the content of the Zarr specifications, such as:
2828
[zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a
2929
few widely-used-but-unspecified entities (e.g. consolidated metadata).
3030
- **Document models** ([`zarr_metadata.model`](api/model.md)): canonical
31-
frozen-dataclass models of whole metadata documents, with structural
31+
frozen-dataclass models of whole metadata documents, with
3232
validators, loc-aware parsers, and store-key (de)serialization. A document
3333
produced by `to_json` shares no mutable state with the model that produced
3434
it.
@@ -72,17 +72,23 @@ members that the strict model parser rejects.
7272
The model validators enforce the declared document structure and a small set
7373
of context-free consistency rules, including fixed format literals, finite
7474
JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
75-
`dimension_names` entry per array dimension. They do not interpret extension
76-
names or configurations, resolve codec pipelines, or decide whether a data
77-
type, chunk grid, codec, or storage transformer is supported. Those decisions
78-
belong to consumer implementations.
75+
`dimension_names` entry per array dimension. In a v3 document they also read
76+
each extension point -- the data type, chunk grid, chunk key encoding, each
77+
codec and each storage transformer -- through the definition that claims its
78+
name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a
79+
configuration its definition refuses is refused, and a key it does not
80+
declare is reported as `unknown_key`. A name nothing in the scope claims is
81+
left unjudged, and whether to support it is the consumer's decision. The
82+
validators do not judge fields against each other: a fill value against its
83+
data type, a codec against the array it is handed, a chunk grid against the
84+
shape.
7985

8086
## Scope
8187

8288
At minimum, this library supports what Zarr-Python needs: the complete
8389
Zarr v2 and v3 specs, consolidated metadata, and a subset of the metadata
8490
defined in `zarr-extensions`. We are generally open to contributions that
85-
add types, models, or structural validation for Zarr metadata with a
91+
add types, models, or validation for Zarr metadata with a
8692
published spec.
8793

8894
Runtime array behavior is out of scope: nothing here encodes or decodes

‎packages/zarr-metadata/src/zarr_metadata/model/__init__.py‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
"""In-memory models for Zarr metadata documents.
22
33
Models are frozen dataclasses that hold a canonical, semantically lossless
4-
representation of the JSON documents; they never interpret extension points
5-
(codecs, chunk grids, data types). Validators check JSON structure, not domain validity.
6-
Each document concept gets a `validate_*` function returning every problem
7-
found (a `list[ValidationProblem]`, each with a machine-readable `kind`), an
8-
`is_*` type guard, and a `parse_*` function that narrows or raises
9-
`MetadataValidationError`. Model `from_json` / `from_key_value` constructors
10-
raise `MetadataValidationError` for every ingestion failure, including
11-
missing store keys and undecodable bytes.
4+
representation of the JSON documents. Validators check a document's JSON
5+
structure and, in a v3 document, read each extension point (codecs, chunk
6+
grids, data types, ...) through the definition that claims its name in a
7+
scope, `CORE_AND_EXTENSIONS` unless a `context` is passed; they do not judge
8+
fields against each other. Each document concept gets a `validate_*`
9+
function returning every problem found (a tuple of `ValidationProblem`, each
10+
with a machine-readable `kind`), an `is_*` type guard, and a `parse_*`
11+
function that narrows or raises `MetadataValidationError`. Model
12+
`from_json` / `from_key_value` constructors raise `MetadataValidationError`
13+
for every ingestion failure, including missing store keys and undecodable
14+
bytes, and the v3 ones take the same `context`.
1215
"""
1316

1417
from zarr_metadata._json import (

‎packages/zarr-metadata/src/zarr_metadata/model/_array.py‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
from zarr_metadata.v2.array import ZARR_V2_ARRAY_METADATA_STORE_KEY
2727
from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY
2828
from zarr_metadata.v3._common import parse_metadata_field_v3
29+
from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS
2930
from zarr_metadata.v3.array import ZARR_V3_ARRAY_METADATA_STORE_KEY
3031

3132
if TYPE_CHECKING:
@@ -40,6 +41,7 @@
4041
from zarr_metadata.v2.attributes import ZarrV2AttributesStoreKey
4142
from zarr_metadata.v2.codec import ZarrV2CodecMetadata
4243
from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON
44+
from zarr_metadata.v3._registry import Context
4345
from zarr_metadata.v3.array import (
4446
ZarrV3ArrayMetadataJSON,
4547
ZarrV3ArrayMetadataStoreKey,
@@ -158,8 +160,11 @@ class ZarrV3ArrayMetadata:
158160
content for an array. Extension points (`data_type`, `chunk_grid`,
159161
`chunk_key_encoding`, `codecs`, `storage_transformers`) are held as
160162
`ZarrV3MetadataField` values (currently `ZarrV3NamedConfig` name,
161-
configuration, and obligation records) and are never interpreted;
162-
`fill_value` is held verbatim in its JSON form. Equivalent extension
163+
configuration, and obligation records). `from_json` and
164+
`from_key_value` read each through the definition that claims its name
165+
in a scope -- `CORE_AND_EXTENSIONS` unless a `context` is passed -- and
166+
the model holds what they read as written; `fill_value` is held
167+
verbatim in its JSON form. Equivalent extension
163168
spellings normalize to shorthand strings when configuration is empty and
164169
understanding is required.
165170
"""
@@ -276,9 +281,11 @@ def to_json(self) -> ZarrV3ArrayMetadataJSON:
276281
return out
277282

278283
@classmethod
279-
def from_json(cls, data: object) -> ZarrV3ArrayMetadata:
284+
def from_json(
285+
cls, data: object, *, context: Context = CORE_AND_EXTENSIONS
286+
) -> ZarrV3ArrayMetadata:
280287
# A read model shares no mutable state with what it read.
281-
parsed = copy.deepcopy(parse_array_metadata_v3(data))
288+
parsed = copy.deepcopy(parse_array_metadata_v3(data, context=context))
282289
extra_fields: dict[str, ZarrV3ExtensionField] = {
283290
k: v for k, v in parsed.items() if k not in ARRAY_METADATA_STANDARD_KEYS_V3
284291
}
@@ -310,15 +317,20 @@ def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]:
310317
return must_understand_subset(self.extra_fields)
311318

312319
@classmethod
313-
def from_key_value(cls, mapping: Mapping[StoreKey, bytes]) -> ZarrV3ArrayMetadata:
314-
return cls.from_json(load_store_json(mapping, ZARR_V3_ARRAY_METADATA_STORE_KEY))
320+
def from_key_value(
321+
cls, mapping: Mapping[StoreKey, bytes], *, context: Context = CORE_AND_EXTENSIONS
322+
) -> ZarrV3ArrayMetadata:
323+
return cls.from_json(
324+
load_store_json(mapping, ZARR_V3_ARRAY_METADATA_STORE_KEY), context=context
325+
)
315326

316327
def to_key_value(
317-
self, *, indent: int | str | None = None
328+
self, *, indent: int | str | None = None, context: Context = CORE_AND_EXTENSIONS
318329
) -> Mapping[ZarrV3ArrayMetadataStoreKey, bytes]:
319330
# A model built by hand is not validated: its document is written only
320-
# if it reads as `from_json` reads one, and every problem is raised.
321-
document = parse_array_metadata_v3(self.to_json())
331+
# if it reads as `from_json` reads one in `context`, and every problem
332+
# is raised.
333+
document = parse_array_metadata_v3(self.to_json(), context=context)
322334
return {ZARR_V3_ARRAY_METADATA_STORE_KEY: dump_store_json(document, indent=indent)}
323335

324336

‎packages/zarr-metadata/src/zarr_metadata/model/_group.py‎

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY
3535
from zarr_metadata.v2.consolidated import ZARR_V2_CONSOLIDATED_METADATA_STORE_KEY
3636
from zarr_metadata.v2.group import ZARR_V2_GROUP_METADATA_STORE_KEY
37+
from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS
3738
from zarr_metadata.v3.consolidated import ZARR_V3_CONSOLIDATED_METADATA_KEY
3839
from zarr_metadata.v3.group import ZARR_V3_GROUP_METADATA_STORE_KEY
3940

@@ -42,6 +43,7 @@
4243
from zarr_metadata.v2.attributes import ZarrV2AttributesStoreKey
4344
from zarr_metadata.v2.consolidated import ZarrV2ConsolidatedMetadataStoreKey
4445
from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON, ZarrV2GroupMetadataStoreKey
46+
from zarr_metadata.v3._registry import Context
4547
from zarr_metadata.v3.array import ZarrV3ExtensionField
4648
from zarr_metadata.v3.consolidated import ZarrV3ConsolidatedMetadataJSON
4749
from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON, ZarrV3GroupMetadataStoreKey
@@ -72,8 +74,9 @@ class ZarrV3GroupMetadata:
7274
7375
A canonical, semantically lossless representation of the `zarr.json`
7476
content for a group. The `consolidated_metadata` reference-implementation
75-
convention is modeled as a typed field holding thin child models; every
76-
other unknown top-level key lands in `extra_fields` verbatim.
77+
convention is modeled as a typed field holding thin child models, each
78+
array read in the scope the group is read in; every other unknown
79+
top-level key lands in `extra_fields` verbatim.
7780
"""
7881

7982
zarr_format: Literal[3] = field(default=3, init=False)
@@ -135,9 +138,11 @@ def to_json(self) -> ZarrV3GroupMetadataJSON:
135138
return out
136139

137140
@classmethod
138-
def from_json(cls, data: object) -> ZarrV3GroupMetadata:
141+
def from_json(
142+
cls, data: object, *, context: Context = CORE_AND_EXTENSIONS
143+
) -> ZarrV3GroupMetadata:
139144
# A read model shares no mutable state with what it read.
140-
parsed = copy.deepcopy(parse_group_metadata_v3(data))
145+
parsed = copy.deepcopy(parse_group_metadata_v3(data, context=context))
141146
consolidated_raw: object = parsed.get(ZARR_V3_CONSOLIDATED_METADATA_KEY, UNSET)
142147
consolidated: ZarrV3ConsolidatedMetadata | UNSET
143148
if consolidated_raw is UNSET or consolidated_raw is None:
@@ -146,7 +151,7 @@ def from_json(cls, data: object) -> ZarrV3GroupMetadata:
146151
# absence and never written back — repaired, not preserved.
147152
consolidated = UNSET
148153
else:
149-
consolidated = ZarrV3ConsolidatedMetadata.from_json(consolidated_raw)
154+
consolidated = ZarrV3ConsolidatedMetadata.from_json(consolidated_raw, context=context)
150155
extra_fields: dict[str, ZarrV3ExtensionField] = {
151156
k: v
152157
for k, v in parsed.items()
@@ -171,15 +176,20 @@ def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]:
171176
return must_understand_subset(self.extra_fields)
172177

173178
@classmethod
174-
def from_key_value(cls, mapping: Mapping[StoreKey, bytes]) -> ZarrV3GroupMetadata:
175-
return cls.from_json(load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY))
179+
def from_key_value(
180+
cls, mapping: Mapping[StoreKey, bytes], *, context: Context = CORE_AND_EXTENSIONS
181+
) -> ZarrV3GroupMetadata:
182+
return cls.from_json(
183+
load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY), context=context
184+
)
176185

177186
def to_key_value(
178-
self, *, indent: int | str | None = None
187+
self, *, indent: int | str | None = None, context: Context = CORE_AND_EXTENSIONS
179188
) -> Mapping[ZarrV3GroupMetadataStoreKey, bytes]:
180189
# A model built by hand is not validated: its document is written only
181-
# if it reads as `from_json` reads one, and every problem is raised.
182-
document = parse_group_metadata_v3(self.to_json())
190+
# if it reads as `from_json` reads one in `context`, and every problem
191+
# is raised.
192+
document = parse_group_metadata_v3(self.to_json(), context=context)
183193
return {ZARR_V3_GROUP_METADATA_STORE_KEY: dump_store_json(document, indent=indent)}
184194

185195

@@ -221,19 +231,21 @@ def to_json(self) -> ZarrV3ConsolidatedMetadataJSON:
221231
}
222232

223233
@classmethod
224-
def from_json(cls, data: object) -> ZarrV3ConsolidatedMetadata:
234+
def from_json(
235+
cls, data: object, *, context: Context = CORE_AND_EXTENSIONS
236+
) -> ZarrV3ConsolidatedMetadata:
225237
normalized = arrays_to_tuples(data)
226-
problems = validate_consolidated_metadata_v3(normalized)
238+
problems = validate_consolidated_metadata_v3(normalized, context=context)
227239
if len(problems) != 0:
228240
raise MetadataValidationError(problems)
229241
env = cast("Mapping[str, object]", normalized)
230242
entries: dict[str, ZarrV3ArrayMetadata | ZarrV3GroupMetadata] = {}
231243
for key, entry in cast("Mapping[str, object]", env["metadata"]).items():
232244
node_type = cast("Mapping[str, object]", entry).get("node_type")
233245
if node_type == "array":
234-
entries[key] = ZarrV3ArrayMetadata.from_json(entry)
246+
entries[key] = ZarrV3ArrayMetadata.from_json(entry, context=context)
235247
else:
236-
entries[key] = ZarrV3GroupMetadata.from_json(entry)
248+
entries[key] = ZarrV3GroupMetadata.from_json(entry, context=context)
237249
return cls(metadata=entries)
238250

239251

0 commit comments

Comments
 (0)