Skip to content

Commit 9c844e2

Browse files
d-v-bclaude
andcommitted
feat(zarr-metadata): three layers of reading, and the third produces the resolved pipeline
Reading a v3 array document is now three explicit layers, ordered by what each needs and each handing the next a typed value with its problems. `well_formed_array_v3(value)` needs only the value: `refine_json` turns it into JSON with arrays as tuples, string keys and finite floats in one walk, or says at which leaves it is not, and the model layer judges the document's shape. Nothing downstream normalizes or checks JSON-ness again: the `envelope_judged` flag and the second normalization inside `coerce` are gone, `Opaque.json` is `JSONValue` and checked, and the class routine takes refined JSON. `read_array_v3(document, context)` needs a scope: each extension point is handed to `read_field`, which is what `resolve` does once a field's envelope is judged. `resolve(data, kind, context)` is the first two layers for a field on its own. `refine_array_v3(array)` needs the array, and returns `RefinedArrayV3`: the `ArrayParts` the pipeline is handed and a `Pipeline` of `PipelineStage`s, each a codec and the array that reaches it, a shard's `codecs` and `index_codecs` refined inside its stage. This is what a codec pipeline is built from; validating the composition is what the walk finds on the way. A codec that holds pipelines declares them through `inner_pipelines(incoming)` and judges nothing inside them itself, so the sharding codec's own walk is gone with `chain_problems`. Unchanged to the 40k-document corpus: 0 verdicts, 0 problems, 0 crashes. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent 6deae48 commit 9c844e2

14 files changed

Lines changed: 667 additions & 331 deletions

File tree

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Reading a v3 array document is three layers, ordered by what each needs,
2+
each handing the next a typed value and its problems. `well_formed_array_v3(value)`
3+
needs only the value: the JSON is refined -- arrays as tuples, string
4+
keys, finite floats -- and the document's shape judged by the model layer;
5+
a value that is not JSON gets that verdict and nothing else is judged.
6+
`read_array_v3(document, context)` needs a scope: each extension point's
7+
name is related to a class through the context and the class is handed
8+
the field. `refine_array_v3(array)` needs the array: the fill value
9+
against its type, the grid against the shape, and the codec pipeline
10+
walked with what reaches each codec.
11+
12+
The third layer's value is the resolved pipeline. `RefinedArrayV3`
13+
carries the `ArrayParts` the pipeline is handed and a `Pipeline` of
14+
`PipelineStage`s, each a codec and the array that reaches it -- None past
15+
the array->bytes boundary or after a codec that could not say what it
16+
does -- with a shard's `codecs` and `index_codecs` refined inside its
17+
stage. This is what a codec pipeline is built from, and what zarr-python
18+
computes separately today when it evolves an array spec codec by codec;
19+
validating the composition is what the walk finds on the way. A codec
20+
that holds pipelines declares them through `inner_pipelines(incoming)`,
21+
each by the member that holds it with the parts it is handed, and
22+
judges nothing inside them itself.
23+
24+
`resolve(data, kind, context)` is the first two layers for a field on
25+
its own. Nothing downstream of the first layer normalizes or checks
26+
JSON-ness again, so the reader no longer carries an `envelope_judged`
27+
flag or re-normalizes arrays inside `coerce`; `chain_problems` is gone
28+
from the door, replaced by the refinement that produces the pipeline.

‎packages/zarr-metadata/src/zarr_metadata/model/_validation.py‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,54 @@ def validate_json(value: object) -> tuple[ValidationProblem, ...]:
134134
return (ValidationProblem((), f"not a JSON-serializable value: {value!r}", "invalid_type"),)
135135

136136

137+
def refine_json(
138+
value: object, loc: tuple[str | int, ...] = ()
139+
) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]:
140+
"""`value` as JSON with arrays as tuples, or None with every reason it is not JSON.
141+
142+
The first layer of reading, which needs nothing but the value. One
143+
walk normalizes and judges: a mapping becomes a `dict` with string
144+
keys, a sequence a tuple, a float must be finite. Everything after
145+
it takes `JSONValue` and normalizes nothing. A value that is not
146+
JSON is None, with the problems located at the leaves that are not:
147+
not JSON is the first verdict, and there is nothing to read.
148+
"""
149+
if isinstance(value, float):
150+
if math.isfinite(value):
151+
return value, ()
152+
return None, (
153+
ValidationProblem(loc, f"non-finite float {value!r} is not JSON", "invalid_value"),
154+
)
155+
if isinstance(value, (str, int, bool)) or value is None:
156+
return value, ()
157+
if isinstance(value, Mapping):
158+
members: dict[str, JSONValue] = {}
159+
problems: list[ValidationProblem] = []
160+
for key, item in cast("Mapping[object, object]", value).items():
161+
if not isinstance(key, str):
162+
problems.append(
163+
ValidationProblem(loc, f"non-string key {key!r} in JSON object", "invalid_type")
164+
)
165+
continue
166+
member, found = refine_json(item, (*loc, key))
167+
problems.extend(found)
168+
if len(found) == 0:
169+
members[key] = member
170+
return (members if len(problems) == 0 else None), tuple(problems)
171+
if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray)):
172+
entries: list[JSONValue] = []
173+
found_in_entries: list[ValidationProblem] = []
174+
for index, item in enumerate(cast("Sequence[object]", value)):
175+
entry, found = refine_json(item, (*loc, index))
176+
found_in_entries.extend(found)
177+
if len(found) == 0:
178+
entries.append(entry)
179+
return (tuple(entries) if len(found_in_entries) == 0 else None), tuple(found_in_entries)
180+
return None, (
181+
ValidationProblem(loc, f"not a JSON-serializable value: {value!r}", "invalid_type"),
182+
)
183+
184+
137185
def _is_canonical_json(value: object) -> TypeIs[JSONValue]:
138186
"""Whether `value` already uses the concrete containers in `JSONValue`."""
139187
if isinstance(value, float):

‎packages/zarr-metadata/src/zarr_metadata/rules/_documents.py‎

Lines changed: 58 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -22,35 +22,38 @@
2222
from zarr_metadata.model._array import ZarrV3ArrayMetadata
2323
from zarr_metadata.model._validation import (
2424
MetadataValidationError,
25-
arrays_to_tuples,
25+
refine_json,
2626
)
2727
from zarr_metadata.model._validation import (
2828
validate_array_metadata_v2 as _validate_structure_v2,
2929
)
30-
from zarr_metadata.model._validation import (
31-
validate_array_metadata_v3 as _validate_structure_v3,
32-
)
3330
from zarr_metadata.model._validation import (
3431
validate_group_metadata_v2 as _validate_group_structure_v2,
3532
)
3633
from zarr_metadata.model._validation import (
3734
validate_group_metadata_v3 as _validate_group_structure_v3,
3835
)
3936
from zarr_metadata.v2._document import array_problems_v2
40-
from zarr_metadata.v3._document import array_problems_v3, group_problems_v3, read_array_v3
37+
from zarr_metadata.v3._document import (
38+
group_problems_v3,
39+
read_array_v3,
40+
refine_array_v3,
41+
well_formed_array_v3,
42+
)
4143
from zarr_metadata.v3._registry import CORE_AND_EXTENSIONS, Context
4244

4345
if TYPE_CHECKING:
4446
from collections.abc import Callable
4547

48+
from zarr_metadata._common import JSONValue
4649
from zarr_metadata.model._validation import ValidationProblem
4750
from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON
4851
from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON
4952
from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON
5053
from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON
5154

5255
_StructuralValidator = Callable[[object], tuple[ValidationProblem, ...]]
53-
_SemanticValidator = Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]]
56+
_SemanticValidator = Callable[[Mapping[str, JSONValue]], tuple[ValidationProblem, ...]]
5457

5558
DocumentT = TypeVar("DocumentT")
5659

@@ -76,50 +79,40 @@ def __post_init__(self) -> None:
7679
raise ValueError(msg)
7780

7881

79-
def _no_semantics(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]:
82+
def _no_semantics(document: Mapping[str, JSONValue]) -> tuple[ValidationProblem, ...]:
8083
"""v2 group documents carry no cross-field constraints."""
8184
return ()
8285

8386

84-
def _array_semantics_v3(context: Context) -> _SemanticValidator:
85-
"""The v3 array semantics, asked in `context`.
86-
87-
The work is `zarr_metadata.v3` asking each entity about itself and
88-
about the parts of the document it meets; what this layer decides is
89-
which entities are in scope while it asks.
90-
"""
91-
92-
def judge(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]:
93-
return array_problems_v3(document, context)
94-
95-
return judge
96-
97-
9887
def _group_semantics_v3(context: Context) -> _SemanticValidator:
9988
"""The v3 group semantics, asked in `context`.
10089
10190
A group's only semantic content is its inline consolidated children,
10291
and those are array and group documents judged in the same scope.
10392
"""
10493

105-
def judge(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]:
94+
def judge(document: Mapping[str, JSONValue]) -> tuple[ValidationProblem, ...]:
10695
return group_problems_v3(document, context)
10796

10897
return judge
10998

11099

111100
def _judged(
112-
normalized: object, structure: _StructuralValidator, semantics: _SemanticValidator
113-
) -> tuple[ValidationProblem, ...]:
114-
"""Structural and semantic problems in an already-normalized document.
101+
value: object, structure: _StructuralValidator, semantics: _SemanticValidator
102+
) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]:
103+
"""`value` refined to JSON, and its structural and semantic problems.
115104
116-
Takes the normalized value rather than the caller's input so that
117-
`validate_*` and `parse_*` each walk the document once.
105+
The layers for a document without a codec pipeline: JSON first,
106+
then the shape, then whatever the semantics need of an object. A
107+
value that is not JSON is None with only that verdict.
118108
"""
119-
problems = structure(normalized)
120-
if isinstance(normalized, Mapping):
121-
problems = problems + semantics(cast("Mapping[str, object]", normalized))
122-
return tuple(problems)
109+
refined, problems = refine_json(value)
110+
if refined is None:
111+
return None, problems
112+
problems = structure(refined)
113+
if isinstance(refined, Mapping):
114+
problems = (*problems, *semantics(cast("Mapping[str, JSONValue]", refined)))
115+
return refined, tuple(problems)
123116

124117

125118
def validate_array_metadata_v3(
@@ -134,13 +127,20 @@ def validate_array_metadata_v3(
134127
that could not be built -- so a document with two defects in one
135128
configuration may need a second pass. The verdict is never affected.
136129
137-
Structural problems (from the model layer) and semantic problems
138-
(from the entities themselves) are reported together. JSON arrays are
139-
normalized to tuples before judgment, so list-spelled documents
140-
(e.g. fresh `json.loads` output) are judged at the canonical data
141-
level rather than rejected for their spelling.
130+
The three layers of reading, in order, each handing the next what it
131+
needs: the value refined to JSON and judged for shape, the extension
132+
points read in `context`, the whole refined against the array. Their
133+
problems are reported together; a value that is not JSON gets only
134+
that verdict. List-spelled documents (fresh `json.loads` output) are
135+
judged at the canonical data level rather than rejected for their
136+
spelling.
142137
"""
143-
return _judged(arrays_to_tuples(value), _validate_structure_v3, _array_semantics_v3(context))
138+
document, problems = well_formed_array_v3(value)
139+
if document is None:
140+
return problems
141+
array, found = read_array_v3(document, context)
142+
_, composed = refine_array_v3(array)
143+
return (*problems, *found, *composed)
144144

145145

146146
def parse_array_metadata_v3(
@@ -152,11 +152,14 @@ def parse_array_metadata_v3(
152152
`MetadataValidationError` carrying every structural and composition
153153
problem found.
154154
"""
155-
normalized = arrays_to_tuples(value)
156-
problems = _judged(normalized, _validate_structure_v3, _array_semantics_v3(context))
157-
if len(problems) != 0:
155+
document, problems = well_formed_array_v3(value)
156+
if document is not None:
157+
array, found = read_array_v3(document, context)
158+
_, composed = refine_array_v3(array)
159+
problems = (*problems, *found, *composed)
160+
if document is None or len(problems) != 0:
158161
raise MetadataValidationError(problems)
159-
return cast("ZarrV3ArrayMetadataJSON", normalized)
162+
return cast("ZarrV3ArrayMetadataJSON", document)
160163

161164

162165
def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]:
@@ -165,7 +168,7 @@ def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]:
165168
JSON arrays are normalized to tuples before judgment, as in
166169
`validate_array_metadata_v3`.
167170
"""
168-
return _judged(arrays_to_tuples(value), _validate_structure_v2, array_problems_v2)
171+
return _judged(value, _validate_structure_v2, array_problems_v2)[1]
169172

170173

171174
def parse_array_metadata_v2(value: object) -> ZarrV2ArrayMetadataJSON:
@@ -175,11 +178,10 @@ def parse_array_metadata_v2(value: object) -> ZarrV2ArrayMetadataJSON:
175178
`MetadataValidationError` carrying every structural and composition
176179
problem found.
177180
"""
178-
normalized = arrays_to_tuples(value)
179-
problems = _judged(normalized, _validate_structure_v2, array_problems_v2)
181+
refined, problems = _judged(value, _validate_structure_v2, array_problems_v2)
180182
if len(problems) != 0:
181183
raise MetadataValidationError(problems)
182-
return cast("ZarrV2ArrayMetadataJSON", normalized)
184+
return cast("ZarrV2ArrayMetadataJSON", refined)
183185

184186

185187
def validate_group_metadata_v3(
@@ -191,20 +193,17 @@ def validate_group_metadata_v3(
191193
consolidated child document invalid under its own rules is reported
192194
here, at its path.
193195
"""
194-
return _judged(
195-
arrays_to_tuples(value), _validate_group_structure_v3, _group_semantics_v3(context)
196-
)
196+
return _judged(value, _validate_group_structure_v3, _group_semantics_v3(context))[1]
197197

198198

199199
def parse_group_metadata_v3(
200200
value: object, *, context: Context = CORE_AND_EXTENSIONS
201201
) -> ZarrV3GroupMetadataJSON:
202202
"""Return `value` as a valid `ZarrV3GroupMetadataJSON`, or raise."""
203-
normalized = arrays_to_tuples(value)
204-
problems = _judged(normalized, _validate_group_structure_v3, _group_semantics_v3(context))
203+
refined, problems = _judged(value, _validate_group_structure_v3, _group_semantics_v3(context))
205204
if len(problems) != 0:
206205
raise MetadataValidationError(problems)
207-
return cast("ZarrV3GroupMetadataJSON", normalized)
206+
return cast("ZarrV3GroupMetadataJSON", refined)
208207

209208

210209
def validate_group_metadata_v2(value: object) -> tuple[ValidationProblem, ...]:
@@ -213,16 +212,15 @@ def validate_group_metadata_v2(value: object) -> tuple[ValidationProblem, ...]:
213212
v2 group documents carry no composition constraints today, so this is
214213
the structural judgment, offered here for a uniform read-side API.
215214
"""
216-
return _judged(arrays_to_tuples(value), _validate_group_structure_v2, _no_semantics)
215+
return _judged(value, _validate_group_structure_v2, _no_semantics)[1]
217216

218217

219218
def parse_group_metadata_v2(value: object) -> ZarrV2GroupMetadataJSON:
220219
"""Return `value` as a valid `ZarrV2GroupMetadataJSON`, or raise."""
221-
normalized = arrays_to_tuples(value)
222-
problems = _judged(normalized, _validate_group_structure_v2, _no_semantics)
220+
refined, problems = _judged(value, _validate_group_structure_v2, _no_semantics)
223221
if len(problems) != 0:
224222
raise MetadataValidationError(problems)
225-
return cast("ZarrV2GroupMetadataJSON", normalized)
223+
return cast("ZarrV2GroupMetadataJSON", refined)
226224

227225

228226
def canonicalize_array_metadata_v3(
@@ -252,12 +250,12 @@ def canonicalize_array_metadata_v3(
252250
is a bug in the entity, not a verdict on the document, which was
253251
valid.
254252
"""
255-
normalized = arrays_to_tuples(document)
256-
problems = _validate_structure_v3(normalized)
257-
if not isinstance(normalized, Mapping):
253+
refined, problems = well_formed_array_v3(document)
254+
if refined is None:
258255
return Invalid(problems)
259-
array, found = read_array_v3(cast("Mapping[str, object]", normalized), context)
260-
problems = (*problems, *found, *array.problems())
256+
array, found = read_array_v3(refined, context)
257+
_, composed = refine_array_v3(array)
258+
problems = (*problems, *found, *composed)
261259
if len(problems) != 0:
262260
return Invalid(problems)
263261
return Canonical(ZarrV3ArrayMetadata.from_json(array.canonical().to_json()).to_json())

0 commit comments

Comments
 (0)