Skip to content

Commit c383b22

Browse files
d-v-bclaude
andauthored
feat(zarr-metadata): check, JSON against a TypedDict as the typing spec reads it (#4432)
* feat(zarr-metadata): the checker: parsers and writers compiled from annotations The second of the stack, and a module on its own: `v3/_typed_json` compiles, once per annotation, a parser for each shape JSON takes -- `int`, `float`, `bool`, `str`, `JSONValue`, a `Literal`, `tuple[T, ...]` and `tuple[T1, T2]`, a union, a TypedDict, a record dataclass, `Mapping[str, V]`, a `NewType`, and `| UNSET` for a member a document may leave out -- and the writer that is its inverse. A parser is `(value, loc, state) -> (typed value, problems)`, generic in a state it hands through untouched, so a caller's leaf can read its own shapes at any depth: the entity layer's leaf, in the next PR, reads a field that holds another entity in a scope. The module knows nothing of entities. Read once per class, cached: `field_hints` resolves a dataclass's annotations where each class was defined, under PEP 649 as before it, and skips class variables, which `declared_class_vars` reports so a base can owe one. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * docs(zarr-metadata): the fragment for the checker Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix(zarr-metadata): the checker reads a TypedDict as the typing spec defines it The definition layer makes a configuration's TypedDict the one declaration of its JSON, so the checker has to read a TypedDict the way the typing spec does, and it did not: - Whether a key is required came from `NotRequired` wrappers alone, so `total=False` was ignored, and callers that asked `__required_keys__` instead got a different answer: the runtime works that out before the annotations are evaluated, and a qualifier written as a string, as `from __future__ import annotations` writes every one, is invisible to it. `typeddict_keys` now reads `Required` and `NotRequired` off the evaluated annotations and falls back on the declaring class's `total` for a key with neither. - Every TypedDict was treated as closed. `closed`, `extra_items` and openness are now read as the spec says, inherited from the bases when a class says nothing, which the runtime does not record; bases that disagree are a `TypeError`. - The typed value kept a closed TypedDict's unknown keys, so a value typed as the TypedDict held keys the type says cannot exist. It is now a new dict of the keys the type admits, in the order they came. - `None` is JSON null, a type alias reads as what it stands for, and a TypedDict or alias that holds itself compiles instead of recursing without end. An annotation that does not resolve is a `TypeError` naming its TypedDict at any depth, not a `NameError` from inside. Parsers are pure, `(value, loc) -> (typed value, problems)`. The state argument was for the entity layer's scoped reads; the definition layer finds nested fields in the typed value instead, so the branch of a union that did not match leaves nothing behind. The record dataclasses, the writers, `| UNSET` members, `field_hints` and `declared_class_vars` were the entity layer's too, and have no caller in this stack. `test_typed_json_properties` holds the checker to a reference predicate worked out from a drawn spec -- TypedDicts with bases, per-class `total`, qualifiers in any order, every kind of openness, classes that hold themselves -- built with evaluated and with postponed annotations, over JSON values of capped depth that conform, nearly conform, or are drawn at large. Each of the bugs above, put back, fails it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(zarr-metadata): check, the checker as a public tool The checker was private, so a layer meant to stand alone gave a caller nothing to call until the definition layer. `zarr_metadata.typed_json` is its door: `check(value, SomeTypedDict)` type-checks a JSON value against any TypedDict -- the package's own document and configuration types, or a caller's -- and gives back a value of it, or None, with every problem located; `typeddict_keys` says what a TypedDict declares. The module moves out of `v3`, since nothing in it is version-specific. Every TypedDict the package declares now compiles, and a test says so: the v2 `dtype` alias names itself through a string, which is now resolved in the alias's module. What other runtime checkers do, and what their test suites catch, was surveyed first -- pydantic, typing-inspection, typeguard, cattrs, msgspec, trycast, typedload, beartype, typing_extensions and CPython -- and taken in: - Each annotation is evaluated in the module of the class that wrote it. `get_type_hints` reads what a subclass inherited in the subclass's module, so a base's `x: Foo` read as the subclass's `Foo`, a silent wrong answer, and a string nested in a base's annotation failed to resolve at all (CPython #90531). pydantic and typeguard get the first right; none of them the second. - A union of TypedDicts tagged by a `Literal` key is read by the branch the tag names, so a gzip with a bad level is reported as a gzip; an untagged union is reported by its closest branch; and of clean TypedDict branches, the one declaring the most of the value's keys reads it. - A key both `Required` and `NotRequired`, and runtime keys that are not the annotated keys, are a `TypeError`. - A recursive alias is named in a message rather than expanded. - Tests: every declaration under both `typing.TypedDict` and `typing_extensions.TypedDict`; bases in other modules holding names their subclasses' modules lack or mean otherwise; redeclared and narrowed keys, diamonds, quoted qualifiers, keys that are not names. The property suite now builds each class in a module of its own, mixes postponed and evaluated classes, quotes names by hand, redeclares read-only keys, notes a failing example's source, and asserts it still reaches each of these. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * build(zarr-metadata): hypothesis for the checker's property tests The first tests that import it are the checker's, so it joins the test group here. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): problem() says what a caller does with it Nothing raises `MetadataValidationError` from a rule; a caller that collects problems extends with the tuple `problem()` returns. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(zarr-metadata): check refuses null `check(None, SomeTypedDict)` came back `(None, ())`: no value and no problem. `refine_json` gives JSON's `null` as None with no problem, and `check` read a None refinement as "not JSON, and its problems say why". It now asks the problems whether the value is JSON, so `null` is checked like any other value, and refused as not an object. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(zarr-metadata): JSON and its problems below the model The checker imported `ValidationProblem`, `ProblemKind`, `refine_json` and `is_json` from the model, so the layer that stands alone depended on the one it feeds. They move, with the rest of what reads JSON and reports on it, into `zarr_metadata._json`: the problem types, `MetadataValidationError`, `refine_json` and `refine_user_data`, `validate_json`, `is_json`, `parse_json` and `arrays_to_tuples`. `zarr_metadata.model` exports every name it did; the checker, and the model's own modules, import from where the names now live. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): the checker's fragment takes this PR's number Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent cad2561 commit c383b22

19 files changed

Lines changed: 3279 additions & 258 deletions
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
`zarr_metadata.typed_json.check(value, SomeTypedDict)` type-checks a JSON
2+
value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`,
3+
`GzipCodecConfiguration`, or one of your own -- and gives back a value of
4+
it, or None, with every problem located. A TypedDict reads as the typing
5+
spec defines it, whatever the runtime's attributes say: a key is required
6+
by its own `Required` or `NotRequired`, or else by the `total` of the class
7+
that declared it, read from the evaluated annotations, so a module written
8+
with `from __future__ import annotations` reads like any other; `closed`,
9+
`extra_items` and openness come from the class, or from its bases when it
10+
says neither; and each annotation is evaluated in the module of the class
11+
that wrote it. A key a closed TypedDict does not declare is reported, as
12+
the new problem kind `unknown_key`, and left out of the value, which still
13+
comes back; a union of TypedDicts tagged by a `Literal` key is read by the
14+
branch the tag names. `typeddict_keys` says what a TypedDict declares.

‎packages/zarr-metadata/docs/api/index.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ The package is organized to mirror the structure of the Zarr specifications:
1010
structural validators, loc-aware parsers, and the `UNSET` sentinel
1111
- [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types
1212
over the models
13+
- [`zarr_metadata.typed_json`](typed_json.md) — `check`, which type-checks
14+
a JSON value against any of the package's `TypedDict`s, read as the
15+
typing spec defines them, with every problem located
1316
- [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents
1417
(`.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`)
1518
- [`zarr_metadata.v3`](v3/index.md) — `TypedDict` shapes for Zarr v3
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
title: typed_json
3+
---
4+
5+
::: zarr_metadata.typed_json

‎packages/zarr-metadata/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ nav:
1616
- api/index.md
1717
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.model</code>': api/model.md
1818
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.pydantic</code>': api/pydantic.md
19+
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.typed_json</code>': api/typed_json.md
1920
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v2</code>': api/v2.md
2021
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3</code>':
2122
- api/v3/index.md

‎packages/zarr-metadata/pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/z
4747
Documentation = "https://zarr-metadata.readthedocs.io/"
4848

4949
[dependency-groups]
50-
test = ["pytest", "pydantic>=2.13", "jsonschema"]
50+
test = ["pytest", "pydantic>=2.13", "jsonschema", "hypothesis"]
5151
docs = [
5252
# Pins match the zarr-python docs environment in the repo-root
5353
# pyproject.toml so every site renders with the same toolchain, and the
Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
"""JSON values, and the problems found with them: where reading starts, below every layer.
2+
3+
`refine_json` is the one walk that normalizes and judges a value, and
4+
`refine_user_data` the same walk over user data, where a non-finite number
5+
is the float it is.
6+
`ValidationProblem` and `MetadataValidationError` are what every
7+
layer reports with: the checker, the definitions and the model alike.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
import math
13+
from collections.abc import Mapping, Sequence
14+
from dataclasses import dataclass
15+
from typing import Literal, TypeGuard, cast, get_args
16+
17+
from zarr_metadata._common import JSONValue
18+
19+
ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json", "unknown_key"]
20+
"""Machine-readable classification of a `ValidationProblem`.
21+
22+
- `missing_key`: a required key (document key or store key) is absent.
23+
- `invalid_type`: a value has the wrong structural type (e.g. a string where
24+
a mapping is required, a non-JSON-serializable object).
25+
- `invalid_value`: a value has an acceptable type but an invalid content
26+
(e.g. `zarr_format: 2` in a v3 document, `order: "Q"`).
27+
- `invalid_json`: bytes that do not decode as JSON.
28+
- `unknown_key`: a key an object's type does not declare, where the type
29+
says it is closed, as a closed TypedDict does. Whether a Zarr
30+
configuration is closed is rarely said (zarr-developers/zarr-specs#270
31+
has been open since 2023), and many readers refuse such a key. It gets
32+
a kind of its own so that a caller who tolerates it can tell it from a
33+
wrong value, and so that it never masks the other findings about the
34+
same object.
35+
"""
36+
37+
38+
@dataclass(frozen=True, slots=True)
39+
class ValidationProblem:
40+
"""A single problem found in a value: where it is, what is wrong, and what kind of wrong.
41+
42+
`loc` is the path from the root of what was judged to the offending
43+
value, e.g. `("codecs", 0, "name")` in a document, and an empty `loc`
44+
refers to that root.
45+
`kind` classifies the failure mode for programmatic dispatch; `message`
46+
is the human-readable description.
47+
"""
48+
49+
loc: tuple[str | int, ...]
50+
message: str
51+
kind: ProblemKind
52+
53+
def __post_init__(self) -> None:
54+
# The runtime half of the annotations: a rule written without a type
55+
# checker, as an extension's may be, fails where it builds a problem
56+
# rather than reporting one at a location that is not one.
57+
loc = cast("object", self.loc)
58+
if not isinstance(loc, tuple) or not all(
59+
isinstance(part, str) or (isinstance(part, int) and not isinstance(part, bool))
60+
for part in cast("tuple[object, ...]", loc)
61+
):
62+
msg = f"a ValidationProblem's loc is a tuple of keys and indices, got {loc!r}"
63+
raise TypeError(msg)
64+
message = cast("object", self.message)
65+
if not isinstance(message, str):
66+
msg = f"a ValidationProblem's message is a string, got {message!r}"
67+
raise TypeError(msg)
68+
kind = cast("object", self.kind)
69+
if not isinstance(kind, str) or kind not in get_args(ProblemKind):
70+
msg = f"a ValidationProblem's kind is one of {get_args(ProblemKind)!r}, got {kind!r}"
71+
raise TypeError(msg)
72+
73+
def __str__(self) -> str:
74+
location = ".".join(str(part) for part in self.loc) if self.loc else "<root>"
75+
return f"{location}: {self.message}"
76+
77+
78+
class MetadataValidationError(ValueError):
79+
"""Raised when a value fails validation, by the entry points that raise rather than report.
80+
81+
Carries every problem found (not just the first) in `.problems`, as an
82+
immutable tuple: a raised error is a finished report, and a caller
83+
inspecting it must not be able to edit the record.
84+
"""
85+
86+
problems: tuple[ValidationProblem, ...]
87+
88+
def __init__(self, problems: Sequence[ValidationProblem]) -> None:
89+
self.problems = tuple(problems)
90+
for entry in cast("tuple[object, ...]", self.problems):
91+
# The runtime half of the annotation: a caller that is not
92+
# type-checked, and hands over anything else, fails here
93+
# rather than far away, where a `loc` is read off it.
94+
if not isinstance(entry, ValidationProblem):
95+
msg = (
96+
"MetadataValidationError takes ValidationProblem values, "
97+
f"got {type(entry).__name__}"
98+
)
99+
raise TypeError(msg)
100+
super().__init__("\n".join(str(problem) for problem in self.problems))
101+
102+
def __reduce__(
103+
self,
104+
) -> tuple[
105+
type[MetadataValidationError], tuple[tuple[ValidationProblem, ...]], dict[str, object]
106+
]:
107+
# An exception pickles and copies as its class called with its
108+
# `args`, which here are the message; it is built from its problems,
109+
# and the rest of its state -- its notes among it -- follows.
110+
return (type(self), (self.problems,), self.__dict__)
111+
112+
113+
def prefixed(
114+
loc_head: str | int, problems: Sequence[ValidationProblem]
115+
) -> tuple[ValidationProblem, ...]:
116+
"""Prepend `loc_head` to the `loc` of every problem (for nested validators)."""
117+
return tuple(ValidationProblem((loc_head, *p.loc), p.message, p.kind) for p in problems)
118+
119+
120+
def validate_json(value: object) -> tuple[ValidationProblem, ...]:
121+
"""Return every reason `value` is not JSON-serializable (recursively): `refine_json`'s problems."""
122+
return refine_json(value)[1]
123+
124+
125+
def refine_json(
126+
value: object, loc: tuple[str | int, ...] = ()
127+
) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]:
128+
"""`value` as JSON with arrays as tuples, or None with every reason it is not JSON.
129+
130+
One walk normalizes and judges: a mapping becomes a `dict` with
131+
string keys, a sequence a tuple, and a float must be finite. A value
132+
that is not JSON is None, with the problems located at the leaves
133+
that are not. JSON's `null` is None too, with no problem, so whether
134+
`value` is JSON is whether there are problems.
135+
"""
136+
return _refine(value, loc, finite=True)
137+
138+
139+
def refine_user_data(
140+
value: object, loc: tuple[str | int, ...] = ()
141+
) -> tuple[JSONValue | None, tuple[ValidationProblem, ...]]:
142+
"""User data refined as `refine_json` refines JSON, a non-finite number being the float it is.
143+
144+
A node's attributes are user data. The spec asks only that each be a
145+
JSON value and interprets none, and zarr-python writes them with the
146+
defaults of Python's `json` module, so an attribute can hold `NaN`,
147+
`Infinity` or `-Infinity` -- xarray's `_FillValue`, a CF
148+
`missing_value` -- which RFC 8259 lacks. Wherever the spec interprets
149+
a value it spells those numbers as strings, so a validator walks what
150+
it interprets with `refine_json`, and only user data with this.
151+
"""
152+
return _refine(value, loc, finite=False)
153+
154+
155+
_Refined = tuple[JSONValue | None, tuple[ValidationProblem, ...]]
156+
157+
158+
def _refine(value: object, loc: tuple[str | int, ...], *, finite: bool) -> _Refined:
159+
"""`refine_json`, a non-finite number being JSON unless `finite`."""
160+
if isinstance(value, float):
161+
if not finite or math.isfinite(value):
162+
return value, ()
163+
return None, (
164+
ValidationProblem(loc, f"non-finite float {value!r} is not JSON", "invalid_value"),
165+
)
166+
if isinstance(value, (str, int, bool)) or value is None:
167+
return value, ()
168+
if isinstance(value, Mapping):
169+
# Walked here rather than through `_refine_members`, so that each
170+
# level of nesting costs one frame, as deep as the interpreter goes.
171+
members: dict[str, JSONValue] = {}
172+
found_in_members: list[ValidationProblem] = []
173+
for key, item in cast("Mapping[object, object]", value).items():
174+
if not isinstance(key, str):
175+
found_in_members.append(
176+
ValidationProblem(loc, f"non-string key {key!r} in JSON object", "invalid_type")
177+
)
178+
continue
179+
member, found = _refine(item, (*loc, key), finite=finite)
180+
found_in_members.extend(found)
181+
if len(found) == 0:
182+
members[key] = member
183+
return (members if len(found_in_members) == 0 else None), tuple(found_in_members)
184+
if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray)):
185+
entries: list[JSONValue] = []
186+
found_in_entries: list[ValidationProblem] = []
187+
for index, item in enumerate(cast("Sequence[object]", value)):
188+
entry, found = _refine(item, (*loc, index), finite=finite)
189+
found_in_entries.extend(found)
190+
if len(found) == 0:
191+
entries.append(entry)
192+
return (tuple(entries) if len(found_in_entries) == 0 else None), tuple(found_in_entries)
193+
return None, (
194+
ValidationProblem(loc, f"not a JSON-serializable value: {value!r}", "invalid_type"),
195+
)
196+
197+
198+
def is_canonical_json(value: object, *, finite: bool = True) -> TypeGuard[JSONValue]:
199+
"""Whether `value` already uses the concrete containers in `JSONValue`.
200+
201+
A non-finite number counts only when `finite` is false, as a document's
202+
guard passes it: where one may be is the document's validator's to say.
203+
"""
204+
if isinstance(value, float):
205+
return not finite or math.isfinite(value)
206+
if isinstance(value, (str, int, bool)) or value is None:
207+
return True
208+
if isinstance(value, (list, tuple)):
209+
sequence = cast("list[object] | tuple[object, ...]", value)
210+
return all(is_canonical_json(item, finite=finite) for item in sequence)
211+
if isinstance(value, dict):
212+
mapping = cast("dict[object, object]", value)
213+
return all(
214+
isinstance(key, str) and is_canonical_json(item, finite=finite)
215+
for key, item in mapping.items()
216+
)
217+
return False
218+
219+
220+
def is_json(value: object) -> TypeGuard[JSONValue]:
221+
"""Whether `value` is a canonical JSON structure (recursively)."""
222+
return is_canonical_json(value)
223+
224+
225+
def parse_json(value: object) -> JSONValue:
226+
"""Return a canonical `JSONValue`, or raise `MetadataValidationError`."""
227+
refined, problems = refine_json(value)
228+
if len(problems) != 0:
229+
raise MetadataValidationError(problems)
230+
return refined
231+
232+
233+
def arrays_to_tuples(obj: object) -> object:
234+
"""Recursively materialize mappings and convert array-like values to tuples."""
235+
if isinstance(obj, Sequence) and not isinstance(obj, (str, bytes, bytearray)):
236+
sequence = cast("Sequence[object]", obj)
237+
converted_sequence = tuple(arrays_to_tuples(item) for item in sequence)
238+
if isinstance(obj, tuple) and all(
239+
converted is original
240+
for converted, original in zip(converted_sequence, sequence, strict=True)
241+
):
242+
return sequence
243+
return converted_sequence
244+
if isinstance(obj, Mapping):
245+
mapping = cast("Mapping[object, object]", obj)
246+
converted: dict[object, object] = {
247+
key: arrays_to_tuples(value) for key, value in mapping.items()
248+
}
249+
if isinstance(obj, dict) and all(converted[key] is value for key, value in mapping.items()):
250+
return mapping
251+
return converted
252+
return obj
253+
254+
255+
__all__ = [
256+
"MetadataValidationError",
257+
"ProblemKind",
258+
"ValidationProblem",
259+
"arrays_to_tuples",
260+
"is_canonical_json",
261+
"is_json",
262+
"parse_json",
263+
"prefixed",
264+
"refine_json",
265+
"refine_user_data",
266+
"validate_json",
267+
]

0 commit comments

Comments
 (0)