Skip to content

Commit 5c37f97

Browse files
authored
fix(dtype): resolve struct fields with the registry that resolves the struct (#4387)
* feat(dtype): resolve nested data types with the registry that resolves their container A structured data type resolved its field data types with the default data type registry, whatever registry resolved the structured data type itself, so a DataTypeRegistry other than the default could not supply field data types. - DTypeResolver (exported from zarr.dtype) is the signature of DataTypeRegistry.match_json. - HasNestedDTypes is a mix-in for data types that contain data types. A registry creates them with the _from_json_nested hook, passing its own match_json as the resolver. Other data types are created with from_json as before, so data types that override from_json keep working. - Structured and Struct are HasNestedDTypes, and their from_json takes an optional resolver (the default registry when omitted). - get_data_type_from_json, get_data_type_from_native_dtype, parse_dtype and parse_data_type take an optional registry (the default registry when omitted). Assisted-by: ClaudeCode:claude-opus-5-5 * fix(dtype): load entry point data types when the registry is first used zarr.registry collects data type entry points into DataTypeRegistry._lazy_load_list, but nothing in src called _lazy_load, so data types registered through the "zarr.data_type" entry point group never loaded. get, match_dtype and match_json now load them first, as the other registries do in their getters. _lazy_load imports each entry point once instead of twice. The test fixture that clears pending entry points now clears the data type registry's too, since _collect_entrypoints does not return it, and the entry point test no longer loads them by hand. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: add changelog fragment Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): resolve nested data types in a DTypeContext with a path Replaces the DTypeResolver callable with a context object, as mature decoders do (Jackson's DeserializationContext, Gson's Gson instance, PyYAML's loader), so it can grow without changing the hook's signature. - DTypeContext (exported from zarr.dtype) holds the registry, the Zarr format and the path of the data type within the data types that contain it. Composites resolve each contained data type with context.child(name).resolve(json). The Zarr format is part of the context because contained data types always share their container's format. - HasNestedDTypes._from_json_nested takes the context. - Structured.from_json / Struct.from_json take registry= like the other entry points, instead of a resolver. - NestedDataTypeValidationError (a DataTypeValidationError) reports a contained data type that matches nothing, with its path. The registry re-raises it instead of trying other data types, since the JSON has the shape of the containing data type. This restores a specific error for an invalid struct field (e.g. "... matches {'name': '|i9', ...} at path ['a']"), which #351 had reduced to a mismatch of the whole struct. Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): split resolution into a Context value and a Reading state DTypeContext mixed two things: which extensions a read may resolve (the same for a whole read, chosen by the caller) and where the read has got to (changing at every nested step). That does not extend to other kinds of extension: a codec read that meets a data type (cast_value) must hand the same extensions to a data type read. - zarr.registry.Context is the set of extensions, by kind. It holds only data_types for now; codecs, chunk key encodings and chunk grids can be added as fields with defaults. Context.default() refers to the default registries. This mirrors zarr-metadata's Context, passed as context=. - Reading (zarr.core.context) is the state of a read: the context, the Zarr format and the location in the document. Composites read contained extensions with reading.at(*keys).resolve_data_type(json). - The location is the JSON location in the document, reported as a JSON Pointer, so it stays unambiguous once codecs nest data types. A Zarr V2 data type is located by its name as written in the document. - The containing data type decides that a failure is nested: Structured raises NestedDataTypeValidationError for a field that matches nothing, since a non-empty location no longer means "inside another data type" once metadata parsing starts reads at /data_type. - The dtype entry points and Structured.from_json take context= instead of registry=. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: name the #4385 changelog fragment after its pull request It was merged under its fork pull request number (351), which towncrier would link to the wrong pull request. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: name the changelog fragment after the upstream pull request Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): rename Reading to Resolver A Resolver is a Context positioned in a document; it resolves the extensions there. The pair mirrors referencing.Registry and referencing.Resolver in python-jsonschema. "Reading" read as a verb. Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): give every data type the resolver hook ZDType._from_json_resolved(data, *, resolver) is the one way a registry creates a data type. Its default ignores the resolver and calls from_json in the resolver's Zarr format, so data types that override from_json keep working; a data type that contains others (Structured, Struct) overrides it to resolve them. This replaces the HasNestedDTypes mix-in and the registry's issubclass branch: every type receives the context and leaves ignore it, as in pydantic's __get_pydantic_core_schema__(source, handler), serde's Deserialize, and zarr-metadata's resolve(data, context). Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): keep the resolution context internal Context, the context= keywords, and the nested data type error were public API without a caller that needs them, and their shape is expected to change once codecs join the context or zarr-python reads metadata through zarr-metadata. - zarr.registry no longer exports Context; it stays in zarr.core.context. - get_data_type_from_json, get_data_type_from_native_dtype, parse_dtype, parse_data_type and Structured.from_json are unchanged from main. Structured's _from_json_v2/_v3 resolve fields with the default registry; a DataTypeRegistry resolves them with itself through _from_json_resolved. - The nested error is _NestedDataTypeValidationError in the dtype registry module, a DataTypeValidationError, and no longer in zarr.errors. What remains user-visible is bug fixes: a structured data type read by a DataTypeRegistry resolves its fields with that registry, a bad field is reported with its JSON location, and entry point data types load. Assisted-by: ClaudeCode:claude-opus-5-5
1 parent 9abad7f commit 5c37f97

8 files changed

Lines changed: 298 additions & 26 deletions

File tree

File renamed without changes.

‎changes/4387.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
A structured data type read by a `DataTypeRegistry` now resolves its field data types with that registry, instead of always with the default registry. A structured data type with a field whose data type matches no data type now reports the location of that field in the JSON (e.g. `at /configuration/fields/1/data_type`), instead of reporting that the whole structured data type matches nothing. Data types registered through the `zarr.data_type` entry point group are now loaded; previously they were collected but never registered.

‎src/zarr/core/context.py‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
"""
2+
The extensions that Zarr metadata is read with, and the resolver that reads with them.
3+
4+
A `Context` is the set of extensions a read may resolve, by kind. It is a value: the same for a
5+
whole read. A `Resolver` is a context positioned in a document: the context, the Zarr format of the
6+
document, and the location in the document. It is re-positioned at every nested step, and is
7+
threaded through the read by zarr. (The pair mirrors `referencing.Registry` and
8+
`referencing.Resolver` in python-jsonschema.)
9+
10+
Both are internal: nothing in the public API takes a context yet.
11+
"""
12+
13+
from __future__ import annotations
14+
15+
from dataclasses import dataclass, replace
16+
from typing import TYPE_CHECKING
17+
18+
if TYPE_CHECKING:
19+
from zarr.core.common import ZarrFormat
20+
from zarr.core.dtype.common import DTypeJSON
21+
from zarr.core.dtype.registry import DataTypeRegistry
22+
from zarr.core.dtype.wrapper import TBaseDType, TBaseScalar, ZDType
23+
24+
25+
@dataclass(frozen=True, kw_only=True)
26+
class Context:
27+
"""
28+
The extensions that Zarr metadata is read with, by kind.
29+
30+
Attributes
31+
----------
32+
data_types : DataTypeRegistry
33+
The data types a read may resolve.
34+
"""
35+
36+
data_types: DataTypeRegistry
37+
38+
@classmethod
39+
def default(cls) -> Context:
40+
"""
41+
The context of the default registries, such as `zarr.dtype.data_type_registry`.
42+
43+
It refers to the registries themselves, so it includes extensions registered after it is
44+
created.
45+
"""
46+
# avoid circular import
47+
from zarr.core.dtype import data_type_registry
48+
49+
return cls(data_types=data_type_registry)
50+
51+
52+
type JSONLocation = tuple[str | int, ...]
53+
54+
55+
def json_pointer(loc: JSONLocation) -> str:
56+
"""
57+
The JSON Pointer (RFC 6901) of a location in a JSON document, e.g. `/codecs/0/configuration`.
58+
"""
59+
return "".join("/" + str(key).replace("~", "~0").replace("/", "~1") for key in loc)
60+
61+
62+
@dataclass(frozen=True, kw_only=True)
63+
class Resolver:
64+
"""
65+
Resolves the extensions in a Zarr metadata document from a context, at a location in the
66+
document.
67+
68+
An extension that contains other extensions, such as a structured data type, resolves each of
69+
them with the resolver returned by `at`, so that they come from the same context, use the same
70+
Zarr format, and report errors with their location.
71+
72+
Attributes
73+
----------
74+
context : Context
75+
The extensions the read may resolve.
76+
zarr_format : ZarrFormat
77+
The Zarr format of the document.
78+
loc : tuple[str | int, ...]
79+
The location in the document of the JSON being read, as keys and indices. A Zarr V2 data
80+
type is located by its name as written in the document (the value of `"dtype"`), not by
81+
the `{"name": ..., "object_codec_id": ...}` form zarr reads it in.
82+
"""
83+
84+
context: Context
85+
zarr_format: ZarrFormat
86+
loc: JSONLocation = ()
87+
88+
def at(self, *keys: str | int) -> Resolver:
89+
"""
90+
The resolver for the JSON at `keys` within the JSON at this resolver's location.
91+
"""
92+
return replace(self, loc=(*self.loc, *keys))
93+
94+
def resolve_data_type(self, data: DTypeJSON) -> ZDType[TBaseDType, TBaseScalar]:
95+
"""
96+
Resolve the JSON representation of a data type at this resolver's location.
97+
98+
Raises
99+
------
100+
ValueError
101+
If no data type in the context matches `data`.
102+
"""
103+
return self.context.data_types._match_json(data, resolver=self)

‎src/zarr/core/dtype/npy/structured.py‎

Lines changed: 53 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
from numpy.lib.recfunctions import repack_fields
1111

1212
from zarr.core.common import NamedConfig
13+
from zarr.core.context import Context, Resolver
1314
from zarr.core.dtype.common import (
1415
DTypeConfig_V2,
1516
DTypeJSON,
@@ -24,6 +25,7 @@
2425
bytes_to_json,
2526
check_json_str,
2627
)
28+
from zarr.core.dtype.registry import _NestedDataTypeValidationError
2729
from zarr.core.dtype.wrapper import TBaseDType, TBaseScalar, ZDType
2830
from zarr.errors import DataTypeValidationError, ZarrUserWarning
2931

@@ -74,20 +76,22 @@ def _unsupported_field_feature(dtype: np.dtype[np.void]) -> str | None:
7476
return None
7577

7678

77-
def _field_from_json(
78-
data: DTypeJSON, *, zarr_format: ZarrFormat
79-
) -> ZDType[TBaseDType, TBaseScalar]:
79+
def _default_resolver(*, zarr_format: ZarrFormat) -> Resolver:
80+
"""A resolver for the default registry, at the top of a document."""
81+
return Resolver(context=Context.default(), zarr_format=zarr_format)
82+
83+
84+
def _resolve_field(data: DTypeJSON, resolver: Resolver) -> ZDType[TBaseDType, TBaseScalar]:
8085
"""
81-
Parse the data type of a structured field, reporting an unrecognized field data type as an
82-
invalid structured data type.
86+
Resolve the data type of a structured field. A field data type that matches nothing makes the
87+
structured data type invalid, rather than a different data type.
8388
"""
84-
# avoid circular import
85-
from zarr.core.dtype import get_data_type_from_json
86-
8789
try:
88-
return get_data_type_from_json(data, zarr_format=zarr_format)
90+
return resolver.resolve_data_type(data)
91+
except DataTypeValidationError:
92+
raise
8993
except ValueError as e:
90-
raise DataTypeValidationError(f"Invalid structured field data type: {e}") from e
94+
raise _NestedDataTypeValidationError(str(e)) from e
9195

9296

9397
class StructuredJSON_V2(DTypeConfig_V2[StructuredName_V2, None]):
@@ -367,32 +371,60 @@ def _check_json_v3(cls, data: DTypeJSON) -> TypeGuard[StructuredJSON_V3]:
367371

368372
@classmethod
369373
def _from_json_v2(cls, data: DTypeJSON) -> Self:
374+
# the fields are resolved from the default registry
375+
return cls._from_json_resolved_v2(data, resolver=_default_resolver(zarr_format=2))
376+
377+
@classmethod
378+
def _from_json_v3(cls, data: DTypeJSON) -> Self:
379+
# the fields are resolved from the default registry
380+
return cls._from_json_resolved_v3(data, resolver=_default_resolver(zarr_format=3))
381+
382+
@classmethod
383+
def _from_json_resolved(cls, data: DTypeJSON, *, resolver: Resolver) -> Self:
384+
if resolver.zarr_format == 2:
385+
return cls._from_json_resolved_v2(data, resolver=resolver)
386+
if resolver.zarr_format == 3:
387+
return cls._from_json_resolved_v3(data, resolver=resolver)
388+
raise ValueError(
389+
f"zarr_format must be 2 or 3, got {resolver.zarr_format}"
390+
) # pragma: no cover
391+
392+
@classmethod
393+
def _from_json_resolved_v2(cls, data: DTypeJSON, *, resolver: Resolver) -> Self:
370394
if cls._check_json_v2(data):
371395
# structured dtypes are constructed directly from a list of lists
372396
# note that we do not handle the object codec here! this will prevent structured
373397
# dtypes from containing object dtypes.
374398
name = data["name"]
375399
return cls(
376-
fields=tuple( # type: ignore[str-unpack]
400+
fields=tuple(
377401
( # type: ignore[misc]
378402
f_name,
379-
_field_from_json({"name": f_dtype, "object_codec_id": None}, zarr_format=2),
403+
_resolve_field(
404+
{"name": f_dtype, "object_codec_id": None},
405+
# the location in the data type as written in the document, which
406+
# is the "name" of the Zarr V2 data type JSON
407+
resolver.at(index, 1),
408+
),
380409
)
381-
for f_name, f_dtype in name
410+
for index, (f_name, f_dtype) in enumerate(name)
382411
)
383412
)
384413
msg = f"Invalid JSON representation of {cls.__name__}. Got {data!r}, expected a JSON array of arrays"
385414
raise DataTypeValidationError(msg)
386415

387416
@classmethod
388-
def _from_json_v3(cls, data: DTypeJSON) -> Self:
417+
def _from_json_resolved_v3(cls, data: DTypeJSON, *, resolver: Resolver) -> Self:
389418
if cls._check_json_v3(data):
390419
config = data["configuration"]
391420
meta_fields = config["fields"]
392421
return cls(
393422
fields=tuple(
394-
(f_name, _field_from_json(f_dtype, zarr_format=3)) # type: ignore[misc]
395-
for f_name, f_dtype in meta_fields
423+
( # type: ignore[misc]
424+
f_name,
425+
_resolve_field(f_dtype, resolver.at("configuration", "fields", index, 1)),
426+
)
427+
for index, (f_name, f_dtype) in enumerate(meta_fields)
396428
)
397429
)
398430
msg = f"Invalid JSON representation of {cls.__name__}. Got {data!r}, expected a JSON object with the key {cls._zarr_v3_name!r}"
@@ -645,19 +677,21 @@ def _check_json_v3(cls, data: DTypeJSON) -> TypeGuard[StructJSON_V3]: # type: i
645677
)
646678

647679
@classmethod
648-
def _from_json_v3(cls, data: DTypeJSON) -> Self:
680+
def _from_json_resolved_v3(cls, data: DTypeJSON, *, resolver: Resolver) -> Self:
649681
if cls._check_json_v3(data):
650682
config = data["configuration"]
651683
meta_fields = config["fields"]
652684
parsed_fields: list[tuple[str, ZDType[TBaseDType, TBaseScalar]]] = []
653-
for field in meta_fields:
685+
for index, field in enumerate(meta_fields):
654686
if isinstance(field, dict):
655687
f_name = field["name"]
656688
f_dtype = field["data_type"]
689+
f_resolver = resolver.at("configuration", "fields", index, "data_type")
657690
else:
658691
# Legacy tuple-style field format from "structured" dtype
659692
f_name, f_dtype = field # type: ignore[unreachable]
660-
parsed_fields.append((f_name, _field_from_json(f_dtype, zarr_format=3))) # type: ignore[arg-type]
693+
f_resolver = resolver.at("configuration", "fields", index, 1)
694+
parsed_fields.append((f_name, _resolve_field(f_dtype, f_resolver))) # type: ignore[arg-type]
661695
return cls(fields=tuple(parsed_fields))
662696
msg = f"Invalid JSON representation of {cls.__name__}. Got {data!r}, expected a JSON object with the key {cls._zarr_v3_name!r}"
663697
raise DataTypeValidationError(msg)

‎src/zarr/core/dtype/registry.py‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66

77
import numpy as np
88

9+
from zarr.core.context import Context, Resolver, json_pointer
910
from zarr.errors import DataTypeValidationError
1011

1112
if TYPE_CHECKING:
@@ -70,6 +71,16 @@ def _v2_spellings(data: DTypeJSON) -> tuple[DTypeJSON, ...]:
7071
return (data,)
7172

7273

74+
class _NestedDataTypeValidationError(DataTypeValidationError):
75+
"""
76+
A data type contains a data type that is invalid, such as a structured data type with a field
77+
whose data type matches no data type.
78+
79+
The JSON has the shape of the containing data type, so a data type registry re-raises this
80+
error rather than trying other data types.
81+
"""
82+
83+
7384
# This class is different from the other registry classes, which inherit from
7485
# dict. IMO it's simpler to just do a dataclass. But long-term we should
7586
# have just 1 registry class in use.
@@ -100,7 +111,8 @@ def _lazy_load(self) -> None:
100111
the registry. After loading, clear the lazy load list.
101112
"""
102113
for e in self._lazy_load_list:
103-
self.register(e.load()._zarr_v3_name, e.load())
114+
cls = e.load()
115+
self.register(cls._zarr_v3_name, cls)
104116

105117
self._lazy_load_list.clear()
106118

@@ -166,6 +178,7 @@ def get(self, key: str) -> type[ZDType[TBaseDType, TBaseScalar]]:
166178
If the key is not found in the registry.
167179
"""
168180

181+
self._lazy_load()
169182
return self.contents[key]
170183

171184
def match_dtype(self, dtype: TBaseDType) -> ZDType[TBaseDType, TBaseScalar]:
@@ -200,6 +213,7 @@ def match_dtype(self, dtype: TBaseDType) -> ZDType[TBaseDType, TBaseScalar]:
200213
constructed.
201214
"""
202215

216+
self._lazy_load()
203217
if dtype == np.dtype("O"):
204218
msg = (
205219
f"Zarr data type resolution from {dtype} failed. "
@@ -256,11 +270,27 @@ def match_json(
256270
If no matching Zarr data type is found for the given JSON data.
257271
"""
258272

259-
candidates = _v2_spellings(data) if zarr_format == 2 else (data,)
273+
resolver = Resolver(context=Context(data_types=self), zarr_format=zarr_format)
274+
return self._match_json(data, resolver=resolver)
275+
276+
def _match_json(
277+
self, data: DTypeJSON, *, resolver: Resolver
278+
) -> ZDType[TBaseDType, TBaseScalar]:
279+
"""
280+
Match a JSON representation of a data type to a registered ZDType, at `resolver`'s location.
281+
"""
282+
self._lazy_load()
283+
candidates = _v2_spellings(data) if resolver.zarr_format == 2 else (data,)
260284
for candidate in candidates:
261285
for val in self.contents.values():
262286
try:
263-
return val.from_json(candidate, zarr_format=zarr_format)
287+
return val._from_json_resolved(candidate, resolver=resolver)
288+
except _NestedDataTypeValidationError:
289+
# the JSON is this data type, and a data type it contains is invalid
290+
raise
264291
except DataTypeValidationError:
265292
pass
266-
raise ValueError(f"No Zarr data type found that matches {data!r}")
293+
msg = f"No Zarr data type found that matches {data!r}"
294+
if resolver.loc:
295+
msg += f" at {json_pointer(resolver.loc)}"
296+
raise ValueError(msg)

‎src/zarr/core/dtype/wrapper.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838

3939
if TYPE_CHECKING:
4040
from zarr.core.common import JSON, ZarrFormat
41+
from zarr.core.context import Resolver
4142
from zarr.core.dtype.common import DTypeJSON, DTypeSpec_V2, DTypeSpec_V3
4243

4344
# This the upper bound for the scalar types we support. It's numpy scalars + str,
@@ -157,6 +158,18 @@ def from_json(cls: type[Self], data: DTypeJSON, *, zarr_format: ZarrFormat) -> S
157158
return cls._from_json_v3(data)
158159
raise ValueError(f"zarr_format must be 2 or 3, got {zarr_format}") # pragma: no cover
159160

161+
@classmethod
162+
def _from_json_resolved(cls: type[Self], data: DTypeJSON, *, resolver: Resolver) -> Self:
163+
"""
164+
Create an instance of this ZDType from JSON data at `resolver`'s location.
165+
166+
A data type registry creates every data type with this method. A data type that contains
167+
other data types, such as a structured data type, overrides it to resolve each of them
168+
with `resolver.at(...).resolve_data_type(...)`, so that they come from the same context.
169+
Other data types ignore the resolver: the default is `from_json` in its Zarr format.
170+
"""
171+
return cls.from_json(data, zarr_format=resolver.zarr_format)
172+
160173
@overload
161174
def to_json(self, zarr_format: Literal[2]) -> DTypeSpec_V2: ...
162175

‎tests/conftest.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
)
4141
from zarr.core.config import config as zarr_config
4242
from zarr.core.dtype import (
43+
data_type_registry,
4344
get_data_type_from_native_dtype,
4445
)
4546
from zarr.core.dtype.common import HasItemSize
@@ -235,6 +236,7 @@ def _clear_registries() -> None:
235236
registries = zarr.registry._collect_entrypoints()
236237
for registry in registries:
237238
registry.lazy_load_list.clear()
239+
data_type_registry._lazy_load_list.clear()
238240

239241

240242
@pytest.fixture

0 commit comments

Comments
 (0)