Skip to content

Commit 4e39185

Browse files
committed
feat: name packages that provide a codec zarr cannot find
Closes zarr-developers#4271. When zarr fails to resolve a codec it now says which Python packages are known to provide it, instead of raising a bare KeyError holding only the codec name: An implementation for codec 'wavpack' is not available. Register one explicitly using the codec registry (see <docs>), or install a Python package that registers a codec implementation with numcodecs. Known packages supporting this codec: wavpack-numcodecs. Two hand-maintained tables in zarr/registry.py hold the mapping, one per Zarr format, because the two formats resolve codecs through different registries and the same name can mean different things in each: `imagecodecs_*` names are declared by `virtual-tiff` under the `zarr.codecs` entry point group and by `imagecodecs-numcodecs` under `numcodecs.codecs`, and `crc32c` is a codec zarr implements itself in format 3 while in format 2 it needs `numcodecs[crc32c]`. Each table has an exact-match and a prefix-match half, since packages that provide many codecs namespace them behind a shared prefix. Entries cover third-party packages and the codecs numcodecs gates behind its own optional dependencies -- `zfpy`, `pcodec`, `crc32c` and `msgpack2` -- which are the most common missing-codec case in practice. Backwards compatibility: `get_codec_class` now raises `zarr.errors.UnknownCodecError` instead of `KeyError`, both for a codec with no registered implementation and for a codec whose configured implementation is not registered. `get_numcodec` raises it instead of the ValueError numcodecs raises for an unregistered format 2 codec id. All are subclasses of `ValueError`. Carrying the message on a `KeyError` was not an option: `KeyError.__str__` reprs its argument, so a multi-sentence message comes back quoted and escaped. `UnknownCodecError` is now exported from `zarr.errors`, since users are being told to catch it. `get_numcodec` supports numcodecs down to the declared 0.14 floor: `numcodecs.errors` only exists from 0.15.1, so the unregistered-codec check prefers that exception type where it is importable and falls back to matching the message otherwise. Signed-off-by: arcusbuilds <[email protected]>
1 parent ce10c0b commit 4e39185

9 files changed

Lines changed: 443 additions & 29 deletions

File tree

‎changes/4277.feature.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
When Zarr cannot find an implementation for a codec, the error now names Python packages known to
2+
provide that codec, e.g. `An implementation for codec 'wavpack' is not available. Register one
3+
explicitly using the codec registry (see ...), or install a Python package that registers a codec
4+
implementation with numcodecs. Known packages supporting this codec: wavpack-numcodecs.` This
5+
covers codecs that `numcodecs` itself gates behind an optional dependency — `zfpy`, `pcodec`,
6+
`crc32c` and `msgpack2` now point at the matching `numcodecs[...]` extra — as well as third-party
7+
packages. Codec authors can add their published package to the table in `zarr/registry.py`.
8+
9+
`zarr.registry.get_codec_class` now raises `zarr.errors.UnknownCodecError` instead of `KeyError`,
10+
both for a codec with no registered implementation and for a codec whose configured
11+
implementation is not registered. `zarr.registry.get_numcodec` raises it instead of the
12+
`ValueError` (`numcodecs.errors.UnknownCodecError` on numcodecs 0.15.1 and later) that numcodecs
13+
raises for an unregistered Zarr format 2 codec id. All of these are subclasses of `ValueError`.
14+
`zarr.errors.UnknownCodecError` is now exported from `zarr.errors`.

‎docs/user-guide/extending.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,13 @@ New codecs need to have their own unique identifier. To avoid naming collisions,
6363
strongly recommended to prefix the codec identifier with a unique name. For example,
6464
the codecs from `numcodecs` are prefixed with `numcodecs.`, e.g. `numcodecs.delta`.
6565

66+
If someone opens an array that uses your codec without your package installed, Zarr raises
67+
[`zarr.errors.UnknownCodecError`][] explaining how to register an implementation. Zarr also
68+
keeps a small table of codec names and the published packages that provide them, and names
69+
those packages in that error. Once your package is on PyPI, please open a pull request adding
70+
it to the codec-package table in `zarr/registry.py`, so that users get a message telling them
71+
exactly what to install.
72+
6673
!!! note
6774
Note that the extension mechanism for the Zarr format 3 is still under development.
6875
Requirements for custom codecs including the choice of codec identifiers might

‎src/zarr/core/metadata/v3.py‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
from zarr.core.dtype.common import check_dtype_spec_v3
3737
from zarr.core.json_parse import parse_field, validate_json_value
3838
from zarr.core.metadata.common import parse_attributes
39-
from zarr.errors import MetadataValidationError, NodeTypeValidationError, UnknownCodecError
39+
from zarr.errors import MetadataValidationError, NodeTypeValidationError
4040
from zarr.registry import get_codec_class
4141

4242
if TYPE_CHECKING:
@@ -74,10 +74,7 @@ def parse_codecs(data: object) -> tuple[Codec, ...]:
7474
else:
7575
name_parsed, _ = parse_named_configuration(c, require_configuration=False)
7676

77-
try:
78-
out += (get_codec_class(name_parsed).from_dict(c),)
79-
except KeyError as e:
80-
raise UnknownCodecError(f"Unknown codec: {e.args[0]!r}") from e
77+
out += (get_codec_class(name_parsed).from_dict(c),)
8178

8279
return out
8380

‎src/zarr/errors.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"MetadataValidationError",
1313
"NegativeStepError",
1414
"NodeTypeValidationError",
15+
"UnknownCodecError",
1516
"UnstableSpecificationWarning",
1617
"VindexInvalidSelectionError",
1718
"ZarrDeprecationWarning",

‎src/zarr/metadata/migrate_v3.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
from zarr.core.metadata.v2 import ArrayV2Metadata
3030
from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata
3131
from zarr.core.sync import sync
32+
from zarr.errors import UnknownCodecError
3233
from zarr.registry import get_codec_class
3334
from zarr.storage import StorePath
3435
from zarr.types import AnyArray
@@ -273,7 +274,7 @@ def _find_numcodecs_zarr3(numcodecs_codec: numcodecs.abc.Codec) -> Codec:
273274

274275
try:
275276
codec_v3 = get_codec_class(numcodec_name)
276-
except KeyError as exc:
277+
except UnknownCodecError as exc:
277278
raise ValueError(
278279
f"Couldn't find corresponding zarr.codecs.numcodecs codec for {numcodecs_codec.codec_id}"
279280
) from exc

‎src/zarr/registry.py‎

Lines changed: 153 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
from zarr.core.config import BadConfigError, config
99
from zarr.core.dtype import data_type_registry
10-
from zarr.errors import ZarrUserWarning
10+
from zarr.errors import UnknownCodecError, ZarrUserWarning
1111

1212
if TYPE_CHECKING:
1313
from importlib.metadata import EntryPoint
@@ -23,7 +23,7 @@
2323
from zarr.abc.numcodec import Numcodec
2424
from zarr.core.buffer import Buffer, NDBuffer
2525
from zarr.core.chunk_key_encodings import ChunkKeyEncoding
26-
from zarr.core.common import JSON
26+
from zarr.core.common import JSON, ZarrFormat
2727

2828
__all__ = [
2929
"Registry",
@@ -39,6 +39,133 @@
3939
"register_pipeline",
4040
]
4141

42+
_ZARR_CODEC_DOCS_URL = "https://zarr.readthedocs.io/en/stable/user-guide/extending/#custom-codecs"
43+
_NUMCODECS_CODEC_DOCS_URL = (
44+
"https://numcodecs.readthedocs.io/en/stable/registry.html#numcodecs.registry.register_codec"
45+
)
46+
47+
# Codecs zarr-python does not implement, mapped to the names of Python packages that do.
48+
# These tables exist purely to make the "no implementation for this codec" error actionable;
49+
# nothing here affects which codecs zarr can actually read or write. Values are what you would
50+
# pass to `pip install`. Only add an entry you have verified against the package's declared
51+
# entry points, and only for a package that is actually published.
52+
#
53+
# The two Zarr formats resolve codecs through different registries, so they get different
54+
# tables: a name can mean one thing as a Zarr format 3 codec name and another as a Zarr
55+
# format 2 codec id. `imagecodecs_*` is exactly that -- `virtual-tiff` declares those names
56+
# under `zarr.codecs`, while `imagecodecs-numcodecs` declares them under `numcodecs.codecs`.
57+
58+
# Zarr format 3 codec names (entry point group "zarr.codecs").
59+
_CODEC_PACKAGES: dict[str, tuple[str, ...]] = {
60+
"gribberish": ("gribberish",),
61+
"n5_default": ("zarr-n5",),
62+
}
63+
64+
# As `_CODEC_PACKAGES`, but each key is matched against the start of the codec name. Packages
65+
# that provide many codecs namespace them behind a shared prefix, so one entry covers them all.
66+
_CODEC_PACKAGE_PREFIXES: dict[str, tuple[str, ...]] = {
67+
"any-numcodecs.": ("zarr-any-numcodecs",),
68+
"imagecodecs_": ("virtual-tiff",),
69+
"omfiles.": ("omfiles",),
70+
"virtual_tiff.": ("virtual-tiff",),
71+
}
72+
73+
# Zarr format 2 codec ids (entry point group "numcodecs.codecs"). `numcodecs` itself gates
74+
# several of its own codecs behind optional dependencies, so the package to install for those
75+
# is an extra of numcodecs rather than a third-party distribution.
76+
_NUMCODEC_PACKAGES: dict[str, tuple[str, ...]] = {
77+
"FITSAscii": ("kerchunk",),
78+
"FITSVarBintable": ("kerchunk",),
79+
"crc32c": ("numcodecs[crc32c]",),
80+
"fill_hdf_strings": ("kerchunk",),
81+
"grib": ("kerchunk",),
82+
"msgpack2": ("numcodecs[msgpack]",),
83+
"pcodec": ("numcodecs[pcodec]",),
84+
"rawgrib": ("gribscan",),
85+
"record_member": ("kerchunk",),
86+
"vc-delta3d": ("vc-delta3d",),
87+
"wavpack": ("wavpack-numcodecs",),
88+
"zfpy": ("numcodecs[zfpy]",),
89+
}
90+
91+
# As `_NUMCODEC_PACKAGES`, but matched against the start of the codec id.
92+
_NUMCODEC_PACKAGE_PREFIXES: dict[str, tuple[str, ...]] = {
93+
"gribscan.": ("gribscan",),
94+
"imagecodecs_": ("imagecodecs-numcodecs",),
95+
}
96+
97+
98+
def _packages_for_codec(name: str, *, zarr_format: ZarrFormat) -> tuple[str, ...]:
99+
"""
100+
Names of Python packages known to provide an implementation of the codec ``name``.
101+
102+
Returns an empty tuple if we don't know of any.
103+
104+
Parameters
105+
----------
106+
name : str
107+
The codec name (Zarr format 3) or codec id (Zarr format 2) we failed to resolve.
108+
zarr_format : ZarrFormat
109+
Which registry the codec was looked up in.
110+
"""
111+
if zarr_format == 2:
112+
exact, prefixes = _NUMCODEC_PACKAGES, _NUMCODEC_PACKAGE_PREFIXES
113+
else:
114+
exact, prefixes = _CODEC_PACKAGES, _CODEC_PACKAGE_PREFIXES
115+
if name in exact:
116+
return exact[name]
117+
for prefix, packages in prefixes.items():
118+
if name.startswith(prefix):
119+
return packages
120+
return ()
121+
122+
123+
def _missing_codec_message(name: str, *, zarr_format: ZarrFormat) -> str:
124+
"""
125+
Build the error message raised when no implementation of the codec ``name`` is available.
126+
127+
Parameters
128+
----------
129+
name : str
130+
The codec name (Zarr format 3) or codec id (Zarr format 2) we failed to resolve.
131+
zarr_format : ZarrFormat
132+
Which registry the codec was looked up in. Zarr format 2 codecs are resolved through
133+
numcodecs, so that case points at the numcodecs registry rather than at zarr's.
134+
"""
135+
if zarr_format == 2:
136+
docs_url, registry = _NUMCODECS_CODEC_DOCS_URL, "numcodecs"
137+
else:
138+
docs_url, registry = _ZARR_CODEC_DOCS_URL, "zarr"
139+
msg = (
140+
f"An implementation for codec {name!r} is not available. Register one explicitly "
141+
f"using the codec registry (see {docs_url}), or install a Python package that "
142+
f"registers a codec implementation with {registry}."
143+
)
144+
packages = _packages_for_codec(name, zarr_format=zarr_format)
145+
if packages:
146+
msg += f" Known packages supporting this codec: {', '.join(packages)}."
147+
return msg
148+
149+
150+
def _is_missing_numcodec_error(exc: ValueError) -> bool:
151+
"""
152+
Whether ``exc`` from ``numcodecs.registry.get_codec`` means "this codec isn't registered".
153+
154+
numcodecs 0.15.1 added ``numcodecs.errors.UnknownCodecError`` for this. Older versions,
155+
down to the 0.14 floor zarr supports, raise a plain ``ValueError`` instead, so fall back to
156+
matching the message numcodecs has used throughout.
157+
158+
Parameters
159+
----------
160+
exc : ValueError
161+
The exception ``numcodecs.registry.get_codec`` raised.
162+
"""
163+
try:
164+
from numcodecs.errors import UnknownCodecError as NumcodecsUnknownCodecError
165+
except ImportError: # pragma: no cover - numcodecs < 0.15.1
166+
return str(exc).startswith("codec not available")
167+
return isinstance(exc, NumcodecsUnknownCodecError)
168+
42169

43170
class Registry[T](dict[str, type[T]]):
44171
def __init__(self) -> None:
@@ -168,7 +295,7 @@ def get_codec_class(key: str, reload_config: bool = False) -> type[Codec]:
168295

169296
codec_classes = _codec_registries[key]
170297
if not codec_classes:
171-
raise KeyError(key)
298+
raise UnknownCodecError(_missing_codec_message(key, zarr_format=3))
172299
config_entry = config.get("codecs", {}).get(key)
173300
if config_entry is None:
174301
if len(codec_classes) == 1:
@@ -179,11 +306,14 @@ def get_codec_class(key: str, reload_config: bool = False) -> type[Codec]:
179306
category=ZarrUserWarning,
180307
)
181308
return list(codec_classes.values())[-1]
182-
selected_codec_cls = codec_classes[config_entry]
183-
184-
if selected_codec_cls:
185-
return selected_codec_cls
186-
raise KeyError(key)
309+
selected_codec_cls = codec_classes.get(config_entry)
310+
if selected_codec_cls is None:
311+
raise UnknownCodecError(
312+
f"Codec {key!r} is configured to use the implementation {config_entry!r}, which is "
313+
f"not registered. Registered implementations of this codec: "
314+
f"{sorted(codec_classes)}."
315+
)
316+
return selected_codec_cls
187317

188318

189319
def _resolve_codec(data: dict[str, JSON]) -> Codec:
@@ -321,6 +451,11 @@ def get_numcodec(data: CodecJSON_V2[str]) -> Numcodec:
321451
-------
322452
codec : Numcodec
323453
454+
Raises
455+
------
456+
UnknownCodecError
457+
If no implementation of the codec is registered with numcodecs.
458+
324459
Examples
325460
--------
326461
```python
@@ -333,4 +468,13 @@ def get_numcodec(data: CodecJSON_V2[str]) -> Numcodec:
333468

334469
from numcodecs.registry import get_codec
335470

336-
return get_codec(data) # type: ignore[no-any-return]
471+
try:
472+
return get_codec(data) # type: ignore[no-any-return]
473+
except ValueError as e:
474+
# Read the codec id from the input rather than from the exception: numcodecs sets its
475+
# `codec_id` attribute to the repr of the id ("'wavpack'") rather than the id itself,
476+
# and older numcodecs versions raise a plain ValueError that carries no id at all.
477+
codec_id = data.get("id")
478+
if not _is_missing_numcodec_error(e) or not isinstance(codec_id, str):
479+
raise
480+
raise UnknownCodecError(_missing_codec_message(codec_id, zarr_format=2)) from e

‎tests/test_codecs/test_numcodecs.py‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -8,15 +8,10 @@
88
import pytest
99
from numcodecs import GZip
1010

11-
try:
12-
from numcodecs.errors import UnknownCodecError
13-
except ImportError:
14-
# Older versions of numcodecs don't have a separate errors module
15-
UnknownCodecError = ValueError
16-
1711
from zarr import config, create_array, open_array
1812
from zarr.abc.numcodec import _is_numcodec, _is_numcodec_cls
1913
from zarr.codecs import numcodecs as _numcodecs
14+
from zarr.errors import UnknownCodecError
2015
from zarr.registry import get_codec_class, get_numcodec
2116

2217
if TYPE_CHECKING:
@@ -297,12 +292,7 @@ def test_generic_checksum(codec_class: type[_numcodecs._NumcodecsBytesBytesCodec
297292
def test_generic_bytes_codec(codec_class: type[_numcodecs._NumcodecsArrayBytesCodec]) -> None:
298293
try:
299294
codec_class()._codec # noqa: B018
300-
except ValueError as e: # pragma: no cover
301-
if "codec not available" in str(e):
302-
pytest.xfail(f"{codec_class.codec_name} is not available: {e}") # type: ignore[misc]
303-
else:
304-
raise
305-
except ImportError as e: # pragma: no cover
295+
except (UnknownCodecError, ImportError) as e: # pragma: no cover
306296
pytest.xfail(f"{codec_class.codec_name} is not available: {e}") # type: ignore[misc]
307297

308298
data = np.arange(0, 256, dtype="float32").reshape((16, 16))

‎tests/test_config.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
from zarr.core.codec_pipeline import BatchedCodecPipeline
2525
from zarr.core.config import BadConfigError, config
2626
from zarr.core.indexing import SelectorTuple
27-
from zarr.errors import ChunkNotFoundError, ZarrUserWarning
27+
from zarr.errors import ChunkNotFoundError, UnknownCodecError, ZarrUserWarning
2828
from zarr.registry import (
2929
fully_qualified_name,
3030
get_buffer_class,
@@ -334,7 +334,7 @@ class NewCodec2(BytesCodec):
334334
pass
335335

336336
# error if codec is not registered
337-
with pytest.raises(KeyError):
337+
with pytest.raises(UnknownCodecError):
338338
get_codec_class("missing_codec")
339339

340340
# no warning if only one implementation is available

0 commit comments

Comments
 (0)