Skip to content

Commit 9b7b93b

Browse files
bhaskargurram-aid-v-bclaude
authored
feat: require opt-in to read data encoded with the pickle codec (zarr-developers#4465)
* feat: require opt-in to read data encoded with the pickle codec Decoding a chunk with the numcodecs `pickle` codec calls `pickle.loads` on bytes from the store, so opening an untrusted Zarr v2 array can run arbitrary code. Reading such data now raises `PickleCodecDisabledError` unless the new `array.allow_pickle` config option is `True`, set with `zarr.config.set({"array.allow_pickle": True})` or the environment variable `ZARR_ARRAY__ALLOW_PICKLE=True`. The check sits in `V2Codec._decode_sync`, before any codec runs, and covers the pickle codec as compressor or filter. Opening the array and writing with the pickle codec are unchanged. Closes zarr-developers#4404 * fix: require numcodecs 0.16.6+ (not 0.17.0) and make the pickle opt-in work with it numcodecs 0.16.6 and 0.17.1 refuse to unpickle unless the process opts in (GHSA-jqc8-qjfv-5m6r); 0.17.0 does not. Raise the floor and exclude 0.17.0 so a zarr upgrade carries the fix. The tests that read pickled data after opting in to zarr now also opt in to numcodecs, because its gate sits behind zarr's. A new test checks that zarr's opt-in does not bypass numcodecs. The PickleCodecDisabledError message, the config docs, and the changelog name both switches. Assisted-by: ClaudeCode:claude-fable-5.1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * refactor: name the zarr pickle opt-in after the numcodecs one The zarr setting is now array.allow_unsafe_pickle_decode (environment variable ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE), matching numcodecs' ALLOW_UNSAFE_PICKLE_DECODE and NUMCODECS_ALLOW_UNSAFE_PICKLE_DECODE, so users who learn one name can guess the other. No behavior change. Assisted-by: ClaudeCode:claude-fable-5.1 Co-Authored-By: Claude Fable 5.1 <[email protected]> --------- Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent e376ace commit 9b7b93b

9 files changed

Lines changed: 163 additions & 3 deletions

File tree

‎changes/4404.feature.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Reading data encoded with the `pickle` codec now requires an explicit opt-in, because unpickling data from an untrusted store can execute arbitrary code ([GHSA-jqc8-qjfv-5m6r](https://github.com/zarr-developers/numcodecs/security/advisories/GHSA-jqc8-qjfv-5m6r)). Enable it with `zarr.config.set({"array.allow_unsafe_pickle_decode": True})` or the environment variable `ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE=True`. Without the opt-in, reading such data raises `zarr.errors.PickleCodecDisabledError`. numcodecs has its own opt-in (`numcodecs.pickles.ALLOW_UNSAFE_PICKLE_DECODE`), which must also be enabled. Writing with the `pickle` codec is unchanged.

‎changes/4539.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
The minimum supported numcodecs version is now 0.16.6, and 0.17.0 is excluded. Those versions leave `Pickle.decode` enabled by default, which allows arbitrary code execution when reading an untrusted Zarr v2 array ([GHSA-jqc8-qjfv-5m6r](https://github.com/zarr-developers/numcodecs/security/advisories/GHSA-jqc8-qjfv-5m6r)).

‎docs/user-guide/config.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ Configuration options include the following:
4040
- Whether empty chunks are written to storage `array.write_empty_chunks`
4141
- Enable experimental rectilinear chunks `array.rectilinear_chunks`
4242
- Whether missing chunks are filled with the array's fill value on read `array.read_missing_chunks` (default `True`). Set to `False` to raise a [`ChunkNotFoundError`][zarr.errors.ChunkNotFoundError] instead.
43+
- Whether data encoded with the `pickle` codec can be read `array.allow_unsafe_pickle_decode` (default `False`). Unpickling can execute arbitrary code, so only enable this for data from a source you trust, e.g. with `zarr.config.set({'array.allow_unsafe_pickle_decode': True})` or the environment variable `ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE=True`. While it is disabled, reading such data raises a [`PickleCodecDisabledError`][zarr.errors.PickleCodecDisabledError]. Writing with the `pickle` codec is not affected. numcodecs has a separate opt-in for the same reason (`numcodecs.pickles.ALLOW_UNSAFE_PICKLE_DECODE = True`, or `NUMCODECS_ALLOW_UNSAFE_PICKLE_DECODE=1` before importing numcodecs); both must be enabled to read pickled data.
4344
- Async and threading options, e.g. `async.concurrency` and `threading.max_workers`
4445
- Selections of implementations of codecs, codec pipelines and buffers
4546
- Enabling GPU support with `zarr.config.enable_gpu()`. See [GPU support](gpu.md) for more.

‎pyproject.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,8 @@ requires-python = ">=3.12"
4747
dependencies = [
4848
'packaging>=22.0',
4949
'numpy>=2',
50-
'numcodecs>=0.16',
50+
# 0.16.6 and 0.17.1 refuse to unpickle untrusted data (GHSA-jqc8-qjfv-5m6r); 0.17.0 does not.
51+
'numcodecs>=0.16.6,!=0.17.0',
5152
'google-crc32c>=1.5',
5253
'typing_extensions>=4.14',
5354
'donfig>=0.8',

‎src/zarr/codecs/_v2.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
from numcodecs.compat import ensure_bytes, ensure_ndarray_like
99

1010
from zarr.abc.codec import ArrayBytesCodec
11+
from zarr.core.config import config
12+
from zarr.errors import PickleCodecDisabledError
1113
from zarr.registry import get_ndbuffer_class
1214

1315
if TYPE_CHECKING:
@@ -16,6 +18,29 @@
1618
from zarr.core.buffer import Buffer, NDBuffer
1719

1820

21+
def _check_pickle_allowed(codecs: tuple[Numcodec | None, ...]) -> None:
22+
"""
23+
Raise if any of ``codecs`` is the pickle codec and reading pickled data has not been enabled.
24+
25+
Unpickling can run arbitrary code, so data from an untrusted store must not be decoded
26+
with the pickle codec unless the user has explicitly opted in.
27+
"""
28+
if not any(getattr(codec, "codec_id", None) == "pickle" for codec in codecs):
29+
return
30+
# Compare with ``True`` rather than testing truthiness: donfig keeps an environment
31+
# variable it cannot parse as a Python literal (e.g. ``false``) as a non-empty string.
32+
if config.get("array.allow_unsafe_pickle_decode") is not True:
33+
raise PickleCodecDisabledError(
34+
"Refusing to decode data with the 'pickle' codec, because unpickling data "
35+
"from an untrusted source can execute arbitrary code. If you trust this data, "
36+
"enable the pickle codec with zarr.config.set({'array.allow_unsafe_pickle_decode': True}) "
37+
"or set the environment variable ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE=True. numcodecs has "
38+
"its own opt-in, which must be enabled as well: set "
39+
"numcodecs.pickles.ALLOW_UNSAFE_PICKLE_DECODE = True, or set the environment "
40+
"variable NUMCODECS_ALLOW_UNSAFE_PICKLE_DECODE=1 before importing numcodecs."
41+
)
42+
43+
1944
@dataclass(frozen=True)
2045
class V2Codec(ArrayBytesCodec):
2146
filters: tuple[Numcodec, ...] | None
@@ -28,6 +53,7 @@ def _decode_sync(
2853
chunk_bytes: Buffer,
2954
chunk_spec: ArraySpec,
3055
) -> NDBuffer:
56+
_check_pickle_allowed((self.compressor, *(self.filters or ())))
3157
cdata = chunk_bytes.as_array_like()
3258
# decompress
3359
if self.compressor:

‎src/zarr/core/config.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,9 @@ def enable_gpu(self) -> ConfigSet:
103103
"rectilinear_chunks": False,
104104
"sharding_coalesce_max_gap_bytes": 1 << 20, # 1 MiB
105105
"sharding_coalesce_max_bytes": 16 << 20, # 16 MiB
106+
# Decoding data with the pickle codec can execute arbitrary code,
107+
# so it must be explicitly enabled.
108+
"allow_unsafe_pickle_decode": False,
106109
},
107110
"async": {"concurrency": 10, "timeout": None},
108111
"threading": {"max_workers": None},

‎src/zarr/errors.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"MetadataValidationError",
1313
"NegativeStepError",
1414
"NodeTypeValidationError",
15+
"PickleCodecDisabledError",
1516
"UnknownCodecError",
1617
"UnstableSpecificationWarning",
1718
"VindexInvalidSelectionError",
@@ -101,6 +102,15 @@ class UnknownCodecError(BaseZarrError):
101102
"""
102103

103104

105+
class PickleCodecDisabledError(BaseZarrError):
106+
"""
107+
Raised when reading data encoded with the pickle codec without opting in.
108+
109+
Unpickling can execute arbitrary code, so decoding with the pickle codec is
110+
disabled unless the ``array.allow_unsafe_pickle_decode`` configuration option is set to ``True``.
111+
"""
112+
113+
104114
class NodeTypeValidationError(MetadataValidationError):
105115
"""
106116
Specialized exception when the node_type of the metadata document is incorrect.

‎tests/test_config.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ def test_config_defaults_set() -> None:
5959
"rectilinear_chunks": False,
6060
"sharding_coalesce_max_gap_bytes": 1 << 20,
6161
"sharding_coalesce_max_bytes": 16 << 20,
62+
"allow_unsafe_pickle_decode": False,
6263
},
6364
"async": {"concurrency": 10, "timeout": None},
6465
"threading": {"max_workers": None},

‎tests/test_v2.py‎

Lines changed: 118 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
import json
2+
import pickle
23
from pathlib import Path
34
from typing import Any, Literal
45

6+
import numcodecs.errors
7+
import numcodecs.pickles
58
import numpy as np
69
import pytest
7-
from numcodecs import Delta, Zlib
10+
from numcodecs import Delta, Pickle, VLenUTF8, Zlib
811
from numcodecs.blosc import Blosc
912
from numcodecs.zstd import Zstd
1013

@@ -20,7 +23,7 @@
2023
from zarr.core.dtype.wrapper import ZDType
2124
from zarr.core.group import Group
2225
from zarr.core.sync import sync
23-
from zarr.errors import ZarrDeprecationWarning
26+
from zarr.errors import PickleCodecDisabledError, ZarrDeprecationWarning
2427
from zarr.storage import MemoryStore, StorePath
2528

2629

@@ -310,3 +313,116 @@ def test_other_dtype_roundtrip(fill_value: bytes | None, tmp_path: Path) -> None
310313
za[...] = a
311314
za = zarr.open_array(store=array_path)
312315
assert (a == za[:]).all()
316+
317+
318+
def _pickle_compressed_array() -> zarr.Array[Any]:
319+
# Writing with the pickle codec is allowed without opting in, only decoding is gated.
320+
arr = zarr.create_array(
321+
store={},
322+
shape=(3,),
323+
dtype=VariableLengthUTF8(), # type: ignore[arg-type]
324+
zarr_format=2,
325+
filters=[VLenUTF8()],
326+
compressors=Pickle(),
327+
fill_value="",
328+
)
329+
arr[:] = np.array(["a", "bb", "ccc"])
330+
return arr
331+
332+
333+
@pytest.fixture
334+
def allow_numcodecs_unpickle(monkeypatch: pytest.MonkeyPatch) -> None:
335+
"""
336+
Opt in to numcodecs' own gate on ``Pickle.decode``.
337+
338+
numcodecs refuses to unpickle unless the process opts in (GHSA-jqc8-qjfv-5m6r). zarr's
339+
opt-in is separate, so tests that read pickled data after opting in to zarr also need this.
340+
"""
341+
monkeypatch.setattr(numcodecs.pickles, "ALLOW_UNSAFE_PICKLE_DECODE", True)
342+
343+
344+
def test_pickle_codec_read_disabled_by_default() -> None:
345+
"""Reading data encoded with the pickle codec fails unless the user opts in."""
346+
arr = _pickle_compressed_array()
347+
with pytest.raises(
348+
PickleCodecDisabledError, match=r"array\.allow_unsafe_pickle_decode"
349+
) as excinfo:
350+
arr[:]
351+
# numcodecs has its own opt-in; the message must name it so users are not left guessing
352+
assert "ALLOW_UNSAFE_PICKLE_DECODE" in str(excinfo.value)
353+
354+
355+
def test_pickle_codec_read_does_not_unpickle() -> None:
356+
"""A malicious chunk is not unpickled when the pickle codec is disabled."""
357+
358+
class Payload:
359+
def __reduce__(self) -> tuple[Any, ...]:
360+
return (exec, ("raise AssertionError('payload was unpickled')",))
361+
362+
meta = {
363+
"zarr_format": 2,
364+
"shape": [1],
365+
"chunks": [1],
366+
"dtype": "|O",
367+
"compressor": {"id": "pickle"},
368+
"fill_value": None,
369+
"filters": [{"id": "vlen-utf8"}],
370+
"order": "C",
371+
}
372+
buffer_cls = default_buffer_prototype().buffer
373+
store = {
374+
".zarray": buffer_cls.from_bytes(json.dumps(meta).encode()),
375+
"0": buffer_cls.from_bytes(pickle.dumps(Payload())),
376+
}
377+
arr = zarr.open_array(store=MemoryStore(store), mode="r", zarr_format=2)
378+
with pytest.raises(PickleCodecDisabledError):
379+
arr[:]
380+
381+
382+
def test_pickle_codec_zarr_opt_in_does_not_bypass_numcodecs(
383+
monkeypatch: pytest.MonkeyPatch,
384+
) -> None:
385+
"""Opting in to zarr does not disable numcodecs' own gate on ``Pickle.decode``."""
386+
monkeypatch.setattr(numcodecs.pickles, "ALLOW_UNSAFE_PICKLE_DECODE", False)
387+
arr = _pickle_compressed_array()
388+
with (
389+
config.set({"array.allow_unsafe_pickle_decode": True}),
390+
pytest.raises(numcodecs.errors.UnsafeDecodeError),
391+
):
392+
arr[:]
393+
394+
395+
@pytest.mark.usefixtures("allow_numcodecs_unpickle")
396+
def test_pickle_codec_read_enabled_via_config() -> None:
397+
arr = _pickle_compressed_array()
398+
with config.set({"array.allow_unsafe_pickle_decode": True}):
399+
np.testing.assert_array_equal(arr[:], np.array(["a", "bb", "ccc"]))
400+
401+
402+
@pytest.mark.usefixtures("allow_numcodecs_unpickle")
403+
@pytest.mark.parametrize(("env_value", "allowed"), [("True", True), ("false", False)])
404+
def test_pickle_codec_read_enabled_via_env_var(
405+
monkeypatch: pytest.MonkeyPatch, env_value: str, allowed: bool
406+
) -> None:
407+
"""
408+
The opt-in can be set with an environment variable. Only a value that parses to ``True``
409+
enables it: donfig keeps ``false`` as a (truthy) string, which must not enable it.
410+
"""
411+
arr = _pickle_compressed_array()
412+
monkeypatch.setenv("ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE", env_value)
413+
config.refresh()
414+
if allowed:
415+
np.testing.assert_array_equal(arr[:], np.array(["a", "bb", "ccc"]))
416+
else:
417+
with pytest.raises(PickleCodecDisabledError):
418+
arr[:]
419+
420+
421+
def test_pickle_opt_in_does_not_affect_other_codecs() -> None:
422+
"""Arrays that do not use the pickle codec are read without opting in."""
423+
assert config.get("array.allow_unsafe_pickle_decode") is False
424+
arr = zarr.create_array(
425+
store={}, shape=(3,), dtype="<i4", zarr_format=2, filters=[Delta(dtype="<i4")]
426+
)
427+
arr[:] = [1, 2, 3]
428+
np.testing.assert_array_equal(arr[:], [1, 2, 3])

0 commit comments

Comments
 (0)