Commit 28768dd

Reading data encoded with the `pickle` codec now requires an explicit opt-in, because unpickling data from an untrusted store can execute arbitrary code ([GHSA-jqc8-qjfv-5m6r](https://github.com/zarr-developers/numcodecs/security/advisories/GHSA-jqc8-qjfv-5m6r)). Enable it with `zarr.config.set({"array.allow_unsafe_pickle_decode": True})` or the environment variable `ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE=True`. Without the opt-in, reading such data raises `zarr.errors.PickleCodecDisabledError`. numcodecs has its own opt-in (`numcodecs.pickles.ALLOW_UNSAFE_PICKLE_DECODE`), which must also be enabled. Writing with the `pickle` codec is unchanged.

`zarr.open_array` and `zarr.open` (sync and async) now apply the `config` argument when opening an existing array. Previously `config` was only used when a new array was created, so `zarr.open_array(store, config={"read_missing_chunks": False})` silently ran an existing array with the global defaults. `config` is now a named parameter of both functions and of `Array.open`/`AsyncArray.open`. `zarr.open` raises a `TypeError` naming the problem when `config` is given for a group.

Fixed several ways `ZipStore` could lose data or fail. A store created with `mode="w"` or `mode="x"` now keeps its entries when it is used again after `close()`, `move()`, or unpickling, instead of truncating the archive. Every method now opens the archive on first use, under a lock, so `get`, `get_partial_values`, `set_if_not_exists`, and `clear` no longer raise `AttributeError` on a store that was never opened, and threads sharing a store no longer open the archive twice. An unpickled store now opens its archive on first use rather than when it is unpickled, so writing a dask array to a `ZipStore` no longer produces an unreadable file. Where reopening an archive would silently drop its entries, the store now raises instead: `RuntimeError` after a failed `close()`, `zipfile.BadZipFile` if the file is not a complete zip archive, and `io.UnsupportedOperation` for a file object that is not readable and seekable. The type of the `mode` parameter now includes `"x"`. Builds on [#3593](https://github.com/zarr-developers/zarr-python/pull/3593) by [@oelhammouchi](https://github.com/oelhammouchi).

The minimum supported numcodecs version is now 0.16.6, and 0.17.0 is excluded. Those versions leave `Pickle.decode` enabled by default, which allows arbitrary code execution when reading an untrusted Zarr v2 array ([GHSA-jqc8-qjfv-5m6r](https://github.com/zarr-developers/numcodecs/security/advisories/GHSA-jqc8-qjfv-5m6r)).

Zarr format 3 stores a single ordered list of codecs in the

field of array metadata. The

[Zarr format 3 specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#chunk-encoding)

classifies each codec by the input and output of its encode transform, as

"array -> array", "array -> bytes", or "bytes -> bytes", and requires the list to

contain zero or more array -> array codecs, followed by exactly one

array -> bytes codec, followed by zero or more bytes -> bytes codecs.

Zarr-Python's array creation functions expose that list through three

parameters: `filters`, `serializer`, and `compressors`. These names are not

defined in the Zarr format 3 specification. They are Zarr-Python aliases for

the specification's codec kinds:

- `filters`: the array -> array codecs, given as

[`zarr.abc.codec.ArrayArrayCodec`][] instances. These transform chunk arrays

into chunk arrays before serialization.

- `serializer`: the array -> bytes codec, given as one

[`zarr.abc.codec.ArrayBytesCodec`][] instance. This transforms a chunk array

into bytes. Every Zarr format 3 array needs exactly one array -> bytes codec,

either supplied explicitly or chosen by default.

- `compressors`: the bytes -> bytes codecs, given as

[`zarr.abc.codec.BytesBytesCodec`][] instances. These transform bytes into

bytes after serialization.

The `compressors` parameter is only for bytes-to-bytes codecs. If a codec is an

`ArrayBytesCodec`, pass it with `serializer`, not `compressors`. For example, the

built-in [`zarr.codecs.BytesCodec`][] can be supplied explicitly as the serializer:

```python exec="true" session="arrays" source="above" result="ansi"

serializer = zarr.codecs.BytesCodec(endian="little")

z_explicit_serializer = zarr.create_array(

store="data/example-explicit-serializer.zarr",

shape=(100,),

chunks=(10,),

dtype="int32",

serializer=serializer,

compressors=None,

)

print(z_explicit_serializer.serializer)

print(f"Compressors: {z_explicit_serializer.compressors}")

The same rule applies to third-party Zarr format 3 codecs: if the codec is

documented as an `ArrayBytesCodec`, provide an instance as `serializer=...`.

Different bytes-to-bytes compressors can be provided via the `compressors` keyword

- Whether data encoded with the `pickle` codec can be read `array.allow_unsafe_pickle_decode` (default `False`). Unpickling can execute arbitrary code, so only enable this for data from a source you trust, e.g. with `zarr.config.set({'array.allow_unsafe_pickle_decode': True})` or the environment variable `ZARR_ARRAY__ALLOW_UNSAFE_PICKLE_DECODE=True`. While it is disabled, reading such data raises a [`PickleCodecDisabledError`][zarr.errors.PickleCodecDisabledError]. Writing with the `pickle` codec is not affected. numcodecs has a separate opt-in for the same reason (`numcodecs.pickles.ALLOW_UNSAFE_PICKLE_DECODE = True`, or `NUMCODECS_ALLOW_UNSAFE_PICKLE_DECODE=1` before importing numcodecs); both must be enabled to read pickled data.

# 0.16.6 and 0.17.1 refuse to unpickle untrusted data (GHSA-jqc8-qjfv-5m6r); 0.17.0 does not.

'numcodecs>=0.16.6,!=0.17.0',

config: ArrayConfigLike | None = None,

return AsyncArray(store_path=store_path, metadata=_metadata_dict, config=config)

_check_no_group_config(config, store_path)

return await open_array(

store=store_path, zarr_format=zarr_format, mode=mode, config=config, **kwargs

)

_check_no_group_config(config, store_path)

def _check_no_group_config(config: ArrayConfigLike | None, store_path: StorePath) -> None:

"""Raise if `config` is given for a node that `open` resolves to a group."""

if config is not None:

raise TypeError(

f"config is the runtime configuration of an array, but {store_path} resolves to a "

"group. Groups have no runtime configuration."

)

config: ArrayConfigLike | None = None,

return await AsyncArray.open(store_path, zarr_format=zarr_format, config=config)

config=config,

config: ArrayConfigLike | None = None,

config=config,

config: ArrayConfigLike | None = None,

config=config,