Summary
The numcodecs.pickles.Pickle codec decodes data by calling pickle.loads on
the input buffer. Deserializing untrusted pickle data can execute arbitrary code
during decoding. Because the Pickle codec is registered by default and is
selected from the codec id stored in array metadata, a maliciously crafted
array can cause arbitrary code execution in the reading process the moment a
chunk is read — the victim never has to choose the codec or call pickle
themselves.
Details
numcodecs codecs are looked up by a string id. get_codec({"id": "pickle"})
resolves to the Pickle class, which is registered at import time. Array
containers such as Zarr v2 store the codec id and its configuration in the
array's metadata (.zarray compressor / filters fields). When a chunk is
read, the library instantiates the codec named in that metadata and calls its
decode method on the chunk bytes.
For the Pickle codec, decode runs pickle.loads on those bytes. The Python
pickle format is not a safe data format: a pickle stream can instruct the
interpreter to import modules and invoke callables during loading (see the
security warning in the standard library pickle documentation). The vulnerable
operation is the decode path, numcodecs.pickles.Pickle.decode →
pickle.loads(buf). encode is not affected.
The codec id that selects Pickle, and the chunk bytes fed to it, both come from
the array being read. Selecting the dangerous codec is therefore entirely under
the attacker's control via the stored metadata; no call to pickle or to the
Pickle codec appears in the victim's own code.
Proof of concept
The following demonstrates that decoding attacker-controlled bytes invokes a
callable chosen by those bytes. It uses a harmless, observable side effect (a
module-level flag) in place of any real payload; a __reduce__ method can name
any importable callable, which is the root of the issue.
import pickle
import numcodecs
executed = {"ran": False}
def _mark():
# stands in for an attacker-chosen callable
executed["ran"] = True
class Evil:
def __reduce__(self):
return (_mark, ())
# Attacker produces bytes; victim's library decodes them.
malicious_bytes = pickle.dumps(Evil())
numcodecs.Pickle().decode(malicious_bytes)
assert executed["ran"] is True # the callable ran during decode
In a realistic attack the victim never calls Pickle directly. Instead the
codec id travels in array metadata (e.g. a Zarr v2 .zarray compressor or
filters entry set to {"id": "pickle"}), and the malicious bytes are stored as
an ordinary chunk. Reading any such chunk resolves the pickle id via
get_codec and runs Pickle.decode on the attacker's bytes, so simply opening
and reading the array triggers execution.
Impact
An attacker who can influence the bytes or metadata that a victim reads — for
example by publishing a dataset, sharing a file, or serving array data over a
network — can achieve arbitrary code execution in the victim's process, with the
victim's privileges. The victim only needs to read the array with a stack that
uses numcodecs for decoding (for example Zarr v2).
This affects numcodecs installed from any source, including conda-forge; the
"Affected products" versions above use the pip ecosystem because that is what
GitHub records, but the same version ranges apply to the conda-forge package.
Summary
The
numcodecs.pickles.Picklecodec decodes data by callingpickle.loadsonthe input buffer. Deserializing untrusted pickle data can execute arbitrary code
during decoding. Because the
Picklecodec is registered by default and isselected from the codec id stored in array metadata, a maliciously crafted
array can cause arbitrary code execution in the reading process the moment a
chunk is read — the victim never has to choose the codec or call
picklethemselves.
Details
numcodecscodecs are looked up by a string id.get_codec({"id": "pickle"})resolves to the
Pickleclass, which is registered at import time. Arraycontainers such as Zarr v2 store the codec id and its configuration in the
array's metadata (
.zarraycompressor/filtersfields). When a chunk isread, the library instantiates the codec named in that metadata and calls its
decodemethod on the chunk bytes.For the
Picklecodec,decoderunspickle.loadson those bytes. The Pythonpickleformat is not a safe data format: a pickle stream can instruct theinterpreter to import modules and invoke callables during loading (see the
security warning in the standard library
pickledocumentation). The vulnerableoperation is the decode path,
numcodecs.pickles.Pickle.decode→pickle.loads(buf).encodeis not affected.The codec id that selects
Pickle, and the chunk bytes fed to it, both come fromthe array being read. Selecting the dangerous codec is therefore entirely under
the attacker's control via the stored metadata; no call to
pickleor to thePicklecodec appears in the victim's own code.Proof of concept
The following demonstrates that decoding attacker-controlled bytes invokes a
callable chosen by those bytes. It uses a harmless, observable side effect (a
module-level flag) in place of any real payload; a
__reduce__method can nameany importable callable, which is the root of the issue.
In a realistic attack the victim never calls
Pickledirectly. Instead thecodec id travels in array metadata (e.g. a Zarr v2
.zarraycompressororfiltersentry set to{"id": "pickle"}), and the malicious bytes are stored asan ordinary chunk. Reading any such chunk resolves the
pickleid viaget_codecand runsPickle.decodeon the attacker's bytes, so simply openingand reading the array triggers execution.
Impact
An attacker who can influence the bytes or metadata that a victim reads — for
example by publishing a dataset, sharing a file, or serving array data over a
network — can achieve arbitrary code execution in the victim's process, with the
victim's privileges. The victim only needs to read the array with a stack that
uses
numcodecsfor decoding (for example Zarr v2).This affects
numcodecsinstalled from any source, including conda-forge; the"Affected products" versions above use the pip ecosystem because that is what
GitHub records, but the same version ranges apply to the conda-forge package.