msgpack.decode leaks TypeError: unhashable type: 'dict' instead of DecodeError when a map or ext is used as a map key #1208

Summary

msgspec.msgpack.decode raises a bare Python TypeError instead of msgspec.DecodeError when a MessagePack map (or ext) is used as a map key. Because TypeError is not a subclass of DecodeError, a normal except msgspec.DecodeError: handler does not catch it, so a three-byte payload escapes the documented error contract.

Versions

Both reproduce the behavior:

• PyPI 0.21.1 (the released version users run today)
• main HEAD e7552f0f3341eabdbab4ecfc63c1b352ad26b7dc
• Test environment: Python 3.12, macOS arm64

Reproduction

Input bytes are given in hex:

818001        -> TypeError: unhashable type: 'dict'
878048        -> TypeError: unhashable type: 'dict'
8bd4ec5c03    -> TypeError: unhashable type: 'msgspec.msgpack.Ext'
819001        -> {(): 1}          (control: array-as-key works, because tuple is hashable)
JSON {{}:1}   -> DecodeError: JSON is malformed: object keys must be strings (byte 1)

Verbatim traceback showing the escape:

Traceback (most recent call last):
  File "/tmp/impact.py", line 8, in <module>
    consume(b'\x81\x80\x01')
  File "/tmp/impact.py", line 4, in consume
    data = msgspec.msgpack.decode(untrusted_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'dict'

Why except DecodeError does not catch it

>>> issubclass(TypeError, msgspec.DecodeError)
False

msgspec.DecodeError.__mro__
(DecodeError, MsgspecError, ValueError, Exception, BaseException, object)

TypeError.__mro__
(TypeError, Exception, BaseException, object)

Root cause

File src/msgspec/_core.c, function mpack_decode_dict. Line numbers differ between versions:

The insert, verbatim:

if (MS_UNLIKELY(PyDict_SetItem(res, key, val) < 0))
    goto error;

The handler, verbatim:

error:
    Py_LeaveRecursiveCall();
    Py_XDECREF(key);
    Py_XDECREF(val);
    Py_DECREF(res);
    return NULL;

Points to state:

• The error: label does not inspect or translate the exception — it only cleans up refs and returns NULL, so CPython's TypeError propagates raw.
• The function body contains zero occurrences of DecodeError.
• The sibling failure paths (mpack_decode_key / mpack_decode returning NULL) set a DecodeError inside the callee before returning NULL, so only the map-key hashing step leaks a foreign exception.
• The JSON path does it correctly by rejecting non-string keys before insert via json_err_invalid(self, "object keys must be strings").

Impact

Anyone decoding untrusted MessagePack — message queues, RPC, caches, sockets. A three-byte payload raises an exception the documented DecodeError contract does not cover, so idiomatic handling (try: ... except msgspec.DecodeError:) does not catch it and it crashes a worker or 500s a handler. It also breaks symmetry with the JSON API.

Duplicate search

12 keyword queries across open and closed found nothing matching. Nearest hits #199, #660, #565, #139 are all Struct/inspect related with different entry points; #241 is a closed feature request, not a bug report.

AI assistance disclosure

This issue report was prepared with the assistance of an AI tool.