Repository navigation
memoryview in msgpack decoder ext_hook can access memory outside its bounds #835
Description
Description
The memoryview object passed to the ext_hook function from a msgspec.msgpack.Decoder object seems able to access memory outside what it's supposed to be able to access.
Here's a reproducible test case:
from typing import Any
from msgspec.msgpack import Decoder
def ext_hook(code: int, data: memoryview) -> Any:
match code:
case 0x40:
print(bytes(data))
print(bytes(data[4:]))
return None
case _:
raise NotImplementedError(f'extension type code {code} is not supported')
decoder = Decoder(ext_hook=ext_hook).decode(b'\xc7\x06@\x07\xe9\x04\x16\t\x1e')When I run this, it prints this:
b'\x07\xe9\x04\x16\t\x1e'
b'\t\x1e\x00co'
The second line should be b'\t\x1e, but it shows 3 extra bytes that seem to be coming from somewhere else. This is not normal behavior in CPython. I can create my own memoryview with the same content, and it behaves as expected:
>>> b = b'\x07\xe9\x04\x16\t\x1e'
>>> m = memoryview(b)
>>> bytes(m)
b'\x07\xe9\x04\x16\t\x1e'
>>> bytes(m[4:])
b'\t\x1e'I've reproduced this on macOS 15.3.1 and Debian 12 with CPython 3.13.3 and msgspec 0.19.0.
Activity
jcrist commentedon Apr 22, 2025
Thanks for opening this! I can confirm the bug.
Py_bufferobjects have both alenandshapeattribute. Previously when exporting a buffer we were only updating thelen, while we also want to update theshape. The reason this went unnoticed is that many consumers only check thelenand not theshape(since they only work with 1d data anyway). This is why you seebytes(data)do the expected thing, since it's only checkinglen, while slicing a memoryview works withshape. For now as a workaround you could work with a copy ofdatalike:which while inefficient will properly set the bounds.
I'll push up a fix later to resolve this for good.