Repository navigation
PyPy compatibility #22
Description
This was raised on twitter, and may be important for some users. Currently msgspec is written as a c extension using some private cpython apis, making it incompatible with pypy. It would be good to have a pypy compatible build, whether through changes to the c extension or through a separate pure python release. Either seems fine to me, provided the result doesn't decrease cpython performance noticeably.
Activity
jcrist commentedon Feb 25, 2021
I personally don't intend to work on this anytime soon, just flagging it so it doesn't get lost. PRs welcome.
jcrist commentedon Jul 13, 2021
This project has grown in scope enough that a separate pypy implementation would be more trouble than it's worth IMO. Closing.
provinzkraut commentedon Dec 13, 2022
Hello there, just wanted to ask if you'd consider revisiting this? I wanted to evaluate Starlite's PyPy compatibility and msgspec is unfortunately a blocker there. Seeing as the project has evolved even more since the original decision, I don't suspect achieving compatibility has become any easier though..
A re-implementation in pure Python would be a really big task. Do you see any other way compatibility could be achieved?
jcrist commentedon Dec 13, 2022
This is not something I plan on supporting in a volunteer-only basis (if someone wanted to pay for this, I might be willing to expand support).
A few reasons:
msgspecis heavily tied to the CPython c-api for performance. PyPy does support a subset of the CPython c-api, but it does so in a way that is not very performant, and isn't 100% compatible. We could add shims everywhere that's needed to makemsgspecPyPy compatible, but that would muddy up the code, increasing our maintenance burden.Goldziher commentedon Dec 13, 2022
Thanks for clarifying this. I think we are fine not having pypy support.
v3ss0n commentedon Jun 17, 2023
Since pypy 3.-10 (7.3.12) released , and CPyext performance and compatibility was being improved in past recent year , this should be able to fixed. What are the missing APIs ? I had contacted pypy-dev mailing list and linked this. They should be able to help.
if this can be solved , we can support that in litestar.
mattip commentedon Jun 17, 2023
@v3ss0n I don't think the reasoning in the comment above has significantly changed even for the last PyPy release. As seen in #443, msgspec wants to use CPython private internals like
((PyListObject *)x)->allocated. There is little chance this will ever be supported on PyPy (I say this as the person who has implemented much of the PyPy C-API interoperability). Even if it was supported, it would not be as fast as CPython. In order to properly support PyPy, the_coremodule (from the 10,000 line_core.cfile would need refactoring.v3ss0n commentedon Jun 18, 2023
Thanks a lot for clarification . for this case then , a lot need to be rewritten for both pypy , and msgspec . What would be the best course of action for developing libraries like this ? On initial development - use HPy ?
mattip commentedon Jun 18, 2023
As @Goldziher stated above
so one path forward would be to put together a proposal to fund a fork or enhancement of msgspec to support PyPy.
Goldziher commentedon Jun 18, 2023
Wasn't me 😀
mattip commentedon Jun 18, 2023
Sorry, that should have been "as @jcrist stated above ..."
paugier commentedon Jun 22, 2023
@jcrist I'm surprised that this argument is used. CPython might get a bit faster in few years but there is a very high probability that PyPy will stay much faster than CPython for most applications.
It's a bit difficult to build one number to quantify the global efficiency of Python interpreters but CPython 3.11 is something like 1.25 times faster than CPython 3.10 and it seems that CPython 3.12 will be something like 1.3 faster than CPython 3.10 (https://github.com/faster-cpython/benchmarking-public). In contrast, PyPy is more than 4 times faster than CPython (https://speed.pypy.org/). It's not easy to compare these numbers but still PyPy is much faster than CPython.
Moreover knowing the different technologies used by CPython and PyPy and the constrains in terms of compatibility on CPython, there won't be miracle and unfortunately CPython will stay quite inefficient for a very long time.
v3ss0n commentedon Jun 22, 2023
Don't install `msgspec` on PyPy
Don't install `msgspec` on PyPy (#507)
pepijndevos commentedon Feb 14, 2026
Could it be reasonable to have a pure python fallback? That would unlock not just pypy but also eg jython, micropython, and cpython installations where compiling C is out of the question.
In an era of LLMs churning out such a python fallback might not be that hard and there seems to be a demand but of course it is an extra maintenance burden
Fix `Meta(pattern=..)` memory leak / corruption (msgspec#1256)