Repository navigation
Description
Myself, @lysnikolaou, and @rgommers are working on a new project in collaboration with the Python Runtime team at Meta to add support for Python 3.13 nogil builds in the scientific python software stack.
Our initial minimal goal is a build of NumPy that runs and passes all the tests in a nogil build of Python.
Resources:
-
Add changelog entry and docs describing the current state of using NumPy in a multithreaded environment.
-
Convert
PyDict_GetItemandPyList_GetItemto use variants returning strong references (MNT: Add linter for thread-unsafe C API uses #26159)- Vendor pythoncapi-compat in a submodule (MNT: Adopt pythoncapi-compat? #26158)
-
Set up CPython 3.13 CI
- 3.13 gil enabled build (TST: run the smoke tests on more python versions #26284)
-
--disable-gilbuild (TST: add basic free-threaded CI testing #26463)
-
Make the test suite pass with the gil disabled
-
Mark NumPy C extensions with
Py_MOD_GIL_NOT_USED. -
Allow building f2py extensions that can be imported without the GIL.
-
Use PyMutex instead of PyThread_type_lock
-
Nightly wheels for
cp313tABI -
Global state
- C static variables in
_multiarray_umathreorganized into structs: MNT: Reorganize non-constant global statics into structs #26607 - Global caches need to be disabled, have locking, serialize creating read-only caches, or converted to thread-local caches.
- Disabled:
- Locking:
- Cache initialization moved to module initialization
- Some runtime caches need a single-init API. MAINT: use an atomic load/store and a mutex to initialize the argparse and runtime import caches #26780
- Dragon4 scientific printing code uses a static global scratch space.
- Triaged low priority:
wrapping_array_method.cuses a freelist that is certainly not thread safe. It has no internal consumers besides_scaled_float_dtypeto test the implementation so ignoring it for now.- the number of legacy user dtypes numpy knows about is held in a global static variable:
NPY_NUMUSERTYPES, which also unfortunately is part of the public C API. This one may just need to remain unfixed, given it's in a legacy API. - The check to force an error on module reloading isn't thread safe. Probably needs an atomic to guard initializing it.
- Global settings and state may need to be made per-thread or synchronized across threads.
- Converted to thread-local state:
np.set_string_functionandPyArray_SetStringFunctionare implemented using static global state.- Make printoptions a context variable. MAINT: back printoptions with a true context variable #26846
- hugepage support is controlled by a single static global. The only public interface to control this is an environment variable read at startup, so this isn't problematic unless someone is using private APIs.
NUMPY_WARN_IF_NO_MEMORY_POLICYis implemented using a global static. The only public interface is via an environment variable read at module initialization but it's tested using a thread-unsafe interface.
- C static variables in
Activity
ogrisel commentedon Apr 9, 2024
Maybe another interesting resource is the past work done by Sam Gross to patch numpy to make it work well enough with the original
nogilfork of CPython 3.9: v1.24.0...colesbury:numpy:v1.24.0-nogilThings might have drifted a bit since that time, but those patches are enough to get the scikit-learn test suite pass with
nogil-CPython 3.9: see e.g. this run of theLinux_nogil pylatest_pip_nogiljob that is scheduled run on a nightly basis on our Azure Pipelines CI.ngoldbaum commentedon Apr 16, 2024
It looks like numpy's main branch passes the full numpy test suite with the gil disabled. I've been testing using the
nogil-integrationbranch in @colesbury's fork of CPython, but i tried just now using the CPythonmainbranch and the test suite passes with the GIL disabled there as well.There are still a ton of threading bugs I'm sure because of limited use of threads in the tests - we need better test coverage for threaded workflows. I'm already aware of a number of internal caches we'll likely need to disable or add locking for.
But, it's a good start!
a-reich commentedon Apr 26, 2024
Just curious, are contributors planning to work on #24755 or is there any overlap where work for nogil would also help with allowing subinterpreters?
ngoldbaum commentedon Apr 26, 2024
While this does have some overlap with work on subinterpreters since the C thread safety issues we're working on apply to subinterpreters as well, the bulk of the work required to support subinterpreters - converting numpy's extension types from single-phase to multi-phase initialization - is not required and we are not currently planning to do that.
[-]ENH: Support nogil python build (tracking issue)[/-][+]ENH: Support free-threaded python build (tracking issue)[/+]ngoldbaum commentedon May 28, 2024
There are now
cp313tnightly wheels up formanylinuxandmusllinuxat our normal spot for nightlies. Windows wheels will need to wait on resolving some windows-specific build issues and Mac wheels will need to wait for 3.13.0b2, which will have mac binaries cibuildwheel can use.To install the wheel on a free-threaded build of python you'll need the prerelease of pip 24.1, installable with e.g.
pip install -U --pre pip. The current stable release doesn't know about the new ABI tag and will install the wheel for thecp313ABI which will then fail to import at runtime because of the ABI mismatch.rgommers commentedon Jun 5, 2024
I think we should do that, at least for now. It may not be tenable in the very long term, but for the coming 1-2 years at least, everyone who is building for free-threaded CPython should know how to install a BLAS library.
There's no dedicated method for "is this a free-threaded build" yet in Meson. Something like this should do it though I think (untested):
rgommers commentedon Jun 5, 2024
As for global state, it'd be good to check
numpy.random. There are nice parallel features for users that know what they are doing (see docs), but the worry is more about single-threaded-like usage:Generatorinstance, create threads, then call methods from multiple threads when there is no GIL. There is pretty careful lock usage inrandom/_generator.pyxand calls that advance the bitgenerator state look atomic, but how sure are we that that's actually the case?np.random.*functions andRandomState) is implemented in the same fashion and so uses locks as well, but probably there are some separate concerns (e.g., callingseed()from one thread while generating numbers in another).@rkern do you happen to know off the top of your head what the state is?
rkern commentedon Jun 5, 2024
Not sure. GILlessness didn't exist at the time of the design, so it surely didn't enter our mental model when thinking about it. We did want to support
with nogil:blocks that just call the C functions that draw data from and advance theBitGeneratorstate, so that is where the careful lock usage comes from. I don't know if that's going to be sufficient for other removals of the GIL where we were expecting there to be a GIL in place in the past.rkern commentedon Jun 5, 2024
And I haven't gotten my head around free-threaded Python yet to have a mental model to reason about it at this time.
ngoldbaum commentedon Jun 5, 2024
I'll try to take a look with an eye toward thread safety. I'm not very familiar with the RNG internals so it'll be a good excuse to poke around. I might ping the mailing list or open an issue if I have questions to give @rkern a chance to help clarify things :)
ngoldbaum commentedon Jun 12, 2024
I looked at the
np.randomcython code today and I think we're fine without any new major updates.All existing usages of the lowlevel bitgen state are guarded by acquiring a lock. We can probably make all of this faster using
PyThread_type_lockdirectly or when it's available, usingPyMutex.The
SeedSequencestate is generated inside__init__, so no need to lock there because__init__will only ever be called on a single thread for a single instance. Any data races are also present with the GIL and require doing silly things like spawning child seed sequences from the same parent sequence in multiple threads.7 remaining items
ngoldbaum commentedon Jul 12, 2024
The Cython PR just got merged so I'll send in the necessary Cython build changes as well whenever the Cython wheel gets regenerated.
rgommers commentedon Jul 12, 2024
In case it helps: you cannot add the flag unconditionally and whether the interpreter is a free-threaded one cannot be (easily) detected in
meson.build, so the way to do this is to add the flag unconditionally for Cython>=3.1.0a0. There's already a Cython version check in the top-levelmeson.buildfile.lysnikolaou commentedon Jul 13, 2024
The flag is okay to be added even under a normal (non-free-threaded) build, where it'll basically do nothing. So, yes, the way to go is add the flag unconditionally for the Cython version that supports it.
ngoldbaum commentedon Jul 17, 2024
I chatted with @colesbury about locking for object arrays today. He thinks we will need some sort of locking or synchronization to fix the threaded object array access issues @seberg is worried about.
Rather than adding a new lock, he suggested using the critical section API on the array object before and after spots we release or acquire the GIL for dtypes that don't need the GIL.
There are quite a few places we use the macros that release and acquire the GIL, but I also think it should be relatively straightforward to grep for those macros and find spots where we have a special check for object arrays so that we don't release the GIL in that case.
In those spots we'd need to use the critical section API. We may also be able to expand usage of
NPY_BEGIN_THREADS_DESCRNPY_END_THREADS_DESCRto handle this without adding a bunch of new boilerplate. The one sticking point that comes to mind is the need to have a reference to the array object, and theArrayMethodAPI intentionally does not expose access to the array object. I'm not sure yet if that's a problem.However all that said, I think given the release timing we might just need to document this limitation for the 2.1 release as a known issue. As far as I know this hasn't caused any crashes with pandas or scikit-learn, and they heavily use object string arrays.
albanD commentedon Jul 17, 2024
seberg commentedon Jul 19, 2024
You would have to walk the base anyway, which isn't super nice, of course you could add an owner field to the additional information that is passed in via the struct, though, if that ends up as the best solution.
I still think the StringDType design is sound to do this via the dtype instance, but without a side-car buffer I dunno if an explicit owner somewhere else might end up easier/cleaner.
There are still some issues and I don't know if they even can be solved fully currently:
basewhich is a bit annoying currently (the dtype instance would solve this a bit).Unless there is a fallback like
Py_BEGIN_CRITICAL_SECTION(NULL)to say that the owning/critical object is unknown?! (Practically re-enabling the GIL, though, so it seems mainly a good fallback for memory sharing.).EwoutH commentedon Aug 1, 2024
3.13 rc1 is released! This means the ABI is stable and wheels can be uploaded to PyPI.
Tracking for cibuildwheel: pypa/cibuildwheel#1949
ngoldbaum commentedon Aug 1, 2024
We're planning to upload
cp313twheels to pypi for Mac and Linux for the NumPy 2.1 release candidate.Hopefully we'll be able to get Windows wheels up too but that's waiting on resolving some tricky merge conflicts in our meson patches, which will also hopefully help us to upstream the meson patches themselves.
EwoutH commentedon Aug 3, 2024
cibuildwheel now uses the latest Python
3.13.0rc1and builds Python 3.13 by default (pypa/cibuildwheel#1950). Next cibuildwheel release should be soon.Free-threaded wheels are still behind a flag (CIBW_FREE_THREADED_SUPPORT), but can also be build and uploaded using the ABI stable Python 3.13.0rc1.
karlotness commentedon Aug 9, 2024
I may have missed discussion of this elsewhere, but I think there are also thread safety issues with in-place updates to ndarray
shape,stridesanddtypeattributes. Setting these with an array shared between threads can cause inconsistent views of the array layout/size.Setting those attributes is documented as a bad idea, but with the GIL I'm not sure it could cause crashes (I might be wrong on that).
As an example, on my machine with recent NumPy nightly and Python 3.13.0rc1 the below script segfaults without the GIL and loops until killed with it.
Example Script
ngoldbaum commentedon Aug 9, 2024
Yup, right now the suggestion is: don't do that.
NumPy isn't thread safe in Python programs and has never been thread safe. Free-threading makes this more acute but it's a pre-existing problem.
My plan for NumPy 2.1 is to document that mutating shared state in multithreaded python code should be done with extreme care and may cause crashes if there are races. This is also true with the GIL. It's certainly easier to cause a crash in the free-threaded build but the same issues about shared mutable state happen if any threads release the GIL and touch shared state.
There are also probably some safety issues around object arrays that are only present in the GIL-disabled build.
We're going to need to think carefully about adding improved locking around ndarray itself. It'll be a big project.
karlotness commentedon Aug 9, 2024
That makes sense, thanks! I figured that was probably already on the radar somewhere.
The thing that prompted this was reviewing some interactions I have with the NumPy C API where currently I make sure to load array shapes, strides, etc. before releasing the GIL. I mostly just wanted to be sure there isn't currently anything more to be done to protect those. I'll definitely check any updated guidance in the future.
Thanks for the work so far, the experimental build works nicely in the little bit of testing that I've done.
rgommers commentedon Oct 15, 2024
If anyone was waiting for Windows wheels: there are nightlies at https://anaconda.org/scientific-python-nightly-wheels/numpy/files now. The next release (2.1.3) will include them on PyPI.
ngoldbaum commentedon Oct 24, 2024
NumPy 2.1 included initial support for free-threading, so I'm closing this issue (a bit belatedly).
The main followup is #27199, which I'm hoping to have time to work on soon.