BUG: avoid deadlocks using NpyString API by ngoldbaum · Pull Request #31682 · numpy/numpy

ngoldbaum · 2026-06-18T22:40:22Z

I'm working on a design for a new ByteStringDType and have come across some design issues with StringDType. In particular, it turns out that there are a number of places we're susceptible to possible deadlocks with CPython or between StringDType instances.

For NpyString_acquire_allocators and NpyString_release_allocators, the fix is to sort the mutexes by address before locking. I added tests for ufuncs that need 2, 3, and 4 allocators as well as one that uses a duplicate allocator that all deadlock on main and don't with the fixes here.

There are also a few more fixes:

StringDType setitem was holding an allocator lock while allowing arbitrary Python code to execute. I added a reentrant example in a test that deadlocks right now.
There are also issues in several casts with lock discipline and not reliably separating sections of code that hole the allocator lock or use the CPython C API.
Python 3.12 using PyThread_type_lock can deadlock with the GIL, so I added a code that checks if the GIL is held before trying to acquire the lock. This is the same deadlock-avoiding behavior as PyMutex.

There are still places that use the C API with the allocator lock held but I'm pretty sure there are no reentrancy issues in those places.

This was all done heavily with AI assistance. Please let me know if you spot anything you think is fishy because of that.

kumaraditya303 · 2026-06-19T03:42:42Z

I think we should harden the error paths as well to not call APIs which require the GIL, I think it is fine as long as there is no re-entrancy but that relies on a Python 3.12+ behavior that gc is only triggered at eval breaker points.

ngoldbaum · 2026-06-24T19:04:16Z

I think we should harden the error paths as well to not call APIs which require the GIL

I looked at this today and fixing this will require a pretty substantial second pass. I think I'd like to do it in a followup since this is already pretty big.

I asked Sebastian to look this over during the triage meeting.

seberg

Before approving it, I think I have to repeat the one question: This PR doesn't just avoid deadlocks in the API, it also changes behavior around NA objects.
I don't care too much about doing it in one PR in principle, but unlike the deadlock fix, changing that isn't an obvious backport to me.

That whole "test deadlocks" looks right. It feels overly clever/engineered, but fine, if it seems to add enough to you then OK.

seberg · 2026-06-25T11:07:21Z

    run_threaded(func, max_workers=8, pass_barrier=True)
+
+
+def assert_no_deadlock(workload, *, args=(), helpers=(), timeout=30,


Part of me does wonder if we shouldn't just use @pytest.mark.timeout(...).
Of course that doesn't:

Gracefully give a traceback (without exiting the whole test run at least, which it does on windows or with mode="thread").

Allow the STALL logic to not wait the whole timeout (I guess that may be if that is important for local debugging of complex tests)

(I mean, for all I know a few other multi-threading tests in theory also check for deadlocks :)).

I'd rather not make pytest-timeout triggering after 10 minutes so load-bearing, particularly for a test that exists solely to detect if a deadlock starts happening again. The test helper is fancy, sure, but it seems to work and it's a nice helper for future work on avoiding deadlocks.

Sorry, but I am confused. I don't care too much, I don't love the idea of killing the test suite on deadlock (I just also am not sure I love the rather complex test harness).
But there is no 10 minute delay, I was asking about a per-test timeout guard which is also 30 seconds so relatively similar to what you go there.

seberg · 2026-06-25T11:16:46Z

+    # complex inputs never map to a NaN-like na_object on cast -- a
+    # real NaN sentinel has no complex equivalent -- so NaN components are
+    # stringified rather than stored as the missing value. This matches both the
+    # historical behavior and stringdtype_setitem.


Hmmmm, I find this confusing. So we are saying that float NaN is clearly a NaN-like object and we don't care if that gets converted into a pandas.NA.

But you are arguing that complex NaN doesn't fit into a set that contains both float NaN and pandas.NA?

I guess you are just testing existing behavior, but are we fixing the float path here as a bug fix?

Let's deal with this in a subsequent PR.

seberg · 2026-06-25T11:20:48Z

+    assert arr[0] == "1.5"
+    assert arr[2] == "-2.0"
+    # the NaN element is the missing value (returned by identity), not "nan"
+    assert arr[1] is na_object


I don't want to block it if you have a good reason, but it isn't obvious to me that this is a reasonable backport, feels like it could regress someone.

Did this show up in an actual use-case?

I'll extract this into its own PR. IMO it's a bugfix and the old behavior is just wrong but you're right that it's conceivably a breaking change.

seberg · 2026-06-25T11:29:15Z

+        npy_string_allocator *next_allocator = NULL;
+        uintptr_t next_addr = UINTPTR_MAX;
+
+        for (size_t i=0; i<n_descriptors; i++) {


Suggested change

for (size_t i=0; i<n_descriptors; i++) {

for (size_t i = 0; i < n_descriptors; i++) {

(feel free to ignore)

seberg · 2026-06-25T11:43:59Z

+        PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
+        Py_DECREF(exc);
+        Py_DECREF(str);
+    }


OK, let's keep this. Part of me thinks that there must be a nicer pattern; this "set up temporaries to use after the goto" feels really strange... Maybe just in-line release + helper function, but I dunno...
(An inline release feels like it would only be nice with some C++ scoped helper that knows it has been cleaned up so it doesn't clean up twice...)

EDIT: I suppose I am thinking of a RAII helper that has an .unlock() and could even be passed on. Although, it feels like there might be an even nicer approach, but I don't see it right now...
(Right now this is a half-hearted fix in one place in a sense, although, I have to say that re-entrance seems like an unlikely problem.)

I refactored it to use RAII helpers for cleanup and it's a lot nicer. Thanks for the suggestion.

ngoldbaum · 2026-06-30T20:45:20Z

@seberg I responded to all your comments I think, could you give this another look?

seberg

Thanks for splitting and doing the release thing! LGTM, I think you like your test harness, so let's not worry about whether we absolutely need or not. It's local to our tests after all.

For NpyString_acquire_allocators and NpyString_release_allocators, the fix is to sort the mutexes by address before locking. I added tests for ufuncs that need 2, 3, and 4 allocators as well as one that uses a duplicate allocator that all deadlock on main and don't with the fixes here. Also releases the string allocator mutex before grabbing the GIL, as this in theory not necessary safe if arbitrary Python could be running (there are more paths that would need this fix, though).

BUG: avoid deadlocks using NpyString API (#31682)

For NpyString_acquire_allocators and NpyString_release_allocators, the fix is to sort the mutexes by address before locking. I added tests for ufuncs that need 2, 3, and 4 allocators as well as one that uses a duplicate allocator that all deadlock on main and don't with the fixes here. Also releases the string allocator mutex before grabbing the GIL, as this in theory not necessary safe if arbitrary Python could be running (there are more paths that would need this fix, though).

Partial backport of the numpy/testing/_private/utils.py hunk of numpy#31607, needed by the multithreaded tests added in numpy#31682. (cherry picked from commit 4cbd55f)

For NpyString_acquire_allocators and NpyString_release_allocators, the fix is to sort the mutexes by address before locking. I added tests for ufuncs that need 2, 3, and 4 allocators as well as one that uses a duplicate allocator that all deadlock on main and don't with the fixes here. Also releases the string allocator mutex before grabbing the GIL, as this in theory not necessary safe if arbitrary Python could be running (there are more paths that would need this fix, though). (cherry picked from commit 4380db7)

This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [numpy](https://github.com/numpy/numpy) ([changelog](https://numpy.org/doc/stable/release)) | dependencies | patch | `2.5.0` → `2.5.2` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/numpy/numpy/badge)](https://securityscorecards.dev/viewer/?uri=github.com/numpy/numpy) | | [uvicorn](https://github.com/Kludex/uvicorn) ([changelog](https://uvicorn.dev/release-notes)) | dependencies | patch | `0.52.1` → `0.52.3` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Kludex/uvicorn/badge)](https://securityscorecards.dev/viewer/?uri=github.com/Kludex/uvicorn) | --- ### Release Notes <details> <summary>numpy/numpy (numpy)</summary> ### [`v2.5.2`](https://github.com/numpy/numpy/releases/tag/v2.5.2): (Aug 9, 2026) [Compare Source](numpy/numpy@v2.5.1...v2.5.2) ### NumPy 2.5.2 Release Notes The NumPy 2.5.2 is a patch release that fixes bugs discovered after the 2.5.1 release. The big news is that it includes wheels for the newly released Python 3.15.0rc1. This release supports Python versions 3.12-3.15 #### C API changes ##### `PyArray_StringDTypeObject` is opaque under the abi3t stable ABI The `PyArray_StringDTypeObject` was accidentally exposed in NumPy 2.5 when targeting the free-threading-compatible stable ABI (`Py_TARGET_ABI3T`). `PyArray_StringDTypeObject` is now an opaque struct: extensions compiled that way cannot access its fields, since the struct layout depends on the size of the object header. Any code that accessed `PyArray_StringDTypeObject` fields in an abi3t build would have crashed, so we are making this API change in a bugfix release. The `NpyString` allocator API remains usable by passing the descriptor object pointer, e.g. `NpyString_acquire_allocator((PyArray_StringDTypeObject *)descr)`. ([gh-31771](numpy/numpy#31771)) #### Contributors A total of 16 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Abhijeetsingh Meena + - Charalampos Stratakis - Charles Harris - Chris Ninham + - David Woods - Geonho + - Gopu Yeshwanth Reddy + - Iason Krommydas - Ijtihed Kilani - Jelle Zijlstra + - Joren Hammudoglu - Kumar Aditya - Mike Boyle - Nathan Goldbaum - Raghuveer Devulapalli - Sebastian Berg #### Pull requests merged A total of 28 pull requests were merged for this release. - [#31864](numpy/numpy#31864): MAINT: Prepare 2.5.x for further development - [#31889](numpy/numpy#31889): TYP: Backport multiple static typing fixes 1. - [#31900](numpy/numpy#31900): TST: add tests for stable ABI numpy extensions ([#31822](numpy/numpy#31822)) - [#31901](numpy/numpy#31901): BUG: fix `StringDType` coerce flag in binary ufunc promotion... - [#31902](numpy/numpy#31902): BLD: fix meson deprecation warnings ([#31892](numpy/numpy#31892)) - [#31921](numpy/numpy#31921): TYP: Backport multiple typing fixes 2. - [#31947](numpy/numpy#31947): MAINT: Update x86-simd-sort subproject ([`5adb334`](numpy/numpy@5adb334) → [`fa944ef`](numpy/numpy@fa944ef)) ([#31908](numpy/numpy#31908)) - [#31949](numpy/numpy#31949): BUG: fix crash on 32 bit systems using abi3t ([#31771](numpy/numpy#31771)) - [#31950](numpy/numpy#31950): MNT: remove some obsolete string to bool workarounds ([#31859](numpy/numpy#31859)) - [#31952](numpy/numpy#31952): BUG: centralized helper for output coerce and na\_object in stringdtype... - [#31953](numpy/numpy#31953): BUG: fix CPU feature env diagnostic buffer overruns ([#31905](numpy/numpy#31905)) - [#31954](numpy/numpy#31954): BUG: restore ndarray.conjugate() for legacy user-defined dtypes... - [#31955](numpy/numpy#31955): TYP: Avoid shadowed dtype annotations - [#32077](numpy/numpy#32077): MAINT: Update verdored-meson/meson to match main. - [#32114](numpy/numpy#32114): BUG: fix refcount leak on overlapping copyto with where=False - [#32115](numpy/numpy#32115): BUG: fix swallowed cast error in fancy indexing assignment ([#31975](numpy/numpy#31975)) - [#32116](numpy/numpy#32116): BUG: Fix buffered iterator stride after removing multi-index - [#32117](numpy/numpy#32117): BUG: fix `np.fromiter` corruption when reusing a `StringDType`... - [#32119](numpy/numpy#32119): BUG: add a special case for StringDType in np.isdtype ([#32030](numpy/numpy#32030)) - [#32121](numpy/numpy#32121): BUG: reference leak in `simd_sequence_from_iterable` ([#32038](numpy/numpy#32038)) - [#32122](numpy/numpy#32122): BUG: ensure lock is held when accessing or writing to RNG state... - [#32123](numpy/numpy#32123): BUG: fully reset cached RNG state for non-MT19937 RNGs ([#32062](numpy/numpy#32062)) - [#32135](numpy/numpy#32135): TYP: type capabilities max dimensions - [#32158](numpy/numpy#32158): BUG: avoid possible stack overflow in arraydescr\_dealloc ([#32133](numpy/numpy#32133)) - [#32206](numpy/numpy#32206): MAINT: Update cibuildwheel to v4.2.0 - [#32214](numpy/numpy#32214): MAINT: Skip limited\_api tests on some platforms. - [#32220](numpy/numpy#32220): TYP: `isclose` shape-typing fix for 2d array-likes ([#32205](numpy/numpy#32205)) - [#32221](numpy/numpy#32221): BUG: avoid segfaults when legacy copyswap slot is not defined... ### [`v2.5.1`](https://github.com/numpy/numpy/releases/tag/v2.5.1): (July 4, 2026) [Compare Source](numpy/numpy@v2.5.0...v2.5.1) ### NumPy 2.5.1 Release Notes The NumPy 2.5.1 is a patch release that fixes bugs discovered after the 2.5.0 release. The most noticeable is the fix is to the numpy datetime cython API which should allow downstream to support NumPy versions older than 2.5. Preparation for Python 3.15 continues along with typing improvements. This release supports Python versions 3.12-3.14 #### Changes - The minimum supported GCC version has been updated from 9.3.0 to 10.3.0 ([gh-31843](numpy/numpy#31843)) #### Contributors A total of 10 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Adhyan Gupta + - Ankit Ahlawat - Charles Harris - Iason Krommydas - Joren Hammudoglu - Kumar Aditya - Nathan Goldbaum - Sebastian Berg - Ties Jan Hefting + - Vineet Kumar #### Pull requests merged A total of 20 pull requests were merged for this release. - [#31707](numpy/numpy#31707): MAINT: Prepare 2.5.x for further development - [#31721](numpy/numpy#31721): CI: fix new `cython-lint` errors ([#31711](numpy/numpy#31711)) - [#31723](numpy/numpy#31723): MAINT: Update meson to match main - [#31729](numpy/numpy#31729): TST: use setup-sde instead of curl to get SDE binaries ([#31727](numpy/numpy#31727)) - [#31829](numpy/numpy#31829): BUG: Relax finfo to be easier accessible for all user dtypes... - [#31831](numpy/numpy#31831): TYP: Fix `flatiter.__next__` return type for `object_` and... - [#31832](numpy/numpy#31832): BUG: avoid deadlocks using NpyString API ([#31682](numpy/numpy#31682)) - [#31833](numpy/numpy#31833): BUG: fix out array leak in reduceat and accumulate when dtype... - [#31835](numpy/numpy#31835): BUG: fix numpy datetime cython APIs to be compatible with older... - [#31836](numpy/numpy#31836): TYP: Fix incorrect dtype inference of `asarray([])` ([#31732](numpy/numpy#31732)) - [#31837](numpy/numpy#31837): TYP: Fix `np.ma.masked_array` 2.5.0 regression - [#31838](numpy/numpy#31838): FIX: Refactor error handling in array\_setstate to prevent typecode... - [#31839](numpy/numpy#31839): TST: xfail multithreaded BLAS test more generously - [#31840](numpy/numpy#31840): MAINT: Rename subroutine for crackfortran tests - [#31842](numpy/numpy#31842): BUG: fix leak in reductions when a ufunc override errors or is... - [#31849](numpy/numpy#31849): BLD: set minimum required gcc version to 10.3 ([#31843](numpy/numpy#31843)) - [#31855](numpy/numpy#31855): CI: fix hangs on MacOS ASan CI ([#31853](numpy/numpy#31853)) - [#31856](numpy/numpy#31856): BUG: fix several bugs in StringDType operations ([#31846](numpy/numpy#31846)) - [#31857](numpy/numpy#31857): BUG: Fix segfault in MT19937 by preventing recursive seed lists... - [#31858](numpy/numpy#31858): BUG: Fix signed integer overflow in datetime.c ([#31688](numpy/numpy#31688)) </details> <details> <summary>Kludex/uvicorn (uvicorn)</summary> ### [`v0.52.3`](https://github.com/Kludex/uvicorn/releases/tag/0.52.3): Version 0.52.3 [Compare Source](Kludex/uvicorn@0.52.2...0.52.3) ##### Changed - Update `zttp` to 0.0.24 and use its combined receive path, improving HTTP/1.1 request parsing performance ([#3067](Kludex/uvicorn#3067)) **Full Changelog**: <Kludex/uvicorn@0.52.2...0.52.3> ### [`v0.52.2`](https://github.com/Kludex/uvicorn/releases/tag/0.52.2): Version 0.52.2 [Compare Source](Kludex/uvicorn@0.52.1...0.52.2) ##### Fixed - Update `zttp` to 0.0.22, fixing bodyless request receives and improving HTTP/1 request parsing performance ([#3063](Kludex/uvicorn#3063)) **Full Changelog**: <Kludex/uvicorn@0.52.1...0.52.2> </details> --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  See merge request swiss-armed-forces/cyber-command/cea/loom!639 Co-authored-by: Loom MR Pipeline Trigger <group_103951964_bot_9504bb8dead6d4e406ad817a607f24be@noreply.gitlab.com>

ngoldbaum added 09 - Backport-Candidate PRs tagged should be backported component: numpy.strings String dtypes and functions 00 - Bug labels Jun 18, 2026

ngoldbaum added this to the 2.5.1 release milestone Jun 18, 2026

BUG: avoid deadlocks using NpyString API

fa76f0b

ngoldbaum force-pushed the lock-ordering-fixes branch from b261063 to fa76f0b Compare June 19, 2026 02:36

DOC: tweak docs for allocator acquire functions

e5f75c2

kumaraditya303 reviewed Jun 19, 2026

View reviewed changes

Comment thread numpy/_core/tests/test_multithreading.py Resolved

ngoldbaum added 2 commits June 18, 2026 21:38

More tests and correctness fixes

69aad6b

MNT: apply Kumar's suggestions

41e7e78

kumaraditya303 reviewed Jun 19, 2026

View reviewed changes

Comment thread numpy/_core/src/multiarray/stringdtype/casts.cpp Resolved

seberg reviewed Jun 25, 2026

View reviewed changes

ngoldbaum added 3 commits June 30, 2026 14:15

Merge branch 'main' into lock-ordering-fixes

4d945b4

MNT: remove changes that might lead to observable behavior changes

7fdb148

MNT: apply Sebastian's suggestions

93dd3fa

seberg approved these changes Jul 1, 2026

View reviewed changes

seberg merged commit 4380db7 into numpy:main Jul 1, 2026
87 checks passed

ngoldbaum mentioned this pull request Jul 1, 2026

BUG/ENH: fix casting and setitem nan behavior between StringDType and float16/32 #31825

Merged

charris mentioned this pull request Jul 2, 2026

BUG: avoid deadlocks using NpyString API (#31682) #31832

Merged

charris removed the 09 - Backport-Candidate PRs tagged should be backported label Jul 2, 2026

charris added a commit that referenced this pull request Jul 2, 2026

Merge pull request #31832 from charris/backport-31682

8e053bd

BUG: avoid deadlocks using NpyString API (#31682)

charris added 09 - Backport-Candidate PRs tagged should be backported and removed 09 - Backport-Candidate PRs tagged should be backported labels Jul 10, 2026

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

BUG: avoid deadlocks using NpyString API (numpy#31682)

84ea7d0

		run_threaded(func, max_workers=8, pass_barrier=True)


		def assert_no_deadlock(workload, *, args=(), helpers=(), timeout=30,

	for (size_t i=0; i<n_descriptors; i++) {
	for (size_t i = 0; i < n_descriptors; i++) {

Conversation

ngoldbaum commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kumaraditya303 commented Jun 19, 2026

Uh oh!

Uh oh!

ngoldbaum commented Jun 24, 2026

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seberg Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Jun 30, 2026

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ngoldbaum commented Jun 18, 2026 •

edited

Loading

seberg Jun 25, 2026 •

edited

Loading