ENH: Migrate _multiarray_umath global variables to per-module state - #31992

This looks correct to me so far. Do we have module teardown tests to make sure the GC paths are hit? You can try adding a temporary debug print statement in the m_traverse, m_clear, m_free functions and then run tests to make sure they are called somewhere.

You can try adding a temporary debug print statement in the m_traverse, m_clear, m_free functions and then run tests to make sure they are called somewhere.

I Added temporary fprintf(stderr, ...) to all three GC hooks and ran gc.collect() test (and later used spin test as well) after importing NumPy. The output confirmed m_traverse is called repeatedly (as expected) but did not get an m_clear or m_free. Is this expected as they happen only after interpreter/subinterpreter shutdown?

sys.stderr.flush()
n = gc.collect()
sys.stderr.flush()

Also I am ready with the next commit locally, please let me know when can I push it!

Is the global state the only blocker for subinterpreter support or are there more nuances I am not aware of? Maybe we could add a xfailing test that adds a subinterpreter and shuts it down, I don't think it should segfault. Then in that test you could make sure it hits the code paths but still fails since we do not yet support subinterpreters.

Go ahead with the next commit as well.

If someone could please "approve" Migrate npy_static_cdata to module state, then I will push the next migration commit afterwards thanks.

Edit: Oops I had to push Migrate npy_global_state to per-module state since I did some minor ref in Migrate npy_static_cdata to module state

Does this have performance overhead? I looked over the diff and didn’t see any spots where we call the function in a tight loop so I doubt there is, but I want to make sure you’re thinking about that.

Why did you choose that benchmark to look at? Would you expect the changes in this PR to have a significant impact? If so, why? It’s not clear to me and I suspect you’re measuring noise.

I first ran spin bench -c main -t bench_core on the multiarray_module_state branch (results above) since I was not sure how much can I trust the benchmarks, that's why intentionally I chose bench_core.CountNonzero to find out how much is the noise because it showed the largest regressions (1.23x) and I also think my changes from this PR shouldn't affect CountNonzero.

Would you expect the changes in this PR to have a significant impact?

Nope, I think it should be little to none till now atleast (this is my intuition - needs reliable testing)

If so, why? It’s not clear to me and I suspect you’re measuring noise.

Yes I was measuring noise. Here are the bench_core.CountNonzero benchmarks against dummy commit I did earlier:
Even with completely identical code, the results fluctuated by ±10% on my pc.

| Change   | Before [941bc425] <main>   | After [c3e0325a] <test_benchmark_noise>   |   Ratio | Benchmark (Parameter)                                                             |
|----------|----------------------------|-------------------------------------------|---------|-----------------------------------------------------------------------------------|
| +        | 601±20μs                   | 664±30μs                                  |    1.1  | bench_core.CountNonzero.time_count_nonzero(3, 1000000, <class 'numpy.int32'>)     |
| +        | 57.6±0.3μs                 | 62.5±0.1μs                                |    1.08 | bench_core.CountNonzero.time_count_nonzero_multi_axis(3, 10000, <class 'str'>)    |
| +        | 6.00±0.09μs                | 6.39±0.3μs                                |    1.07 | bench_core.CountNonzero.time_count_nonzero_multi_axis(2, 100, <class 'object'>)   |
| -        | 56.0±2μs                   | 50.6±0.2μs                                |    0.9  | bench_core.CountNonzero.time_count_nonzero(3, 10000, <class 'str'>)               |
| -        | 305±10μs                   | 274±2μs                                   |    0.9  | bench_core.CountNonzero.time_count_nonzero_multi_axis(2, 10000, <class 'object'>) |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.

Dug into the remaining scattered globals (PyDataMem_DefaultHandler,userdescrs/NPY_NUMUSERTYPES, and the DType singleton pointers in dtypemeta.c/abstractdtypes.c). These cannot be migrated right now since all of them are entries in PyArray_API, one literal array compiled into the .so, so there's exactly one copy per process. A module state field only fixes our internal copy, external code still reads the old shared slot directly.

Two categories:

1. PyDataMem_DefaultHandler and NPY_NUMUSERTYPES are raw variable slots, external code dereferences PyArray_API[n] as the variable's own memory. Making them real per interpreter means turning them into accessor functions instead, same pattern as PyDataMem_SetHandler and GetHandler already use. That's an ABI break per numpy_api.py's own rules and needs a C API version change.

2. The DType pointers store the address of one canonical type object. Heap typing them fixes a different problem, one struct being mutated by multiple interpreters, not the table only being able to hold one address per slot. Same category as PyArray_Type.

Questions for you all:

• PyDataMem_DefaultHandler is basically read only after init. Is it fine to document as a permanent shared exception or do we want the ABI change?
• NPY_NUMUSERTYPES can't just be documented as shared, PyArray_RegisterDataType still mutates it live, so migrating userdescrs alone would desync registrations across interpreters. ABI fix, or do we document legacy user dtype registration as not subinterpreter safe?
• DType singletons, is shared type identity across interpreters acceptable? or do we want per interpreter type tables eventually?

None of this blocks what this PR already have, it only blocks the final Py_mod_multiple_interpreters flip and the TestSubinterpreterTeardown xfail. I am not sure if this is the right place to discuss this, maybe a separate issue?

I'll merge this so that @prathamhole14 can work on further limited API changes, the remaining fixups can be done in follow up PRs. Thanks all for reviews.