You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an issue to discuss the future of the npymath library in Numpy.
Numpy ships with a static library in core/lib named npymath.lib (Windows) or libnpymath.a (Unices).
It provides a suitable platform-specific implementation of various math routines.
Scipy links to npymath during its build process, in various places.
We (Scipy) have run into problems linking against Npymath, because of the combination of Numpy's use of the latest Visual Studio toolchain, and Scipy's use of the Mingw-w64 toolchain - discussed in MacPython/numpy-wheels#145 .
The issue that arises is that it is in general more difficult to link static libraries across toolchains, than it is to link against dynamic libraries (e.g. DLLs) or to recompile the sources. This issue is to discuss whether we should think of other solutions. Possible options are:
Leave things as they are.
Ship the C header and sources for Npymath, so external libraries and packages can recompile them.
Create a dynamic library from Npymath, and ship that, for linkage against external libraries.
Your option here.
This issue is to house the discussion of different options.
Note that whether or not this is a priority is kinda coupled with the vc141 vs. newer MS compiler toolchain discussion in MacPython/numpy-wheels#145. For vc141 it seems we can work with libnpymath.a in a Mingw-w64 based SciPy build, which with vc142 things just seem broken and likely hard/fragile to repair.
We could do something proactively, but the urgency is a lot lower as long as there's no need to upgrade from vc141.
leave npymath.lib and npyrandom.lib as they are for numpy.
Numpy could create two additional shared binaries npymath.dll and npyrandom.dll from these static libraries.
during the Numpy installation, place these two DLLs in a common location where Scipy or other third-party packages can locate them both during the build process and during import.
This approach has the following benefits:
no need to change the build process of Numpy as it is only an extension of the existing build process.
packages which depends on these libraries can still use the static libraries if and only if they use the same VS toolset as the Numpy uses.
packages which depends on these libraries using a different toolset or a different compiler can link against the shared libraries instead.
For a test I have successfully build npymath.dll with VS link.exe. It only needs a def file with all symbol names to be exported. A test to build scipy with this change will follow.
There is a disadvantage as well:
One has to consider the situation that someone installs an Numpy version without theses additional DLLs and then Scipy which needs them. Mitigation could be:
build Scipy for windows with a different minimal Numpy version to avoid this situation.
copy these DLLs to the Scipy wheel as well and load them on import.
Ship the C header and sources for Npymath, so external libraries and packages can recompile them.
Not a terrible idea in principle, but it would probably be as much work as shipping a DLL and permanently increase build times for libraries using that C code.
Create a dynamic library from Npymath, and ship that, for linkage against external libraries.
This is probably best. I agree with @carlkl's suggestion's mostly, with two tweaks:
It should be done on all platforms and not just on Windows
The install location isn't arbitrary, we should install to libdir (sysconfig.get_config_var('LIBDIR'))
build Scipy for windows with a different minimal Numpy version to avoid this situation.
This would be the better option - if you need a newer NumPy, then bump the minimum version.
I came across this again when building NumPy from source when a wheel was not available. It would be nice to solve this for 1.23 since moving forward more people will be hitting this when they compile NumPy and Scipy from source with a modern visual studio.
One complication is that on windows, sysconfig.get_config_var('LIBDIR') does not exist. There is a DLL directory under sysconfig.get_config_var('base') (which is the same as sysconfig.get_path('base')) which would be appropriate. The import lib npymath.lib needed for linking currently lives in site-packages\numpy\core\lib, so it can probably stay there.
I am becoming convinced that the DLLs directory in windows is not a good target. It is not added to the dll search path (see os.add_dll_directory), rather it is added to sys.path. So any project linking to npymath would need to call `os.add_dll_directory(), and there is no canonical way to discover what that directory is. In fact on vanilla python from python.org, no directory is added to the dll search path, so there is no reason to prefer the DLLs directory over any random directory in site-packages. Are there any best-practices around this subject?
I came across this again when building NumPy from source when a wheel was not available.
How exactly? This issue was opened when we discovered it with the mingw-w64 compilers. What's your exact config here, and how does this turn up when building numpy only rather than linking libnpymath in another project?
One complication is that on windows, sysconfig.get_config_var('LIBDIR') does not exist.
I'm not sure what sysconfig.get_config_var('LIBDIR') is supposed to do, so ignore that part of my suggestion. What I meant is: the lib directory where the libpython for the Python version you are using is located. You have to link that too, so it should be discoverable somehow.
Linking is the easy part. The harder problem is to find the DLL at runtime. Like LD_LIBRARY_PATH on Linux, windows has an internal list of directories it will search. For the NumPy/SciPy OpenBLAS dll, we call os.add_dll_directory in __config__.py. If we ship npmath.dll, consumers would need to also add something like this to their code.
I would like to return to the idea of shipping the C sources. Perhaps that would just be as simple as copying the generated .c and .h sources in the npymath directory into a directory, such as <np-include>/npymath.
I think that would be easy to maintain - in practice, surely easier than working out a general mechanism for Numpy to export a DLL directory? Or do we need that for some other reason?
I would propose to copy npymath.dll from numpy into scipy's .libs folder during the scipy build process. And to import npymath.dll from there during scipy import.
As npymath.dll is quite small (I tried this out it in januarý and got a 56kB sized npymath.dll) it could be a good idea to simply load this DLL during numpy import. Once the DLL is loaded into process space it will be available for all packages depending on this DLL. Symbols of this DLL does not interfere with the static linked symbols used by numpy, that means numpy will just ignore this DLL.
I think that would be easy to maintain - in practice, surely easier than working out a general mechanism for Numpy to export a DLL directory? Or do we need that for some other reason?
As npymath.dll is quite small (I tried this out it in januarý and got a 56kB sized npymath.dll) it could be a good idea to simply load this DLL during numpy import.
Of the three ways of using a DLL, this is probably the nicest one - assuming we won't run into trouble with changes in the library & versioning. libnpyrandom.a is ~25% larger than libnpymath.a, so that's still okay.
I agree - that's an excellent idea - preloading the DLL. Would this be the right approach for npyrandom? Or is that more likely to be compiled into projects that don't import Numpy?
Removes dependency on npymath and npyrandom static libraries shipped with numpy
Adds npymath as a meson subproject
Vendors minimal libnpyrandom directly
I created the libnpymath repo under my own username for the time being and it probably needs to be reviewed, since I changed quite a few things (including a rewrite of the C++ parts to C 🎉 )
So the current state is this:
npyrandom: Maintain inside numpy, but without any dependencies on numpy or npymath
npymath: Meson subproject / git submodule inside numpy, stand-alone repo available that can be built indepndently, pure C
scipy: No dependencies on the static libraries shipped by numpy, vendor all of npymath as a meson suproject / git submodule, vendor a very minimal version of npyrandom directly
That means that we can effectively stop shipping both of the static libraries and SciPy will work okay. Should we?
Also, if people think all this looks okay, should we move the libnpymath repo to the numpy org after someone reviews it?
Thanks for the update @lysnikolaou, this is starting to shape up nicely.
I created the libnpymath repo under my own username for the time being and it probably needs to be reviewed, since I changed quite a few things (including a rewrite of the C++ parts to C 🎉 )
Would it make sense to separate out that C++ to C rewrite and get that merged as a separate PR to the main numpy repo first? That would make review easier, because it's then split into smaller chunks and also because the primary reviewers for C++/C and build&packaging topics tend to no be the same.
That means that we can effectively stop shipping both of the static libraries and SciPy will work okay. Should we?
We should finish the SciPy changes first, and get those in before branching 1.13.x in 1-2 months. And then announce the plan to shop shipping static libraries - which may be on the late side for 2.0, not sure about that one yet. If it isn't feasible for 2.0, then it'll be by 2.2.0 probably if there aren't too many objections.
Also, if people think all this looks okay, should we move the libnpymath repo to the numpy org after someone reviews it?
+1 to that. It also seems like a reasonable name to continue using.
Did the symbol naming issue to avoid npy_* name clashes get resolved yet, and if so, how?
Would it make sense to separate out that C++ to C rewrite and get that merged as a separate PR to the main numpy repo first?
Sure! Actually, I opened a PR for that on my fork as well, cause I figured it might be needed, so that can be reverted, reviewed and reapplied as well, if adding it on numpy first seems like more effort than needed.
We should finish the SciPy changes first, and get those in before branching 1.13.x in 1-2 months.
The SciPy PR could be merged as-is as far as I'm concerned (after the move of the libnpymath repo), but people need to review it and see whether there's anything wrong with it, since I might be missing something.
Did the symbol naming issue to avoid npy_* name clashes get resolved yet, and if so, how?
Yes, all the internal symbols were renamed to npymath_* and all the external symbols that are exported on numpy are still npy_*, but that's no problem since those do not get defined on the numpy side anymore. The only real issue was getting calls to npymath exported functions to work, since they're defined as accepting npymath_* types, but they're being passed npy_* types. Because of the nature of the typedefs, most of this played out of the box, with the exception of complex types under C++, which are defined as opaque types in npymath when it detects that it is built for use inside numpy.
Yes, all the internal symbols were renamed to npymath_* and all the external symbols that are exported on numpy are still npy_*, but that's no problem since those do not get defined on the numpy side anymore.
I checked the symbol names in the subproject on your SciPy PR with nm build/subprojects/libnpymath/src/libnpymath.a and they're still npy_* indeed. I'm not sure I understand the end of your sentence here - this should still be problematic. If we build a SciPy wheel against NumPy 2.0 and then at runtime use NumPy 1.25, we may get conflicts, right?
Indeed I have tested this and it all plays well. I built a wheel with NumPy 2.0 and then run the whole suite with the wheel alongside both NumPy 2.0 and 1.26. All seems to be fine.
My concern is more about the readability of the whole thing. libnpymath basically has all npymath_* symbols, and then all of the sudden some symbols are named npy_*. I'm wondering whether there's a better solution, maybe rename all symbols to npymath_* and instead of installing libnpymath's version of npy_math.h (which is what the current solution does), #include it in a different header that's part of the NumPy source and which provides the compatibility layer with things like:
Unfortunately, I haven't had time to work on this for a while. Last I did, there were some build complications around how complex numbers interact with each other. I'd documented that here.
Another reason to get rid of libnpymath: it turns out we cannot inline the very small float16 conversion functions since they reside in a separate library.
Another reason to get rid of libnpymath: it turns out we cannot inline the very small float16 conversion functions since they reside in a separate library.
We discussed this in the community meeting last week, and I think the outcome is that there's an action to take functionality like this and either put it in the NumPy C API or use it internally from source and not through the static library (whatever is most appropriate). NumPy itself should be decoupled from using libnpymath as a static library.
@carlkl yes it is still valid. We finally managed to remove libnpymath usage from SciPy (see scipy/scipy#24788), that will be in the 2.5.0 release in 1-2 months from now. That means we no longer have any known large users - the only other user I'm aware of is @WarrenWeckesser with ufunclab and IIRC one or two other small packages. Those shouldn't be blockers I'd expect, if we could get statsmodels and scipy cleaned up, we should be able to clean up anything else as well.
Progress is very slow, but eventually we should get there.
Activity
rgommers commentedon Jan 23, 2022
Note that whether or not this is a priority is kinda coupled with the
vc141vs. newer MS compiler toolchain discussion in MacPython/numpy-wheels#145. Forvc141it seems we can work withlibnpymath.ain a Mingw-w64 based SciPy build, which withvc142things just seem broken and likely hard/fragile to repair.We could do something proactively, but the urgency is a lot lower as long as there's no need to upgrade from
vc141.carlkl commentedon Jan 25, 2022
my suggestion (Windows only) is as follows:
npymath.libandnpyrandom.libas they are for numpy.npymath.dllandnpyrandom.dllfrom these static libraries.This approach has the following benefits:
For a test I have successfully build
npymath.dllwith VSlink.exe. It only needs a def file with all symbol names to be exported. A test to build scipy with this change will follow.There is a disadvantage as well:
One has to consider the situation that someone installs an Numpy version without theses additional DLLs and then Scipy which needs them. Mitigation could be:
rgommers commentedon Jan 29, 2022
Not a terrible idea in principle, but it would probably be as much work as shipping a DLL and permanently increase build times for libraries using that C code.
This is probably best. I agree with @carlkl's suggestion's mostly, with two tweaks:
sysconfig.get_config_var('LIBDIR'))This would be the better option - if you need a newer NumPy, then bump the minimum version.
mattip commentedon Apr 14, 2022
I came across this again when building NumPy from source when a wheel was not available. It would be nice to solve this for 1.23 since moving forward more people will be hitting this when they compile NumPy and Scipy from source with a modern visual studio.
One complication is that on windows,
sysconfig.get_config_var('LIBDIR')does not exist. There is a DLL directory undersysconfig.get_config_var('base')(which is the same assysconfig.get_path('base')) which would be appropriate. The import libnpymath.libneeded for linking currently lives insite-packages\numpy\core\lib, so it can probably stay there.mattip commentedon Apr 15, 2022
I am becoming convinced that the DLLs directory in windows is not a good target. It is not added to the dll search path (see os.add_dll_directory), rather it is added to
sys.path. So any project linking to npymath would need to call `os.add_dll_directory(), and there is no canonical way to discover what that directory is. In fact on vanilla python from python.org, no directory is added to the dll search path, so there is no reason to prefer the DLLs directory over any random directory in site-packages. Are there any best-practices around this subject?rgommers commentedon Apr 15, 2022
How exactly? This issue was opened when we discovered it with the mingw-w64 compilers. What's your exact config here, and how does this turn up when building numpy only rather than linking
libnpymathin another project?I'm not sure what
sysconfig.get_config_var('LIBDIR')is supposed to do, so ignore that part of my suggestion. What I meant is: the lib directory where thelibpythonfor the Python version you are using is located. You have to link that too, so it should be discoverable somehow.mattip commentedon Apr 15, 2022
See this to build Scipy on PyPy using meson: it builds NumPy from source on the CI machine.
Linking is the easy part. The harder problem is to find the DLL at runtime. Like LD_LIBRARY_PATH on Linux, windows has an internal list of directories it will search. For the NumPy/SciPy OpenBLAS dll, we call
os.add_dll_directoryin__config__.py. If we shipnpmath.dll, consumers would need to also add something like this to their code.numpy/numpy/distutils/misc_util.py
Lines 2354 to 2359 in 46ec71f
matthew-brett commentedon Apr 15, 2022
I would like to return to the idea of shipping the C sources. Perhaps that would just be as simple as copying the generated
.cand.hsources in thenpymathdirectory into a directory, such as<np-include>/npymath.I think that would be easy to maintain - in practice, surely easier than working out a general mechanism for Numpy to export a DLL directory? Or do we need that for some other reason?
carlkl commentedon Apr 16, 2022
I would propose to copy npymath.dll from numpy into scipy's
.libsfolder during the scipy build process. And to import npymath.dll from there during scipy import.carlkl commentedon Apr 16, 2022
Another proposal:
As npymath.dll is quite small (I tried this out it in januarý and got a 56kB sized npymath.dll) it could be a good idea to simply load this DLL during numpy import. Once the DLL is loaded into process space it will be available for all packages depending on this DLL. Symbols of this DLL does not interfere with the static linked symbols used by numpy, that means numpy will just ignore this DLL.
rgommers commentedon Apr 18, 2022
Looking at the code to build
npymathin https://github.com/numpy/numpy/blob/main/numpy/core/setup.py, it's not exactly trivial with compiler flags, depending on math libs, etc. (although consumers may already have similar code in their own build setup, not sure). Also, note that we have to givenpyrandomthe same treatment.I think I like the DLL plan a little better.
Of the three ways of using a DLL, this is probably the nicest one - assuming we won't run into trouble with changes in the library & versioning.
libnpyrandom.ais ~25% larger thanlibnpymath.a, so that's still okay.matthew-brett commentedon Apr 18, 2022
I agree - that's an excellent idea - preloading the DLL. Would this be the right approach for npyrandom? Or is that more likely to be compiled into projects that don't import Numpy?
rgommers commentedon Apr 18, 2022
I can't find many users, but SciPy uses both, and Erotemic/xdev@77a5712#diff-ffb8093dfe715993aed3918bfa9b1fb2e59c13d4a5699aab5088df611f960209R230 is an example of use in a random project I hadn't seen before (still uses
numpy). So I'd say use cases are similar.51 remaining items
rkern commentedon Dec 8, 2023
That all sounds reasonable, but I never really had a firm first-hand grasp on the problems these static libraries were meant to solve.
rgommers commentedon Dec 15, 2023
xref gh-25390, which attempts to perform the libnpyrandom surgery.
lysnikolaou commentedon Dec 19, 2023
So a status update here:
I've opened two PRs that try to do a quite big overhaul in the hopes of improving the situation:
MAINT: Add stand-alone libnpymath and remove dependency to numpy from npyrandom #25390
MAINT: Vendor npymath as git submodule and vendor npyrandom directly scipy/scipy#19638
I created the libnpymath repo under my own username for the time being and it probably needs to be reviewed, since I changed quite a few things (including a rewrite of the C++ parts to C 🎉 )
So the current state is this:
npyrandom: Maintain inside numpy, but without any dependencies on numpy or npymathnpymath: Meson subproject / git submodule inside numpy, stand-alone repo available that can be built indepndently, pure Cscipy: No dependencies on the static libraries shipped by numpy, vendor all of npymath as a meson suproject / git submodule, vendor a very minimal version of npyrandom directlyThat means that we can effectively stop shipping both of the static libraries and SciPy will work okay. Should we?
Also, if people think all this looks okay, should we move the
libnpymathrepo to the numpy org after someone reviews it?rgommers commentedon Dec 20, 2023
Thanks for the update @lysnikolaou, this is starting to shape up nicely.
Would it make sense to separate out that C++ to C rewrite and get that merged as a separate PR to the main numpy repo first? That would make review easier, because it's then split into smaller chunks and also because the primary reviewers for C++/C and build&packaging topics tend to no be the same.
We should finish the SciPy changes first, and get those in before branching 1.13.x in 1-2 months. And then announce the plan to shop shipping static libraries - which may be on the late side for 2.0, not sure about that one yet. If it isn't feasible for 2.0, then it'll be by 2.2.0 probably if there aren't too many objections.
+1 to that. It also seems like a reasonable name to continue using.
Did the symbol naming issue to avoid
npy_*name clashes get resolved yet, and if so, how?lysnikolaou commentedon Dec 20, 2023
Sure! Actually, I opened a PR for that on my fork as well, cause I figured it might be needed, so that can be reverted, reviewed and reapplied as well, if adding it on numpy first seems like more effort than needed.
The SciPy PR could be merged as-is as far as I'm concerned (after the move of the
libnpymathrepo), but people need to review it and see whether there's anything wrong with it, since I might be missing something.Yes, all the internal symbols were renamed to
npymath_*and all the external symbols that are exported on numpy are stillnpy_*, but that's no problem since those do not get defined on the numpy side anymore. The only real issue was getting calls to npymath exported functions to work, since they're defined as acceptingnpymath_*types, but they're being passednpy_*types. Because of the nature of the typedefs, most of this played out of the box, with the exception of complex types under C++, which are defined as opaque types in npymath when it detects that it is built for use inside numpy.rgommers commentedon Jan 12, 2024
I checked the symbol names in the subproject on your SciPy PR with
nm build/subprojects/libnpymath/src/libnpymath.aand they're stillnpy_*indeed. I'm not sure I understand the end of your sentence here - this should still be problematic. If we build a SciPy wheel against NumPy 2.0 and then at runtime use NumPy 1.25, we may get conflicts, right?carlkl commentedon Jan 13, 2024
the npymath objects are linked statically, so I can't see a problem here. Any problem with symbol names should show up during the linking step.
lysnikolaou commentedon Jan 16, 2024
Indeed I have tested this and it all plays well. I built a wheel with NumPy 2.0 and then run the whole suite with the wheel alongside both NumPy 2.0 and 1.26. All seems to be fine.
My concern is more about the readability of the whole thing.
libnpymathbasically has allnpymath_*symbols, and then all of the sudden some symbols are namednpy_*. I'm wondering whether there's a better solution, maybe rename all symbols tonpymath_*and instead of installinglibnpymath's version ofnpy_math.h(which is what the current solution does),#includeit in a different header that's part of the NumPy source and which provides the compatibility layer with things like:And then use the
npymath_*symbols directly in SciPy.Obviously, these can also be post-release fixes/improvements if we're in a hurry to make it before NumPy 2.
mattip commentedon Sep 14, 2024
@lysnikolaou where does this work stand? The dust has settled around NumPy 2.0, can we move forward with the separate repo for npmath?
lysnikolaou commentedon Sep 17, 2024
Thanks for the ping, @mattip!
Unfortunately, I haven't had time to work on this for a while. Last I did, there were some build complications around how complex numbers interact with each other. I'd documented that here.
I'll try to find some time to get back to this!
mattip commentedon Sep 8, 2025
Another reason to get rid of libnpymath: it turns out we cannot inline the very small float16 conversion functions since they reside in a separate library.
rgommers commentedon Sep 12, 2025
We discussed this in the community meeting last week, and I think the outcome is that there's an action to take functionality like this and either put it in the NumPy C API or use it internally from source and not through the static library (whatever is most appropriate). NumPy itself should be decoupled from using
libnpymathas a static library.carlkl commentedon May 10, 2026
This scams one minute ago pops this issue in my E-Mail folder.....
However: @rgommers: may I ask you if this is issue still valid? Or in others words: are there still plans to remove this static library from numpy?
rgommers commentedon May 11, 2026
@carlkl yes it is still valid. We finally managed to remove
libnpymathusage from SciPy (see scipy/scipy#24788), that will be in the 2.5.0 release in 1-2 months from now. That means we no longer have any known large users - the only other user I'm aware of is @WarrenWeckesser with ufunclab and IIRC one or two other small packages. Those shouldn't be blockers I'd expect, if we could getstatsmodelsandscipycleaned up, we should be able to clean up anything else as well.Progress is very slow, but eventually we should get there.
rgommers commentedon Sep 3, 2026
Periodic revisiting because of another issue that intersects with this (gh-32482).
SciPy has now also removed its last usages (in 1.18.0, see scipy/scipy#24788), so the major consumers are gone.
@WarrenWeckesser are you able to get rid of the
npymathlinkage? It'd be great if we could go ahead with a deprecation.