Skip to content

ENH: Future of npymath library #20880

Open

Description

@matthew-brett

Proposed new feature or change:

This is an issue to discuss the future of the npymath library in Numpy.

Numpy ships with a static library in core/lib named npymath.lib (Windows) or libnpymath.a (Unices).

It provides a suitable platform-specific implementation of various math routines.

Scipy links to npymath during its build process, in various places.

We (Scipy) have run into problems linking against Npymath, because of the combination of Numpy's use of the latest Visual Studio toolchain, and Scipy's use of the Mingw-w64 toolchain - discussed in MacPython/numpy-wheels#145 .

The issue that arises is that it is in general more difficult to link static libraries across toolchains, than it is to link against dynamic libraries (e.g. DLLs) or to recompile the sources. This issue is to discuss whether we should think of other solutions. Possible options are:

  • Leave things as they are.
  • Ship the C header and sources for Npymath, so external libraries and packages can recompile them.
  • Create a dynamic library from Npymath, and ship that, for linkage against external libraries.
  • Your option here.

This issue is to house the discussion of different options.

Activity

  1. rgommers commented on Jan 23, 2022

    @rgommers
    Member

    Note that whether or not this is a priority is kinda coupled with the vc141 vs. newer MS compiler toolchain discussion in MacPython/numpy-wheels#145. For vc141 it seems we can work with libnpymath.a in a Mingw-w64 based SciPy build, which with vc142 things just seem broken and likely hard/fragile to repair.

    We could do something proactively, but the urgency is a lot lower as long as there's no need to upgrade from vc141.

  2. carlkl commented on Jan 25, 2022

    @carlkl
    Member

    my suggestion (Windows only) is as follows:

    • leave npymath.lib and npyrandom.lib as they are for numpy.
    • Numpy could create two additional shared binaries npymath.dll and npyrandom.dll from these static libraries.
    • during the Numpy installation, place these two DLLs in a common location where Scipy or other third-party packages can locate them both during the build process and during import.

    This approach has the following benefits:

    • no need to change the build process of Numpy as it is only an extension of the existing build process.
    • packages which depends on these libraries can still use the static libraries if and only if they use the same VS toolset as the Numpy uses.
    • packages which depends on these libraries using a different toolset or a different compiler can link against the shared libraries instead.

    For a test I have successfully build npymath.dll with VS link.exe. It only needs a def file with all symbol names to be exported. A test to build scipy with this change will follow.

    There is a disadvantage as well:

    One has to consider the situation that someone installs an Numpy version without theses additional DLLs and then Scipy which needs them. Mitigation could be:

    • build Scipy for windows with a different minimal Numpy version to avoid this situation.
    • copy these DLLs to the Scipy wheel as well and load them on import.
  3. rgommers commented on Jan 29, 2022

    @rgommers
    Member

    Ship the C header and sources for Npymath, so external libraries and packages can recompile them.

    Not a terrible idea in principle, but it would probably be as much work as shipping a DLL and permanently increase build times for libraries using that C code.

    Create a dynamic library from Npymath, and ship that, for linkage against external libraries.

    This is probably best. I agree with @carlkl's suggestion's mostly, with two tweaks:

    1. It should be done on all platforms and not just on Windows
    2. The install location isn't arbitrary, we should install to libdir (sysconfig.get_config_var('LIBDIR'))

    build Scipy for windows with a different minimal Numpy version to avoid this situation.

    This would be the better option - if you need a newer NumPy, then bump the minimum version.

  4. mattip commented on Apr 14, 2022

    @mattip
    Member

    I came across this again when building NumPy from source when a wheel was not available. It would be nice to solve this for 1.23 since moving forward more people will be hitting this when they compile NumPy and Scipy from source with a modern visual studio.

    One complication is that on windows, sysconfig.get_config_var('LIBDIR') does not exist. There is a DLL directory under sysconfig.get_config_var('base') (which is the same as sysconfig.get_path('base')) which would be appropriate. The import lib npymath.lib needed for linking currently lives in site-packages\numpy\core\lib, so it can probably stay there.

  5. mattip commented on Apr 15, 2022

    @mattip
    Member

    I am becoming convinced that the DLLs directory in windows is not a good target. It is not added to the dll search path (see os.add_dll_directory), rather it is added to sys.path. So any project linking to npymath would need to call `os.add_dll_directory(), and there is no canonical way to discover what that directory is. In fact on vanilla python from python.org, no directory is added to the dll search path, so there is no reason to prefer the DLLs directory over any random directory in site-packages. Are there any best-practices around this subject?

  6. rgommers commented on Apr 15, 2022

    @rgommers
    Member

    I came across this again when building NumPy from source when a wheel was not available.

    How exactly? This issue was opened when we discovered it with the mingw-w64 compilers. What's your exact config here, and how does this turn up when building numpy only rather than linking libnpymath in another project?

    One complication is that on windows, sysconfig.get_config_var('LIBDIR') does not exist.

    I'm not sure what sysconfig.get_config_var('LIBDIR') is supposed to do, so ignore that part of my suggestion. What I meant is: the lib directory where the libpython for the Python version you are using is located. You have to link that too, so it should be discoverable somehow.

  7. mattip commented on Apr 15, 2022

    @mattip
    Member

    How exactly?

    See this to build Scipy on PyPy using meson: it builds NumPy from source on the CI machine.

    You have to link that too

    Linking is the easy part. The harder problem is to find the DLL at runtime. Like LD_LIBRARY_PATH on Linux, windows has an internal list of directories it will search. For the NumPy/SciPy OpenBLAS dll, we call os.add_dll_directory in __config__.py. If we ship npmath.dll, consumers would need to also add something like this to their code.

    if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    if sys.version_info >= (3, 8):
    os.add_dll_directory(extra_dll_dir)
    else:
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir
    .

  8. matthew-brett commented on Apr 15, 2022

    @matthew-brett
    ContributorAuthor

    I would like to return to the idea of shipping the C sources. Perhaps that would just be as simple as copying the generated .c and .h sources in the npymath directory into a directory, such as <np-include>/npymath.

    I think that would be easy to maintain - in practice, surely easier than working out a general mechanism for Numpy to export a DLL directory? Or do we need that for some other reason?

  9. carlkl commented on Apr 16, 2022

    @carlkl
    Member

    I would propose to copy npymath.dll from numpy into scipy's .libs folder during the scipy build process. And to import npymath.dll from there during scipy import.

  10. carlkl commented on Apr 16, 2022

    @carlkl
    Member

    Another proposal:

    As npymath.dll is quite small (I tried this out it in januarý and got a 56kB sized npymath.dll) it could be a good idea to simply load this DLL during numpy import. Once the DLL is loaded into process space it will be available for all packages depending on this DLL. Symbols of this DLL does not interfere with the static linked symbols used by numpy, that means numpy will just ignore this DLL.

  11. rgommers commented on Apr 18, 2022

    @rgommers
    Member

    I think that would be easy to maintain - in practice, surely easier than working out a general mechanism for Numpy to export a DLL directory? Or do we need that for some other reason?

    Looking at the code to build npymath in https://github.com/numpy/numpy/blob/main/numpy/core/setup.py, it's not exactly trivial with compiler flags, depending on math libs, etc. (although consumers may already have similar code in their own build setup, not sure). Also, note that we have to give npyrandom the same treatment.

    I think I like the DLL plan a little better.

    As npymath.dll is quite small (I tried this out it in januarý and got a 56kB sized npymath.dll) it could be a good idea to simply load this DLL during numpy import.

    Of the three ways of using a DLL, this is probably the nicest one - assuming we won't run into trouble with changes in the library & versioning. libnpyrandom.a is ~25% larger than libnpymath.a, so that's still okay.

  12. matthew-brett commented on Apr 18, 2022

    @matthew-brett
    ContributorAuthor

    I agree - that's an excellent idea - preloading the DLL. Would this be the right approach for npyrandom? Or is that more likely to be compiled into projects that don't import Numpy?

  13. rgommers commented on Apr 18, 2022

    @rgommers
    Member

    I can't find many users, but SciPy uses both, and Erotemic/xdev@77a5712#diff-ffb8093dfe715993aed3918bfa9b1fb2e59c13d4a5699aab5088df611f960209R230 is an example of use in a random project I hadn't seen before (still uses numpy). So I'd say use cases are similar.

  14. 51 remaining items

  15. rkern commented on Dec 8, 2023

    @rkern
    Member

    That all sounds reasonable, but I never really had a firm first-hand grasp on the problems these static libraries were meant to solve.

  16. rgommers commented on Dec 15, 2023

    @rgommers
    Member

    xref gh-25390, which attempts to perform the libnpyrandom surgery.

  17. lysnikolaou commented on Dec 19, 2023

    @lysnikolaou
    Member

    So a status update here:

    I've opened two PRs that try to do a quite big overhaul in the hopes of improving the situation:

    I created the libnpymath repo under my own username for the time being and it probably needs to be reviewed, since I changed quite a few things (including a rewrite of the C++ parts to C 🎉 )

    So the current state is this:

    • npyrandom: Maintain inside numpy, but without any dependencies on numpy or npymath
    • npymath: Meson subproject / git submodule inside numpy, stand-alone repo available that can be built indepndently, pure C
    • scipy: No dependencies on the static libraries shipped by numpy, vendor all of npymath as a meson suproject / git submodule, vendor a very minimal version of npyrandom directly

    That means that we can effectively stop shipping both of the static libraries and SciPy will work okay. Should we?

    Also, if people think all this looks okay, should we move the libnpymath repo to the numpy org after someone reviews it?

  18. rgommers commented on Dec 20, 2023

    @rgommers
    Member

    Thanks for the update @lysnikolaou, this is starting to shape up nicely.

    I created the libnpymath repo under my own username for the time being and it probably needs to be reviewed, since I changed quite a few things (including a rewrite of the C++ parts to C 🎉 )

    Would it make sense to separate out that C++ to C rewrite and get that merged as a separate PR to the main numpy repo first? That would make review easier, because it's then split into smaller chunks and also because the primary reviewers for C++/C and build&packaging topics tend to no be the same.

    That means that we can effectively stop shipping both of the static libraries and SciPy will work okay. Should we?

    We should finish the SciPy changes first, and get those in before branching 1.13.x in 1-2 months. And then announce the plan to shop shipping static libraries - which may be on the late side for 2.0, not sure about that one yet. If it isn't feasible for 2.0, then it'll be by 2.2.0 probably if there aren't too many objections.

    Also, if people think all this looks okay, should we move the libnpymath repo to the numpy org after someone reviews it?

    +1 to that. It also seems like a reasonable name to continue using.

    Did the symbol naming issue to avoid npy_* name clashes get resolved yet, and if so, how?

  19. lysnikolaou commented on Dec 20, 2023

    @lysnikolaou
    Member

    Would it make sense to separate out that C++ to C rewrite and get that merged as a separate PR to the main numpy repo first?

    Sure! Actually, I opened a PR for that on my fork as well, cause I figured it might be needed, so that can be reverted, reviewed and reapplied as well, if adding it on numpy first seems like more effort than needed.

    We should finish the SciPy changes first, and get those in before branching 1.13.x in 1-2 months.

    The SciPy PR could be merged as-is as far as I'm concerned (after the move of the libnpymath repo), but people need to review it and see whether there's anything wrong with it, since I might be missing something.

    Did the symbol naming issue to avoid npy_* name clashes get resolved yet, and if so, how?

    Yes, all the internal symbols were renamed to npymath_* and all the external symbols that are exported on numpy are still npy_*, but that's no problem since those do not get defined on the numpy side anymore. The only real issue was getting calls to npymath exported functions to work, since they're defined as accepting npymath_* types, but they're being passed npy_* types. Because of the nature of the typedefs, most of this played out of the box, with the exception of complex types under C++, which are defined as opaque types in npymath when it detects that it is built for use inside numpy.

  20. rgommers commented on Jan 12, 2024

    @rgommers
    Member

    Yes, all the internal symbols were renamed to npymath_* and all the external symbols that are exported on numpy are still npy_*, but that's no problem since those do not get defined on the numpy side anymore.

    I checked the symbol names in the subproject on your SciPy PR with nm build/subprojects/libnpymath/src/libnpymath.a and they're still npy_* indeed. I'm not sure I understand the end of your sentence here - this should still be problematic. If we build a SciPy wheel against NumPy 2.0 and then at runtime use NumPy 1.25, we may get conflicts, right?

  21. carlkl commented on Jan 13, 2024

    @carlkl
    Member

    the npymath objects are linked statically, so I can't see a problem here. Any problem with symbol names should show up during the linking step.

  22. lysnikolaou commented on Jan 16, 2024

    @lysnikolaou
    Member

    Indeed I have tested this and it all plays well. I built a wheel with NumPy 2.0 and then run the whole suite with the wheel alongside both NumPy 2.0 and 1.26. All seems to be fine.

    My concern is more about the readability of the whole thing. libnpymath basically has all npymath_* symbols, and then all of the sudden some symbols are named npy_*. I'm wondering whether there's a better solution, maybe rename all symbols to npymath_* and instead of installing libnpymath's version of npy_math.h (which is what the current solution does), #include it in a different header that's part of the NumPy source and which provides the compatibility layer with things like:

    #include <npymath/npymath.h>
    
    #define npy_log2 npymath_log2
    

    And then use the npymath_* symbols directly in SciPy.

    Obviously, these can also be post-release fixes/improvements if we're in a hurry to make it before NumPy 2.

  23. mattip commented on Sep 14, 2024

    @mattip
    Member

    @lysnikolaou where does this work stand? The dust has settled around NumPy 2.0, can we move forward with the separate repo for npmath?

  24. lysnikolaou commented on Sep 17, 2024

    @lysnikolaou
    Member

    Thanks for the ping, @mattip!

    Unfortunately, I haven't had time to work on this for a while. Last I did, there were some build complications around how complex numbers interact with each other. I'd documented that here.

    I'll try to find some time to get back to this!

  25. mattip commented on Sep 8, 2025

    @mattip
    Member

    Another reason to get rid of libnpymath: it turns out we cannot inline the very small float16 conversion functions since they reside in a separate library.

  26. rgommers commented on Sep 12, 2025

    @rgommers
    Member

    Another reason to get rid of libnpymath: it turns out we cannot inline the very small float16 conversion functions since they reside in a separate library.

    We discussed this in the community meeting last week, and I think the outcome is that there's an action to take functionality like this and either put it in the NumPy C API or use it internally from source and not through the static library (whatever is most appropriate). NumPy itself should be decoupled from using libnpymath as a static library.

  27. carlkl commented on May 10, 2026

    @carlkl
    Member

    This scams one minute ago pops this issue in my E-Mail folder.....

    However: @rgommers: may I ask you if this is issue still valid? Or in others words: are there still plans to remove this static library from numpy?

  28. rgommers commented on May 11, 2026

    @rgommers
    Member

    @carlkl yes it is still valid. We finally managed to remove libnpymath usage from SciPy (see scipy/scipy#24788), that will be in the 2.5.0 release in 1-2 months from now. That means we no longer have any known large users - the only other user I'm aware of is @WarrenWeckesser with ufunclab and IIRC one or two other small packages. Those shouldn't be blockers I'd expect, if we could get statsmodels and scipy cleaned up, we should be able to clean up anything else as well.

    Progress is very slow, but eventually we should get there.

  29. rgommers commented on Sep 3, 2026

    @rgommers
    Member

    Periodic revisiting because of another issue that intersects with this (gh-32482).

    SciPy has now also removed its last usages (in 1.18.0, see scipy/scipy#24788), so the major consumers are gone.

    @WarrenWeckesser are you able to get rid of the npymath linkage? It'd be great if we could go ahead with a deprecation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions