Repository navigation
scipy.fft: Can OpenMP be used portably #10239
Description
Related to #10238
Can we build and use OpenMP in a portable way?
This will require compiler dependent build arguments which I can't see explicitly mentioned in the numpy distutils guide. It also needs to link against the OpenMP runtime which may be an issue.
Activity
larsoner commentedon May 31, 2019
Can we start without it and add it as an enhancement later?
If it's an integral / non-optional part of the software we'll have to figure this out now. But if not, I'd rather wait. As soon as you talk about multithreading the C code, it can be very useful, but you have to think about things like can you set the number of threads for each call, what's the interface for doing so, etc. Because even on a multicore machine, you might know that you are going to use
joblibto do N tasks over N CPUs, each of which uses calls toscipy.fftso really you want each of these to do a single-threaded computation. So things like this can increase the complexity level.You'll probably notice a theme in these responses: trying to start with something small that works and build up from there, rather than trying to include everything at once. It allows for easier review and also easier division of labor (e.g., someone else could do OpenMP once the basics of
scipy.fftare in place if they are motivated to do so).peterbell10 commentedon May 31, 2019
Okay, I'll remove the worker argument for now and build without OpenMP on all platforms.
rgommers commentedon May 31, 2019
I am interested in figuring out the status of OpenMP though. I've looked for an up-to-date description before, but didn't find one. Does anyone know of one? It would need to cover:
grlee77 commentedon May 31, 2019
Here is detailed OpenMP compiler support info
Calls to
omp_set_num_threadstake precedence over theOMP_NUM_THREADSenvironment variable. There is also anum_threadsclause that can be used to override either of these. This is explained in detail in here.grlee77 commentedon May 31, 2019
For reference, here is a concrete example from the DiPy project which uses OpenMP within Cython code (probably a bit more complex that what would be needed here as we probably don't need to call
omp_set_num_threads, etc, explicitly if pypocketfft does that for us):At build time,
setup.pyattempts to compile a simple source importingomp.hand if it succeeds, OpenMP support is enabled:https://github.com/nipy/dipy/blob/4b24d30ba6e4cb959d2cc788362241361c80f024/setup.py#L123-L141
where the
add_flag_checkingclass is defined in setup_helpers.py:https://github.com/nipy/dipy/blob/4b24d30ba6e4cb959d2cc788362241361c80f024/setup_helpers.py#L81
Calls to
omp_set_num_threads,omp_get_num_threadsetc can be made at theCythonlevel via:https://github.com/nipy/dipy/blob/master/dipy/utils/omp.pyx
which relies on:
https://github.com/nipy/dipy/blob/master/src/safe_openmp.pxd
https://github.com/nipy/dipy/blob/master/src/conditional_omp.h
Finally, an example of dynamically setting and then restoring the number of threads based on user input from within a Cython function:
https://github.com/nipy/dipy/blob/4b24d30ba6e4cb959d2cc788362241361c80f024/dipy/denoise/denspeed.pyx#L96-L111
andyfaff commentedon Jun 1, 2019
The openmp discussion is getting a bit fragmented now because it's spread over #10242, #10238, and here. This makes it harder for other devs to follow.
In my experience I just couldn't iron out all the issues with openmp - I would be very, very, cautious about making it an integral part of code from the outset (I was scarred by the entire experience).The following points are from that experience:
This would mean that macOS users wouldn't get the performant behaviour. I tried to get around this by installing the
llvm-openmpandclangconda packages, then setting the CC and CXX environment variables to choose the new compiler. However, you then run into the issues of distributing wheels - the openmp library needs to be distributed as well.libiomp,libomp,libgomp. I'm not sure if this issue also applies to other OS's, I develop on macOS.multiprocessing.PoolSee A plan to safely support OpenMP in our Cython code base scikit-learn/scikit-learn#7650 and also https://joblib.readthedocs.io/en/latest/parallel.html#bad-interaction-of-multiprocessing-and-third-party-libraries
I also experienced this when running some Linux tests on through a Docker image. To get around this one can use
joblibwith theLokybackend.joblibis a nice project, but I found (for my particular code) that joblib wasn't as performant as multiprocessing.One also has to consider the overall picture of how the parallelisation is carried out, it's going to be in such a heterogeneous and nested environment. Here are some cases:
differential_evolution) using aPoolof workers] --> want to use as many workers as there are cores. However, each distributed computation (using FFT) should only be single threaded.The bottom line is that precise control is needed over how parallelisation is achieved, with no recourse to environment variables (see point 2), global state, etc.
I went down this path because I was using
pthreadsfor macOS/posix, and Windows parallelisation calls (_beginthreadex,_endthreadex,WaitForMultipleObjects, etc). Using openmp would've done a slight amount of cleanup in the code. However, in the end it just wasn't worth it, there were too many difficulties.Would it be possible to continue discussions on this kind of thing on scipy-dev?
larsoner commentedon Jun 1, 2019
Yep, sounds good. Perhaps copy-paste what you wrote above to move the discussion there?
For this specific issue, I think this implies we should not include it right now and add it later (if ever).
mreineck commentedon Aug 16, 2019
Proposed parallelization for
scipy.fftis now standard multithreading instead of OpenMP (#10614), so this can perhaps be closed.larsoner commentedon Aug 16, 2019
Agreed, thanks @mreineck
rgommers commentedon Jan 9, 2021
Linking the related mailing list thread which has quite a bit of relevant content, like how scikit-learn does manage to ship wheels with OpenMP: https://mail.python.org/pipermail/scipy-dev/2019-August/023635.html
rgommers commentedon Mar 10, 2021
This is now the canonical issue for "why is OpenMP not allowed", and I wanted a shorter summary to reference. So here's an attempt:
multiprocessing.libgompandlibiomp) don't play too well together.h-vetinari commentedon Mar 20, 2021
Also adding the following 2019 discussion from the ML for reference for anyone interested.
(Not my sleuthing, mentioned by @jb-leger in #13576)
ogrisel commentedon Nov 22, 2021
Only libgomp (the default OpenMP runtime implementation that comes with GCC) is not fork-safe. The Microsoft OpenMP runtime (used by default when building with MSVC) and the LLVM OpenMP runtime (used by default when building with clang) are all safe.
On Linux, it's possible to build with GCC and then replace libgomp by llvm-openmp. It should also be possible to build with clang directly. More details in the conda-forge documentation: https://conda-forge.org/docs/maintainer/knowledge_base.html#openmp
mreineck commentedon Nov 23, 2021
If the OpenMP standard itself does not mandate this form of fork safety, I'd personally not rely on added guarantees that are are made by specific implementations.
At least for C++ the current approach with a thread pool implemented with standard language features is very convenient to use, and (although I was reluctant to switch away from OpenMP myself at first) I'm now using it in my other C++ projects as well and am really happy with reliability and portability.
However I admit that this is probably not a great help when multithreading is needed in extensions written in C or Fortran...
ogrisel commentedon Nov 23, 2021
or Cython, or pythran or numba which all have OpenMP integration.
peterbell10 commentedon Nov 25, 2021
The problem isn't that you can't use a fork-safe OpenMP; it's that you need to be able to guarantee no user at runtime will end up running
libgomp. On PyTorch for example, we've had some bug reports that depend on whetherimport numpyorimport torchhappens first because on some environments you end up using a different version of OpenMP.