Repository navigation
BUG: Undefined math symbols when compiling from source against intel MKL #27111
Description
Describe the issue:
I have compiled python 3.12.4 from source on a computing cluster, and I am trying to compile numpy. The goal is to install it in the same tree as the locally compiled python binary. We have intel compilers on the cluster, so I am trying to get numpy to use intel MKL libraries. I compiled numpy with
CC=icx CXX=icpx python -m pip install --no-build-isolation -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-static-lp64-iomp -Csetup-args=-Dlapack=mkl-static-lp64-iomp
This installed and compiled without complaints. However, when I try to import numpy from within python, I get
$ python
Python 3.12.4 (main, Jul 29 2024, 10:14:44) [Clang 18.0.0 (icx 2024.1.0.20240308)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/__init__.py", line 23, in <module>
from . import multiarray
File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/multiarray.py", line 10, in <module>
from . import overrides
File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/overrides.py", line 8, in <module>
from numpy._core._multiarray_umath import (
ImportError: /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: pow8_h
An invocation of ldd shows the following libraries linked to the above mentioned shared object file:
$ ldd /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffef53dd000)
libimf.so => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libimf.so (0x000014a0f6287000)
libiomp5.so => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libiomp5.so (0x000014a0f5ea1000)
libsvml.so => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libsvml.so (0x000014a0f484d000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000014a0f4626000)
libintlc.so.5 => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libintlc.so.5 (0x000014a0f45c4000)
libc.so.6 => /usr/lib64/libc.so.6 (0x000014a0f43bb000)
librt.so.1 => /usr/lib64/librt.so.1 (0x000014a0f43b6000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x000014a0f43b1000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x000014a0f43aa000)
/lib64/ld-linux-x86-64.so.2 (0x000014a0f8526000)
libm.so.6 => /usr/lib64/libm.so.6 (0x000014a0f42cf000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x000014a0f42b4000)
So what is missing here?
Reproduce the code example:
import numpyError message:
Traceback (most recent call last):
File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/__init__.py", line 23, in <module>
from . import multiarray
File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/multiarray.py", line 10, in <module>
from . import overrides
File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/overrides.py", line 8, in <module>
from numpy._core._multiarray_umath import (
ImportError: /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: pow8_hPython and NumPy Versions:
Python 3.12.4
Numpy 2.0.1
Runtime Environment:
Cannot show this because importing numpy fails
Context for the issue:
No response
Activity
AgilentGCMS commentedon Aug 4, 2024
@rgommers Apologies for bothering you, but in the past you've helped solve issues at the confluence of numpy and MKL, so perhaps this problem has an obvious solution to you?
mattip commentedon Aug 5, 2024
The missing symbol is
pow8_h. What cpu architecture are you targeting? Google does not seem to be able to find that symbol at all.rgommers commentedon Aug 5, 2024
I can't find any reports about
pow8_hin this issue tracker, but I'm sure I've seen it before - it's a known issue with SVML IIRC.AgilentGCMS commentedon Aug 5, 2024
Intel Ice Lake, or, from
/proc/cpuinfo,r-devulap commentedon Aug 5, 2024
I have seen this before, let me try and find what the mitigation was
AgilentGCMS commentedon Aug 5, 2024
Not that this helps, when I compile against the dynamic library (i.e.,
mkl-dynamic-lp64-iompinstead ofmkl-static-lp64-iomp), I get an error aboutundefined symbol: tan8_h. It's probably the first undefined symbol found, i.e., I don't believe thattan8_hwas defined in the static library but is undefined in the dynamic library.r-devulap commentedon Aug 5, 2024
Could you add this to CFLAGS and rebuild?
-fveclib=none. That should fix the error.r-devulap commentedon Aug 5, 2024
I had filed this issue a while back and forgot about it :) #26257. This flag should be added into our meson build for icx/icpx compiler. I will submit a PR to do that.
AgilentGCMS commentedon Aug 5, 2024
You mean just CFLAGS='-fveclib=none' on the same line where I'm defining CC and CXX? I am not setting any other CFLAGS anywhere.
r-devulap commentedon Aug 5, 2024
yup
AgilentGCMS commentedon Aug 5, 2024
@r-devulap That indeed fixed the issue. Thanks!
AgilentGCMS commentedon Aug 8, 2024
I think I need to reopen this issue, unless what I'm seeing is unrelated. With that addition to CFLAGS, numpy built successfully. However, it fails
import numpy; numpy.test()with the following:6 remaining items
r-devulap commentedon Aug 8, 2024
compile with the
-fp-model=strictflag enabled, obviously :)AgilentGCMS commentedon Aug 8, 2024
@r-devulap I compiled your code with
icx -fp-model=strict test_rdevulap.cand ran the executable:seberg commentedon Aug 9, 2024
The errors look pretty harmless overall. I would guess the math libraries are still using fast-math, so anything that calls out of NumPy into math libs is still sloppy about FPEs.
(Of course they might still require users to disable checking FPEs.)
EDIT: Ah, not sure about the
longdoubleversions being calls out into math libs, I suppose they may also be related tolongdoubleitself. I don't think the f2py thing is a problem.AgilentGCMS commentedon Aug 9, 2024
@seberg Any reason why math libraries would still use fast-math even after I've specified
-fp-model=strict?seberg commentedon Aug 9, 2024
They are dynamically linked and there is only one version of them, no? So either strict, or not.
r-devulap commentedon Aug 9, 2024
@seberg I agree, these look specific to ICPX. Most the test failures seem harmless, except I am not sure about these two:
Do you know what these failures are?
AgilentGCMS commentedon Aug 9, 2024
So these will disappear if I link with the static library? I'll try.
seberg commentedon Aug 9, 2024
I forgot the exact cause but IIRC they are some wrong dtype mappings in f2py (and with that I think just the tests) and probably harmless as well, i.e.the test setup is brittle w.r.t. to new platforms.
r-devulap commentedon Aug 9, 2024
static linking or dynamic link shouldn't fix it, I think. The libraries are precompiled and shipped, changing the way you link to them won't affect their behavior.
r-devulap commentedon Aug 9, 2024
From the docs, -fp-model=strict should raise all appropriate FP exceptions.
AgilentGCMS commentedon Aug 9, 2024
I can confirm that linking to the static library did not fix/change anything. I get the same failures, including the two you were concerned about.
mattip commentedon Aug 11, 2024
I seem to remember this as well.
seberg commentedon Aug 11, 2024
@AgilentGCMS I am not sure about those exact test failures. However:
log2andarcsin) So you can probably get away with this by telling your users that these are harmless warnings and that they can suppress them withnp.errstate. (Not suppressing warnings does come with a non-negligible overhead for small arrays.)AgilentGCMS commentedon Aug 11, 2024
OK, I'm going to close this for now. Thanks for all your help!