Repository navigation
BUG(?): Symbol not found: _cblas_caxpy$NEWLAPACK #25026
Description
Describe the issue:
On my M1 mac, I can build commit d10d17e and import NumPy without issue. With the next commit, 34afbb9, the build seems to be successful, but I get an error when I try to import numpy at the Python prompt.
I get the full error message below when I use an editible install and try to import NumPy. When I use spin test -v, the build seems OK, but it fails with the same ImportError.
In case it matters, I use Miniforge mamba to set up the numpy-dev environment. This is on a fresh system; I did a factory reset in my attempt to debug this. I've read through https://numpy.org/devdocs/user/troubleshooting-importerror.html as recommended and couldn't find a solution there.
Update: I'm upgrading from Ventura to Sonoma to see if that helps.
Reproduce the code example:
import numpyError message:
Traceback (most recent call last):
File "/Users/matthaberland/numpy/numpy/core/__init__.py", line 24, in <module>
from . import multiarray
File "/Users/matthaberland/numpy/numpy/core/multiarray.py", line 10, in <module>
from . import overrides
File "/Users/matthaberland/numpy/numpy/core/overrides.py", line 8, in <module>
from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): Symbol not found: _cblas_caxpy$NEWLAPACK
Referenced from: <47F947F4-FE38-32EE-BDCE-DBA4188AAD7D> /Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so
Expected in: <E6C2D470-8878-372D-B9DB-0EDCA1303272> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/matthaberland/numpy/numpy/__init__.py", line 114, in <module>
from numpy.__config__ import show as show_config
File "/Users/matthaberland/numpy/build/cp39/numpy/__config__.py", line 4, in <module>
from numpy.core._multiarray_umath import (
File "/Users/matthaberland/numpy/numpy/core/__init__.py", line 50, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.9 from "/Users/matthaberland/miniforge3/envs/numpy-dev/bin/python"
* The NumPy version is: "2.0.0.dev0+git20231007.f204c02"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: dlopen(/Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): Symbol not found: _cblas_caxpy$NEWLAPACK
Referenced from: <47F947F4-FE38-32EE-BDCE-DBA4188AAD7D> /Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so
Expected in: <E6C2D470-8878-372D-B9DB-0EDCA1303272> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/matthaberland/numpy/numpy/__init__.py", line 119, in <module>
raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there.Runtime information:
>>> import sys, numpy; print(numpy.__version__); print(sys.version); print(numpy.show_runtime())
2.0.0.dev0+git20231010.d10d17e
3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:53:08)
[Clang 15.0.7 ]
WARNING: `threadpoolctl` not found in system! Install it by `pip install threadpoolctl`. Once installed, try `np.show_runtime` again for more detailed build information
[{'numpy_version': '2.0.0.dev0+git20231010.d10d17e',
'python': '3.9.18 | packaged by conda-forge | (main, Aug 30 2023, '
'03:53:08) \n'
'[Clang 15.0.7 ]',
'uname': uname_result(system='Darwin', node='Matts-MacBook-Air.local', release='22.1.0', version='Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103', machine='arm64')},
{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
'found': ['ASIMDHP'],
'not_found': ['ASIMDFHM']}}]
None
Activity
mattip commentedon Oct 29, 2023
Does
conda listshow any openblas alternatives? Can you attach the meson build log for the failed build? The commit you point to is a massive refactor of the way BLAS is detected, perhaps something in this particular build configuration is off.mdhaber commentedon Oct 29, 2023
Output of
conda list:Details
Log is at https://gist.github.com/mdhaber/cad25624c63a15fa9c392716799b4657
I was planning to update macOS to Sonoma. LMK if I should hold off.
mattip commentedon Oct 29, 2023
The relevant part is this, your build is using a version of Accelerate. @rgommers might know what is going on.
mdhaber commentedon Oct 29, 2023
Yes, that's why I was thinking of updating my OS. I saw that 34afbb9 is just tested with macOS 13.3+. I currently have 13.0 after the factory reset. Not sure what I had before, though.
andyfaff commentedon Oct 30, 2023
The following works for me:
On macOS 14.0 with Xcode 15 and gfortran12 from https://github.com/fxcoudert/gfortran-for-macOS/releases. I wonder if it's a consequence of using the conda compilers?
rgommers commentedon Oct 30, 2023
@mdhaber yes it'd be great if you could hold off on upgrading to 14.0 to see if we can sort this out. This is the first time a bug against building with Accelerate was reported, and I'd like to get to the bottom of it.
The way this is supposed to work is that Accelerate is only detected for macOS >=13.3 and if XCode is new enough so the relevant SDK is installed. This is tested during the build with:
so the output of that on your local system is relevant.
Conda env/compilers yes/no should not be relevant here. The build log output for this Accelerate detection looks as expected. It's kinda unfortunate that you don't know what OS version you were on before. Can you still reproduce the issue? If so, can you check:
xcruncommand above?mdhaber commentedon Oct 30, 2023
Yes, I can still reproduce the issue.
OS: Ventura 13.0
XCode: How do you want me to check independently? Here is the output of a few commands I found online.
xcrunoutput: 13.3rgommers commentedon Oct 30, 2023
Thanks, that helps. This is kinda the opposite problem of #24053 (comment), which was "OS is 13.3, but SDK is older". What is happening here is that your OS version is 13.0 but you do have an SDK for 13.3, which is picked as the default. Hence, a binary is built that will run on 13.3 - but not on 13.0. I didn't realize that this was a thing, I thought the default would always be the current OS version (and using
MACOSX_DEPLOYMENT_TARGETto target an older version, but never a newer version). It's kinda weird that the default picked by the Apple tooling here is something that results in binary that won't run on the system on which it is built.We're going to need an extra safeguard for this. Just checking: does
sw_vers --productVersionreturn13.0for you?rgommers commentedon Oct 30, 2023
Ah, why this "SDK too new" normally isn't really a problem is because the default deployment target indeed is "current OS version" (see https://opensource.apple.com/source/cctools/cctools-667.10.0/libstuff/macosx_deployment_target.c.auto.html) and then using a header will hide the too-new stuff. But in this case we're not actually using the
Accelerate.hheader at all; rather we're usingnpy_cblas.hwhich is a portable-across-blas-libs header. So we won't notice that the NEWLAPACK-suffixed symbols are not present.rgommers commentedon Oct 30, 2023
@mdhaber I think this change will fix it: rgommers/meson@1435e7e. Could you try that? It should have the effect of not selecting Accelerate when you're on <13.3, and once you upgrade to 13.3 or later it will pick it up (and then it'll work).
rgommers commentedon Oct 30, 2023
Updated commit linked above to also include
MACOSX_DEPLOYMENT_TARGET.mdhaber commentedon Oct 31, 2023
sw_vers -productVersion(single dash) returns 13.0, yes. And with that change (single dash), it's working!mdhaber commentedon Oct 31, 2023
While we're at it, Testing Builds says to
python -m pip install -r test_requirements.txt, but if the user just created a conda environment fromenvironment.yml, those packages are already installed. Should the instructions branch between conda and virtualenv like the "Building SciPy from Source" do?rgommers commentedon Oct 31, 2023
Yes, I planned to rewrite basically all the docs, and mostly copying the structure SciPy has because that is much clearer and more comprehensive.
BLD: make macOS version check for Accelerate NEWLAPACK more robust
rgommers commentedon Oct 31, 2023
Figured out there's no need to call
sw_vers, we can useplatform.mac_ver()instead. gh-25040 should close this issue.