Repository navigation
Numpy 1.26.0 doesn't work on freebsd #24873
Description
Steps to reproduce:
python3.11 -m venv npv
. npv/bin/activate
pip install numpy==1.26.0
python
>>> import numpy
Error message:
Python 3.11.5 (main, Sep 7 2023, 03:42:48) [Clang 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386 on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "/usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/__init__.py", line 130, in <module>
from numpy.__config__ import show as show_config
File "/usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/__config__.py", line 4, in <module>
from numpy.core._multiarray_umath import (
File "/usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/core/__init__.py", line 100, in <module>
from . import _add_newdocs
File "/usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/core/_add_newdocs.py", line 4973, in <module>
add_newdoc('numpy.core._multiarray_tests', 'format_float_OSprintf_g',
File "/usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/core/function_base.py", line 543, in add_newdoc
new = getattr(__import__(place, globals(), {}, [obj]), obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: /usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/core/_multiarray_tests.cpython-311.so: Undefined symbol "ccosl"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/home/lmoskala/npv/lib/python3.11/site-packages/numpy/__init__.py", line 135, in <module>
raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there.
>>>Additional information:
FreeBSD 13.2 with python3.11.5 (python 3.11.2 has the same issue).
If I do pip install numpy==1.24.1, then import numpy succeeds. Numpy 1.25.2 also works. So I'd guess that somewhere between 1.25.2 and 1.26.0 #22850 stopped working? of a new function appeared?
I'll try to provide any informations needed.
Reacted by tehfink
Activity
rgommers commentedon Oct 7, 2023
Thanks for the report @LukaszMoskala. Looking at gh-22850, I noticed it was just linked to from ggml-org/llama.cpp#3511 with a very similar error. So it looks like the complex long double feature checks are somehow not working as they should. They're defined here:
numpy/numpy/core/meson.build
Lines 168 to 187 in 195d85b
@LukaszMoskala the build log should contain the result of every feature check, like this:
Could you re-run the build with the
-vflag forpip, and check what the result of those checks is?rgommers commentedon Oct 7, 2023
I think I see the problem: the use of
cdata.set10will result in: EDIT: probably not true, theif cc.has_functionshould prevent this)however it shouldn't be defined at all - because the C code will check with
#ifndef HAVE_CCOSL.I think it's time to add a FreeBSD CI job when fixing this, because it's too easy to cause a regression on a platform that isn't C99 compliant.
LukaszMoskala commentedon Oct 7, 2023
Sure, here are all my "checking for function..." lines.
rgommers commentedon Oct 7, 2023
Thanks @LukaszMoskala. So the check passes. Something is very badly wrong with those checks, if I look in
build/meson-logs/meson-log.txton Linux I see:but the check passes!
Once I improve the check by adding the missing
complex.hinclude:it starts failing with:
Okay, way more is broken here with the
cc.has_functionchecks, it just happens to work in our CI because we don't test anything on a non-C99-complete platform. This needs fixing asap.rgommers commentedon Oct 7, 2023
The correct invocation is:
Which results in the function being detected and the output being clean:
rgommers commentedon Oct 7, 2023
Upstream issues:
The problem is that if the check is wrong, Meson tries to detect the
__builtinvariant and that may then succeed even if the OS doesn't provide an implementation.rgommers commentedon Oct 7, 2023
Part of the problem here is that a generic
has_functionimplementation isn't going to have the correct prototype for the function, which (according to this comment), makes the__has_builtincheck problematic. Better to have explicit prototypes - we had those in the setup.py build (insidefeature_detection_math.h& co). I guess we should revive those headers - also good for making the configure checks faster.BLD: fix incorrect feature checks for mandatory math functions
rgommers commentedon Oct 7, 2023
@LukaszMoskala I think gh-24876 will fix this - could you please check if that works for you?
LukaszMoskala commentedon Oct 7, 2023
With your fix I get diffrent error:
I used
pip install git+https://github.com/rgommers/numpy.git@fix-math-featurechecks, please correct me if that's wrong.The same error appears when using numpy installed using
pip install git+https://github.com/numpy/numpy.git@mainso that may be unrelated (or I'm doing something very wrong)I'll try applying your patch on 1.26.0
LukaszMoskala commentedon Oct 7, 2023
After applying your patch on 1.26.0 error changes to:
It's still possible that I'm doing something very wrong
rgommers commentedon Oct 7, 2023
I think that that works, although I am not 100% sure about
pipchecking out git submodules correctly (by network is very slow right now, so hard to check). I'd normally clone the repo, then rungit submodule update --initand then check out the desired branch. That's a lot more efficient anyway when testing more than once, because pointingpipat GitHub directly will clone into a tmpdir and then throw everything away.I don't quite understand what the problem there is. The
_npy_scaled_cexplfunction should be defined when the sameHAVE_CEXPLmacro is not defined as used fornpy_cexpl:numpy/numpy/core/src/npymath/npy_math_complex.c.src
Lines 166 to 199 in 195d85b
With a local patch I can ensure
HAVE_CEXP*remains undefined, and I then get the expected symbols inlibnpymath.a(and that is linked intomultiarray_umath.so):Could you post your full build log (in a gist if it's too large)?
LukaszMoskala commentedon Oct 7, 2023
Sure, here it is.
Also I'll try python 3.9
Edit: Exactly the same thing happens on python3.9
rgommers commentedon Oct 7, 2023
Thanks. It looks like that fixed the checks, because
csinlandccoslare now correctly reported asNO:And I see now what's wrong with
_npy_scaled_cexpl- it's used from within a#ifndef HAVE_CCOSH@C@code block, and that is going wrong whenHAVE_CEXPLis true andHAVE_CCOSLis false. That's a clear bug in the code, and I think it's there in 1.25.2 as well after some larger reshuffles innpy_math_complex.c.src(or it may always have been there). The switch to Meson may have exposed it by detectingcexplas available whileccosl/csinlare unavailable - thesetup.pybuild must have seen them all as unavailable.I'm going to try to get that CI job going, because there may be more such issues.
rgommers commentedon Oct 7, 2023
cexplwas actually implemented recently, it landed in FreeBSD 13: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216862. So the feature check results make sense, and this new implementation exposed the bug with_npy_scaled_cexpl.BLD: fix incorrect feature checks for mandatory math functions
BLD: fix math func feature checks, fix FreeBSD build, add CI job (#24876
BLD: fix math func feature checks, fix FreeBSD build, add CI job (num…
BLD: fix math func feature checks, fix FreeBSD build, add CI job (#24876
LukaszMoskala commentedon Oct 9, 2023
I just want to leave instructions on backporting this to 1.26.0 for example for homeassistant which currently uses numpy 1.26.0.
Basically did pip update homeassistant and then:
I hope that this helps someone that needs to use older version with this fix. Also I can confirm that this fix works.
Thank you @charris and @rgommers for quick fix!
ryanwinter commentedon Oct 10, 2023
Thanks @LukaszMoskala, I need this for my home assistant install.
BLD: fix math func feature checks, fix FreeBSD build, add CI job (num…
Fantmur commentedon Nov 13, 2023
Great Thanks @LukaszMoskala,
For Android Termux :
outside of homeassistant venv
git clone https://github.com/numpy/numpy.git numpy-git
cd numpy-git
git checkout v1.26.0
git cherry-pick 040ed2d
git submodule update --init
cd ..
now enter homeassistant venv
LDFLAGS=" -lm -lpython3" pip install numpy-git/