BUG: distutils: fix building mixed C/Fortran extensions#20886
BUG: distutils: fix building mixed C/Fortran extensions#20886charris merged 1 commit intonumpy:mainfrom
Conversation
In SciPy we had a couple of cases where we build a Python extension with C source files but linked against static libraries built from Fortran code. Those should be using the Fortran linker, but this was broken in 1.22.0 by numpygh-19713 (the PR that introduced C++ in NumPy). This fixes a few issues in the `build_ext` command, and documents better what is going on there. Should close SciPy issues 8325 and 15414.
rgommers
added
00 - Bug
component: numpy.distutils
09 - Backport-Candidate
|
Thanks Ralf. |
There was a problem hiding this comment.
Just to report on the results downstream, this does seem to fix the issue we are seeing with SciPy 1.7.3 missing symbols when built with latest NumPy using spack.
There are some unrelated curiosities I had to patch separately:
- version handling at untagged commits looks a bit fragile maybe, perhaps
spacks fault, not sure about theversioneerstuff:
==> [2022-01-24-09:06:59.729376] '/vast/home/treddy/github_projects/spack/opt/spack/linux-centos8-skylake_avx512/gcc-9.4.0/python-3.9.9-dpafrcjrc3ajwmfwm6jpvrfn2fuolebn/bin/python3.9' '-s' 'setup.py' '--no-user-cfg' 'build'
Traceback (most recent call last):
File "/ram/tmp/treddy/spack-stage/spack-stage-py-numpy-fix-fortran-linkage-2x66mvauoa65wm5krimrheektwfp4twc/spack-src/setup.py", line 64, in <module>
raise RuntimeError(f'Cannot parse version {FULLVERSION}')
RuntimeError: Cannot parse version 0+untagged.1.g1e83f83
Looks like the numexpr package isn't compatiable with NumPy main yet, which is probably fair until there's an actual release. FWIW, here's the traceback (the py-numpy version was artificially set to something high by me to use a "fake tag" on a debug version of Ralf's branch):
==> Installing py-numexpr-2.7.3-32jnylupzrtvdk5aatwpxtaaqas5gvug
==> No binary for py-numexpr-2.7.3-32jnylupzrtvdk5aatwpxtaaqas5gvug found: installing from source
==> Using cached archive: /vast/home/treddy/github_projects/spack/var/spack/cache/_source-cache/archive/00/00d6b1518605afe0ed10417e0ff07123e5d531c02496c6eed7dd4b9923238e1e.tar.gz
==> No patches needed for py-numexpr
==> py-numexpr: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 1:
'/vast/home/treddy/github_projects/spack/opt/spack/linux-centos8-skylake_avx512/gcc-9.4.0/python-3.9.9-dpafrcjrc3ajwmfwm6jpvrfn2fuolebn/bin/python3.9' '-s' 'setup.py' '--no-user-cfg' 'build'
2 errors found in build log:
288 INFO: compile options: '-I/vast/home/treddy/github_projects/spack/opt/spack/linux-centos8-skylake_avx512/gcc-9.4.0/py-numpy-1.28.0-6jc4hgb7r7umhyjaltfgjkzv6tl5x4dv/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/vast/home/treddy/github_projects/spa
ck/opt/spack/linux-centos8-skylake_avx512/gcc-9.4.0/python-3.9.9-dpafrcjrc3ajwmfwm6jpvrfn2fuolebn/include/python3.9 -c'
289 extra options: '-funroll-all-loops -msse -msse2 -msse3'
290 INFO: g++: numexpr/interpreter.cpp
291 INFO: g++: numexpr/module.cpp
292 INFO: g++: numexpr/numexpr_object.cpp
293 numexpr/interpreter.cpp: In function 'PyObject* NumExpr_run(NumExprObject*, PyObject*, PyObject*)':
>> 294 numexpr/interpreter.cpp:1272:59: error: 'NPY_ARRAY_UPDATEIFCOPY' was not declared in this scope; did you mean 'NPY_ITER_UPDATEIFCOPY'?
295 1272 | NPY_ARRAY_ALIGNED|NPY_ARRAY_UPDATEIFCOPY);
296 | ^~~~~~~~~~~~~~~~~~~~~~
297 | NPY_ITER_UPDATEIFCOPY
>> 298 error: Command "/vast/home/treddy/github_projects/spack/lib/spack/env/gcc/g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/vast/home/treddy/github_projects/spack/opt/spack/linux-centos8-skylake_avx512/gcc-9.4.0/py-numpy-1.28.0-6jc4hgb7r7umhyjaltfgjkzv6tl5x4dv/lib/pytho
n3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/vast/home/treddy/github_projects/spack/opt/spack/linux-centos8-skylake_avx512/gcc-9.4.0/python-3.9.9-dpafrcjrc3ajwmfwm6jpvrfn2fuolebn/include/python3.9 -c numexpr/interpreter.cpp -o build/temp.linux-x86_64-3.9
/numexpr/interpreter.o -MMD -MF build/temp.linux-x86_64-3.9/numexpr/interpreter.o.d -funroll-all-loops -msse -msse2 -msse3" failed with exit status 1
299 INFO:
300 ########### EXT COMPILER OPTIMIZATION ###########
301 INFO: Platform :
302 Architecture: x64
303 Compiler : gcc
304
|
That error is because we removed I was wondering if we should just keep the symbol around, at least for a bit, but I would hope that |
|
Thanks for testing Tyler! Guess we're good then once this lands in numpy
I don't have energy for it right now, but if we do the build system conversion later this year I'd like to rip it out. |
That looks like the problem that occurs when you clone the main branch of the numpy git repo with |
5 participants
In SciPy we had a couple of cases where we build a Python extension with C source files but linked against static libraries built from Fortran code. Those should be using the Fortran linker, but this was broken in 1.22.0 by gh-19713 (the PR that introduced C++ in NumPy).
This fixes a few issues in the
build_extcommand, and documents better what is going on there.Should close SciPy issues scipy/scipy#8325 and scipy/scipy#15414.
In the build log, this should show up:
And then the resetting should actually happen. So not this:
But this: