Repository navigation
fortran extensions on OSX, issue with defining LDFLAGS #7427
Description
I ran into a problem while trying to build a conda package (https://github.com/praxes/hedm-conda-recipes/tree/master/imaged11) for a project with fortran extensions. python setup.py build works fine, but if`conda build`` simply invokes that same command, I get a string of errors about f2py not finding python symbols:
[...]
compiling Fortran sources
Fortran f77 compiler: /Users/darren/.conda/envs/_build/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -arch x86_64 -fPIC -O3 -funroll-loops
Fortran f90 compiler: /Users/darren/.conda/envs/_build/bin/gfortran -fopenmp -O2 -arch x86_64 -fPIC -O3 -funroll-loops
Fortran fix compiler: /Users/darren/.conda/envs/_build/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fopenmp -O2 -arch x86_64 -fPIC -O3 -funroll-loops
creating build/temp.macosx-10.5-x86_64-2.7/fsrc
compile options: '-Ibuild/src.macosx-10.5-x86_64-2.7 -I/Users/darren/.conda/envs/_build/lib/python2.7/site-packages/numpy/core/include -I/Users/darren/.conda/envs/_build/include/python2.7 -c'
gfortran:f90: fsrc/fImageD11.f90
/Users/darren/.conda/envs/_build/bin/gfortran -Wall -g -Wl,-rpath,/Users/darren/.conda/envs/_build/lib -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/build/src.macosx-10.5-x86_64-2.7/fImageD11module.o build/temp.macosx-10.5-x86_64-2.7/build/src.macosx-10.5-x86_64-2.7/fortranobject.o build/temp.macosx-10.5-x86_64-2.7/fsrc/fImageD11.o -L/Users/darren/.conda/envs/_build/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5 -L/Users/darren/.conda/envs/_build/lib -lgomp -lpthread -lgfortran -o build/lib.macosx-10.5-x86_64-2.7/ImageD11/fImageD11.so
Undefined symbols for architecture x86_64:
"_PyArg_ParseTupleAndKeywords", referenced from:
_f2py_rout_fImageD11_compute_xlylzl in fImageD11module.o
_f2py_rout_fImageD11_assign in fImageD11module.o
_f2py_rout_fImageD11_compute_gv in fImageD11module.o
"_PyCObject_AsVoidPtr", referenced from:
_initfImageD11 in fImageD11module.o
_F2PyCapsule_AsVoidPtr in fortranobject.o
"_PyCObject_FromVoidPtr", referenced from:
_fortran_getattr in fortranobject.o
_F2PyCapsule_FromVoidPtr in fortranobject.o
[...]
What I found was that conda build defines LDFLAGS= -Wl,-rpath,/Users/darren/.conda/envs/_build/lib -arch x86_64, and if I unset LDFLAGS in my conda recipe's build script, the package will compile. I asked about it on the conda mailing list, and someone replied that it appears f2py does not extend user-defined LDFLAGS, but instead let's user-defined LDFLAGS overwrite its own (https://lists.macosforge.org/pipermail/macports-users/2010-November/022574.html), which on osx include "-undefined dynamic_lookup -bundle". As it stands, it appears that anyone who needs to build a conda package on osx that includes fortran extensions will have to add the following to their build scripts:
LDFLAGS="$LDFLAGS -undefined dynamic_lookup -bundle"
Could f2py or numpy distutils extend user's LDFLAGS with these required settings?
Activity
rgommers commentedon Mar 17, 2016
This is known
numpy.distutilsbehavior for as long as I can remember.Some digging would be needed to figure out why it was chosen to implement things this way. I can imagine there are usecases where one needs to get rid of something that
numpy.distutilsordistutilsadds by default.ddale commentedon Mar 17, 2016
Maybe a warning would be in order then, advising that if one is going to define their own LDFLAGS, it is highly advisable to include x, y, and z.
isuruf commentedon Jul 7, 2018
This is a really annoying behaviour, we see in conda-forge. Would you accept a patch to fix this?
Appending is the normal behaviour for C extensions, so why not Fortran?
isuruf commentedon Jul 7, 2018
Also flags for
linker_soandlinker_exeare both overriden by the same flagLDFLAGS, which makes it impossible to have extra LDFLAGS since they need different sets of flags.rgommers commentedon Jul 7, 2018
I'm a little hesitant here, because it will break existing build automation for people who use
LDFLAGSnow. So this will need discussion on the mailing list.Question: is it just annoying, or does it actually prevent some things you want to do?
This seems possible to change without backwards compatibility issues, by introducing some new envvars.
pv commentedon Jul 7, 2018
isuruf commentedon Jul 7, 2018
At conda-forge, we set some LDFLAGS for external libraries to be found (-L$PREFIX/lib) and flags like
-headerpad_max_install_names. This is done for all packages using compilers, not just python packages. Flags in LDFLAGS env var is meant to be appended in every other build system including python's distutils, so that's why conda-forge sets it.rgommers commentedon Jul 7, 2018
That makes sense to me. Then we could add some noisy warnings if it's not set, and we'd be in no hurry to finally flip the switch, maybe leave that for 2.0 or 4-5 releases.
rainwoodman commentedon Dec 9, 2018
Is the noisy warning there yet?
I landed here seeing symbol errors running runtests.py using the conda-build compilers without setting this flag.
rgommers commentedon Dec 15, 2018
No. I missed that when merging the PR. I'll look at adding a warning now.
rgommers commentedon Dec 15, 2018
Done in gh-9537
charris commentedon Dec 16, 2018
Done in #12551 I think.
MAINT: add warning to numpy.distutils for LDFLAGS append behavior.
BLD: MAINT: change default behavior of build flag appending.