Unable to build with Spack/from source, ffmpeg libraries not found #2591
Description
🐛 Bug
I'm trying to install torchvision from source using the Spack package manager, but the build can't find my ffmpeg libraries, which are in a non-standard location. There are a couple of reasons for this:
- Mismatch of compilers. At some points of the build, it is using the compiler I defined in
CXX, which is a compiler wrapper designed to automatically inject the appropriate linker args. At other points of the build, it is using the system compiler, possibly picked up from Python's distutils, which is not a compiler wrapper. TORCHVISION_LIBRARYseems to be ignored. The README says to setTORCHVISION_LIBRARYwith the path to all dependency libraries, but this isn't working either.
To Reproduce
$ spack install -v py-torchvision
...
[15/15] /Users/Adam/spack/lib/spack/env/clang/clang++ -MMD -MF /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/video_reader/VideoReader.o.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder -I/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/video_reader -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/ffmpeg-4.2.2-hoilb6sa2j6ggatrb2xvmuazqrtgcoa7/include -I/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/include -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/include/TH -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/include/THC -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/python-3.7.8-skt55xepunmc5rj2diqoljahpau3zwrm/include/python3.7m -c -c /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/video_reader/VideoReader.cpp -o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/video_reader/VideoReader.o -std=c++14 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=video_reader -D_GLIBCXX_USE_CXX11_ABI=0
...
/usr/bin/clang++ -bundle -undefined dynamic_lookup /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/video_reader/VideoReader.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/seekable_buffer.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/cc_stream.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/util.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/subtitle_sampler.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/audio_stream.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/subtitle_stream.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/audio_sampler.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/stream.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/time_keeper.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/memory_buffer.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/decoder.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/sync_decoder.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/video_stream.o /private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/build/temp.macosx-10.15.6-x86_64-3.7/private/var/folders/21/hwq39zyj4g36x6zjfyl5l8080000gn/T/Adam/spack-stage/spack-stage-py-torchvision-0.7.0-732klihrqrcpdkrv4mispep23skf2cbr/spack-src/torchvision/csrc/cpu/decoder/video_sampler.o -L/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/lib -L/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/python-3.7.8-skt55xepunmc5rj2diqoljahpau3zwrm/lib -lavcodec -lavformat -lavutil -lswresample -lswscale -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.macosx-10.15.6-x86_64-3.7/torchvision/video_reader.so -std=c++14
ld: library not found for -lavcodec
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang++' failed with exit status 1As you can see, at some points of the build, it is using the compiler wrapper /Users/Adam/spack/lib/spack/env/clang/clang++, while at other parts it's using the actual compiler /usr/bin/clang++. This is true whether or not I use the ninja backend for building.
Note that I have a few local patches to Spack's torchvision recipe to add TORCHVISION_LIBRARY and TORCHVISION_INCLUDE, see below.
Expected behavior
If CXX is set to a particular compiler, I would expect the build to use that compiler for all compilation. Secondly, if TORCHVISION_LIBRARY is mentioned in the README as a way to locate libraries, I would expect that to work.
Environment
- PyTorch / torchvision Version: 1.6.0 / 0.7.0 (also reproduced with 1.5.1 / 0.6.1)
- OS: macOS 10.15.6
- Compiler: Apple Clang 11.0.3
- How you installed PyTorch / torchvision:
spack(source) - Build command you used (if compiling from source):
python setup.py build && python setup.py install - Python version: 3.7.8
- CUDA/cuDNN version: N/A
- GPU models and configuration: N/A
- Any other relevant information: build environment contains the following env vars:
TORCHVISION_INCLUDE=/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/python-3.7.8-skt55xepunmc5rj2diqoljahpau3zwrm/include/python3.7m:/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/include:/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/ffmpeg-4.2.2-hoilb6sa2j6ggatrb2xvmuazqrtgcoa7/include
TORCHVISION_LIBRARY=/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/python-3.7.8-skt55xepunmc5rj2diqoljahpau3zwrm/lib:/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/py-torch-1.6.0-7zhzk2mza5nl3hvalsoql43dq56p7vs6/lib/python3.7/site-packages/torch/lib:/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-11.0.3/ffmpeg-4.2.2-hoilb6sa2j6ggatrb2xvmuazqrtgcoa7/libAdditional context
I realize that you likely don't get a lot of bug reports from people installing torchvision with Spack. I'm a Spack developer, so I'm happy to handle all of the Spack-specific questions. Also, if any torch/torchvision developers would like to help me maintain our Spack recipes, let me know!