Skip to content

NCCL PyCaffe build errors #5541

@antran89

Description

@antran89

Issue summary

Building latest PyCaffe with NCCL not successful.

Steps to reproduce

CMakeCache.txt file: CMakeCache.txt

cmake -D USE_NCCL=ON ..

Output:

tranlaman@VIML2:~/Desktop/caffe-workspace/BLVC-caffe/cmake-build$ make all -j8
-- Boost version: 1.55.0
-- Found the following Boost libraries:
--   system
--   thread
--   filesystem
-- Found gflags  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
-- Found glog    (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
-- Found PROTOBUF Compiler: /usr/bin/protoc
-- Found lmdb    (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
-- Found Snappy  (include: /usr/include, library: /usr/lib/libsnappy.so)
-- CUDA detected: 8.0
-- Found cuDNN: ver. 5.1.5 found (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Added CUDA NVCC flags for: sm_61
-- Cuda + Boost 1.55: Applying noinline work around
-- Found NCCL    (include: /usr/local/include, library: /usr/local/lib/libnccl.so)
-- OpenCV found (/usr/share/OpenCV)
-- Found Atlas (include: /usr/include library: /usr/lib/libatlas.so lapack: /usr/lib/liblapack.so
-- NumPy ver. 1.11.1 found (include: /usr/local/lib/python2.7/dist-packages/numpy/core/include)
-- Boost version: 1.55.0
-- Found the following Boost libraries:
--   python
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE) 
-- 
-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0
--   Git               :   1.0-dirty
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
-- 
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   USE_NCCL          :   ON
--   ALLOW_LMDB_NOLOCK :   OFF
-- 
-- Dependencies:
--   BLAS              :   Yes (Atlas)
--   Boost             :   Yes (ver. 1.55)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 2.5.0)
--   lmdb              :   Yes (ver. 0.9.10)
--   LevelDB           :   Yes (ver. 1.15)
--   Snappy            :   Yes (ver. 1.1.0)
--   OpenCV            :   Yes (ver. 2.4.8)
--   CUDA              :   Yes (ver. 8.0)
-- 
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_61
--   cuDNN             :   Yes (ver. 5.1.5)
-- 
-- Python:
--   Interpreter       :   /usr/bin/python2.7 (ver. 2.7.6)
--   Libraries         :   /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.6)
--   NumPy             :   /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.1)
-- 
-- Documentaion:
--   Doxygen           :   No
--   config_file       :   
-- 
-- Install:
--   Install path      :   /home/tranlaman/Desktop/caffe-workspace/BLVC-caffe/cmake-build/install
-- 
CMake Warning (dev) in src/caffe/CMakeLists.txt:
  Policy CMP0022 is not set: INTERFACE_LINK_LIBRARIES defines the link
  interface.  Run "cmake --help-policy CMP0022" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "caffe" has an INTERFACE_LINK_LIBRARIES property which differs from
  its LINK_INTERFACE_LIBRARIES properties.

  INTERFACE_LINK_LIBRARIES:

    proto;/usr/lib/x86_64-linux-gnu/libboost_system.so;/usr/lib/x86_64-linux-gnu/libboost_thread.so;/usr/lib/x86_64-linux-gnu/libboost_filesystem.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libglog.so;/usr/lib/x86_64-linux-gnu/libgflags.so;$<$<NOT:$<CONFIG:DEBUG>>:/usr/lib/x86_64-linux-gnu/libprotobuf.so>;$<$<CONFIG:DEBUG>:/usr/lib/x86_64-linux-gnu/libprotobuf.so>;-lpthread;/usr/lib/x86_64-linux-gnu/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/libhdf5.so;/usr/lib/x86_64-linux-gnu/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/libhdf5.so;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libleveldb.so;/usr/local/cuda/lib64/libcudart.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcudnn.so;/usr/local/lib/libnccl.so;opencv_core;opencv_highgui;opencv_imgproc;/usr/lib/liblapack.so;/usr/lib/libcblas.so;/usr/lib/libatlas.so;/usr/lib/x86_64-linux-gnu/libboost_python.so

  LINK_INTERFACE_LIBRARIES:

    proto;/usr/lib/x86_64-linux-gnu/libboost_system.so;/usr/lib/x86_64-linux-gnu/libboost_thread.so;/usr/lib/x86_64-linux-gnu/libboost_filesystem.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libglog.so;/usr/lib/x86_64-linux-gnu/libgflags.so;/usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread;/usr/lib/x86_64-linux-gnu/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/libhdf5.so;/usr/lib/x86_64-linux-gnu/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/libhdf5.so;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libleveldb.so;/usr/local/cuda/lib64/libcudart.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcudnn.so;/usr/local/lib/libnccl.so;opencv_core;opencv_highgui;opencv_imgproc;/usr/lib/liblapack.so;/usr/lib/libcblas.so;/usr/lib/libatlas.so;/usr/lib/x86_64-linux-gnu/libboost_python.so

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/tranlaman/Desktop/caffe-workspace/BLVC-caffe/cmake-build
[  1%] Built target proto
Scanning dependencies of target caffe
Linking CXX shared library ../../lib/libcaffe.so
[ 90%] Built target caffe
Linking CXX executable finetune_net
Linking CXX executable compute_image_mean
Linking CXX executable caffe
Linking CXX executable convert_imageset
Linking CXX executable extract_features
Linking CXX executable device_query
Linking CXX executable test_net
Linking CXX executable net_speed_benchmark
[ 91%] Built target finetune_net
[ 93%] Built target device_query
[ 93%] Built target net_speed_benchmark
Linking CXX executable upgrade_net_proto_binary
[ 93%] Linking CXX executable train_net
Built target test_net
Linking CXX executable upgrade_net_proto_text
Linking CXX executable upgrade_solver_proto_text
[ 93%] Built target train_net
Linking CXX executable cpp_classification/classification
[ 93%] Built target caffe.bin
Linking CXX executable cifar10/convert_cifar_data
[ 94%] Built target compute_image_mean
[ 95%] Built target convert_imageset
Linking CXX executable mnist/convert_mnist_data
Linking CXX executable siamese/convert_mnist_siamese_data
[ 95%] Built target extract_features
[ 97%] Building CXX object python/CMakeFiles/pycaffe.dir/caffe/_caffe.cpp.o
[ 98%] [ 98%] Built target upgrade_net_proto_binary
Built target upgrade_net_proto_text
[100%] Built target upgrade_solver_proto_text
[100%] Built target classification
[100%] Built target convert_cifar_data
[100%] Built target convert_mnist_siamese_data
[100%] Built target convert_mnist_data
/home/tranlaman/Desktop/caffe-workspace/BLVC-caffe/python/caffe/_caffe.cpp: In function ‘boost::python::api::object caffe::NCCL_New_Uid()’:
/home/tranlaman/Desktop/caffe-workspace/BLVC-caffe/python/caffe/_caffe.cpp:374:10: error: could not convert ‘uid’ from ‘std::string {aka std::basic_string<char>}’ to ‘boost::python::api::object’
   return uid;
          ^
/home/tranlaman/Desktop/caffe-workspace/BLVC-caffe/python/caffe/_caffe.cpp:376:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
make[2]: *** [python/CMakeFiles/pycaffe.dir/caffe/_caffe.cpp.o] Error 1
make[1]: *** [python/CMakeFiles/pycaffe.dir/all] Error 2
make: *** [all] Error 2

If you are having difficulty building Caffe or training a model, please ask the caffe-users mailing list. If you are reporting a build error that seems to be due to a bug in Caffe, please attach your build configuration (either Makefile.config or CMakeCache.txt) and the output of the make (or cmake) command.

Your system configuration

Operating system: Ubuntu 14.04
Compiler: gcc484
CUDA version (if applicable): 8.0
CUDNN version (if applicable): 5.0
BLAS:
Python or MATLAB version (for pycaffe and matcaffe respectively):2.7

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions