Skip to content

Pytorch 1 build from source fails with cudnn error #17572

@Morpheus3000

Description

@Morpheus3000

I am trying to install pytorch 1 from source on a server where I do not have sudo access. I have installed Anaconda and the required packages as instructed on the getting started page on the pytorch website. But it fails when compiling some caffe2 stuff. I can't seem to find what the error exactly is. At first I thought that it was because of cudnn, because I would see output like this:

[21/604] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/mpi/mpi_ops_gpu.cc.o
In file included from ../caffe2/core/context_gpu.h:20:0,
from ../caffe2/mpi/mpi_ops_gpu.cc:2:
../caffe2/core/common_cudnn.h:23:17: note: #pragma message: CUDNN version under 6.0 is supported at best effort.
#pragma message "CUDNN version under 6.0 is supported at best effort."
^
../caffe2/core/common_cudnn.h:24:17: note: #pragma message: We strongly encourage you to move to 6.0 and above.
#pragma message "We strongly encourage you to move to 6.0 and above."
^
../caffe2/core/common_cudnn.h:25:17: note: #pragma message: This message is intended to annoy you enough to update.
#pragma message "This message is intended to annoy you enough to update."
^
ninja: build stopped: subcommand failed.

But the CMake config output confirms that cudnn 7 is installed and detected, along with cuda9.1 Anyone faced this issue before, and knows how to fix this?

The full log is as follows:

Building wheel torch-1.1.0a0+13e6326
-- Building version 1.1.0a0+13e6326
[0/1] Re-running CMake...
-- std::exception_ptr is supported.
-- NUMA is available
-- Turning off deprecation warning due to glog.
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/var/scratch/pdas/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- MKL libraries: /var/node433/local/pdas/anaconda3/lib/libmkl_intel_lp64.so;/var/node433/local/pdas/anaconda3/lib/libmkl_gnu_thread.so;/var/node433/local/pdas/anaconda3/lib/libmkl_core.so;-fopenmp;/usr/lib64/libpthread.so;/usr/lib64/lib
m.so;/usr/lib64/libdl.so
-- MKL include directory: /var/node433/local/pdas/anaconda3/include
-- MKL OpenMP type: GNU
-- MKL OpenMP library: -fopenmp
-- Brace yourself, we are building NNPACK
-- Found PythonInterp: /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/bin/python (found version "3.7.1")
-- Failed to find LLVM FileCheck
-- git Version: v1.4.0-505be96a
-- Version: 1.4.0
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK -- success
-- [asmjit]
BuildMode=Static
BuildTest=Off
ASMJIT_DIR=/var/scratch/pdas/pytorch/build/third_party/fbgemm/third_party/asmjit
ASMJIT_DEPS=pthread;rt
ASMJIT_LIBS=asmjit;pthread;rt
ASMJIT_CFLAGS=-DASMJIT_STATIC
ASMJIT_SOURCE_DIR=/var/scratch/pdas/pytorch/build/third_party/fbgemm/third_party/asmjit/src
ASMJIT_INCLUDE_DIR=/var/scratch/pdas/pytorch/build/third_party/fbgemm/third_party/asmjit/src
ASMJIT_PRIVATE_CFLAGS=
-DASMJIT_STATIC
-std=c++17
-fno-tree-vectorize
-fvisibility=hidden
-O2 [RELEASE]
-fno-keep-static-consts [RELEASE]
-fmerge-all-constants [RELEASE]
-- Found Numa (include: /usr/include, library: /usr/lib64/libnuma.so)
-- Using third party subdirectory Eigen.
Python 3.7.1
-- Found PythonInterp: /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/bin/python (found suitable version "3.7.1", minimum required is "2.7")
-- Could NOT find pybind11 (missing: pybind11_DIR)
-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)
-- Using third_party/pybind11.
-- MPI support found
-- MPI compile flags: -pthread
-- MPI include path: /cm/shared/apps/openmpi/gcc/64/1.10.3/include
-- MPI LINK flags path: -Wl,-rpath -Wl,/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64 -Wl,--enable-new-dtags -pthread
-- MPI libraries: /cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi_cxx.so/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi.so
CMake Warning at cmake/Dependencies.cmake:665 (message):
OpenMPI found, but it is not built with CUDA support.
Call Stack (most recent call first):
CMakeLists.txt:224 (include)

-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /var/node433/local/pdas/anaconda3/lib/libgomp.so;/lib64/libpthread.so
CMake Warning (dev) at cmake/public/cuda.cmake:19 (find_package):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

/cm/shared/apps/cuda91/toolkit/9.1.85

For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
cmake/Dependencies.cmake:735 (include)
CMakeLists.txt:224 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Caffe2: CUDA detected: 9.1
-- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda91/toolkit/9.1.85/bin/nvcc
-- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda91/toolkit/9.1.85
-- Caffe2: Header version is: 9.1
-- Found cuDNN: v7.0.2 (include: /cm/shared/package/cuDNN/cuda90rc/7.0.2/cudnn/cuda/include, library: /cm/shared/package/cuDNN/cuda90rc/7.0.2/cudnn/cuda/lib64/libcudnn.so.7)
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.0;3.5;5.0;5.2;6.0;6.1;7.0;7.0+PTX
-- Added CUDA NVCC flags for: -gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,co
de=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70
-- Could NOT find CUB (missing: CUB_INCLUDE_DIR)
-- MPI include path: /cm/shared/apps/openmpi/gcc/64/1.10.3/include
-- MPI libraries: /cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi_cxx.so/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi.so
CMake Warning (dev) at third_party/gloo/cmake/Cuda.cmake:133 (find_package):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

/cm/shared/apps/cuda91/toolkit/9.1.85

For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
third_party/gloo/cmake/Dependencies.cmake:44 (include)
third_party/gloo/CMakeLists.txt:52 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda91/toolkit/9.1.85 (found suitable version "9.1", minimum required is "7.0")
-- CUDA detected: 9.1

-- ******** Summary ********
-- CMake version : 3.12.2
-- CMake command : /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/bin/cmake
-- System : Linux
-- C++ compiler : /var/node433/local/pdas/anaconda3/bin/c++
-- C++ compiler version : 5.2.0
-- CXX flags : -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -Wnon-virtual-dtor
-- Build type : Release
-- Compile definitions : TH_BLAS_MKL
-- CMAKE_PREFIX_PATH : /var/node433/local/pdas/anaconda3/condabin/../
-- CMAKE_INSTALL_PREFIX : /var/scratch/pdas/pytorch/torch
-- CMAKE_MODULE_PATH : /var/scratch/pdas/pytorch/cmake/Modules;/var/scratch/pdas/pytorch/cmake/public/../Modules_CUDA_fix

-- ONNX version : 1.4.1
-- ONNX NAMESPACE : onnx_torch
-- ONNX_BUILD_TESTS : OFF
-- ONNX_BUILD_BENCHMARKS : OFF
-- ONNX_USE_LITE_PROTO : OFF
-- ONNXIFI_DUMMY_BACKEND : OFF [1043/1953]
-- ONNXIFI_ENABLE_EXT : OFF

-- Protobuf compiler :
-- Protobuf includes :
-- Protobuf libraries :
-- BUILD_ONNX_PYTHON : OFF

-- ******** Summary ********
-- CMake version : 3.12.2
-- CMake command : /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/bin/cmake
-- System : Linux
-- C++ compiler : /var/node433/local/pdas/anaconda3/bin/c++
-- C++ compiler version : 5.2.0
-- CXX flags : -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -Wnon-virtual-dtor
-- Build type : Release
-- Compile definitions : TH_BLAS_MKL
-- CMAKE_PREFIX_PATH : /var/node433/local/pdas/anaconda3/condabin/../
-- CMAKE_INSTALL_PREFIX : /var/scratch/pdas/pytorch/torch
-- CMAKE_MODULE_PATH : /var/scratch/pdas/pytorch/cmake/Modules;/var/scratch/pdas/pytorch/cmake/public/../Modules_CUDA_fix

-- ONNX version : 1.4.1
-- ONNX NAMESPACE : onnx_torch
-- ONNX_BUILD_TESTS : OFF
-- ONNX_BUILD_BENCHMARKS : OFF
-- ONNX_USE_LITE_PROTO : OFF
-- ONNXIFI_DUMMY_BACKEND : OFF

-- Protobuf compiler :
-- Protobuf includes :
-- Protobuf libraries :
-- BUILD_ONNX_PYTHON : OFF
-- Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor
-- Removing -DNDEBUG from compile flags
-- Compiling with MAGMA support
-- MAGMA INCLUDE DIRECTORIES: /var/node433/local/pdas/anaconda3/include
-- MAGMA LIBRARIES: /var/node433/local/pdas/anaconda3/lib/libmagma.a
-- MAGMA V2 check: 1
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- AVX compiler support found
-- AVX2 compiler support found
-- Atomics: using GCC intrinsics
-- Found a library with BLAS API (mkl).
-- Found a library with LAPACK API (mkl).
disabling ROCM because NOT USE_ROCM is set
-- MIOpen not found. Compiling without MIOpen support
-- OpenMP lib: provided by compiler
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- VTune profiling environment is unset
-- Found MKL-DNN: TRUE
-- GCC 5.2.0: Adding gcc and gcc_s libs to link line
-- NUMA paths:
-- /usr/include
-- /usr/lib64/libnuma.so
-- Configuring build for SLEEF-v3.2
Target system: Linux-3.10.0-693.17.1.el7.x86_64
Target processor: x86_64
Host system: Linux-3.10.0-693.17.1.el7.x86_64
Host processor: x86_64
Detected C compiler: GNU @ /var/node433/local/pdas/anaconda3/bin/cc
-- Using option -Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math to compile libsleef
-- Building shared libs : OFF
-- MPFR : /var/node433/local/pdas/anaconda3/lib/libmpfr.so
-- MPFR header file in /var/node433/local/pdas/anaconda3/include
-- MPFR header file in /var/node433/local/pdas/anaconda3/include [978/1953]
-- GMP : /var/node433/local/pdas/anaconda3/lib/libgmp.so
-- RUNNING_ON_TRAVIS : 0
-- COMPILER_SUPPORTS_OPENMP : 1
-- pytorch is compiling with OpenMP.
OpenMP CXX_FLAGS: -fopenmp.
OpenMP libraries: /var/node433/local/pdas/anaconda3/lib/libgomp.so;/lib64/libpthread.so.
-- /var/node433/local/pdas/anaconda3/bin/c++ /var/scratch/pdas/pytorch/torch/abi-check.cpp -o /var/scratch/pdas/pytorch/build/abi-check
-- Determined _GLIBCXX_USE_CXX11_ABI=1
CMake Warning (dev) at torch/lib/THD/CMakeLists.txt:50 (FIND_PACKAGE):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

/cm/shared/apps/cuda91/toolkit/9.1.85

For compatibility, CMake is ignoring the variable.
This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda91/toolkit/9.1.85 (found suitable version "9.1", minimum required is "7.5")
-- MPI_LIBRARIES: /cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi_cxx.so;/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi.so
-- Building the gloo backend with TCP support only
-- MPI_COMPILE_FLAGS: -pthread
-- MPI_LINK_FLAGS: -Wl,-rpath -Wl,/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64 -Wl,--enable-new-dtags -pthread
CMake Warning (dev) at torch/lib/c10d/CMakeLists.txt:10 (find_package):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

/cm/shared/apps/cuda91/toolkit/9.1.85

For compatibility, CMake is ignoring the variable.
This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda91/toolkit/9.1.85 (found version "9.1")
-- Building C10D with CUDA support
-- MPI_INCLUDE_PATH: /cm/shared/apps/openmpi/gcc/64/1.10.3/include
-- MPI_LIBRARIES: /cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi_cxx.so;/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi.so
-- MPIEXEC: /cm/shared/apps/openmpi/gcc/64/1.10.3/bin/mpiexec
-- Include NCCL operators
-- Including IDEEP operators
-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv
-- Include Observer library
-- Caffe2 is compiling with OpenMP.
OpenMP CXX_FLAGS: -fopenmp.
OpenMP libraries: /var/node433/local/pdas/anaconda3/lib/libgomp.so;/lib64/libpthread.so.
-- Using lib/python3.7/site-packages as python relative installation path
CMake Warning at CMakeLists.txt:421 (message):
Generated cmake files are only fully tested if one builds with system glog,
gflags, and protobuf. Other settings may generate files that are not well
tested.

--
-- ******** Summary ********
-- General:
-- CMake version : 3.12.2
-- CMake command : /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/bin/cmake
-- System : Linux
-- C++ compiler : /var/node433/local/pdas/anaconda3/bin/c++
-- C++ compiler id : GNU
-- C++ compiler version : 5.2.0 [912/1953]
-- BLAS : MKL
-- CXX flags : -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-com
pare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast
-Wno-unused-but-set-variable -Wno-maybe-uninitialized
-- Build type : Release
-- Compile definitions : TH_BLAS_MKL;ONNX_NAMESPACE=onnx_torch;MAGMA_V2;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
-- CMAKE_PREFIX_PATH : /var/node433/local/pdas/anaconda3/condabin/../
-- CMAKE_INSTALL_PREFIX : /var/scratch/pdas/pytorch/torch

-- TORCH_VERSION : 1.1.0
-- CAFFE2_VERSION : 1.1.0
-- BUILD_ATEN_MOBILE : OFF
-- BUILD_ATEN_ONLY : OFF
-- BUILD_BINARY : False
-- BUILD_CUSTOM_PROTOBUF : ON
-- Link local protobuf : ON
-- BUILD_DOCS : OFF
-- BUILD_PYTHON : True
-- Python version : 3.7.1
-- Python executable : /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/bin/python
-- Pythonlibs version : 3.7.1
-- Python library : /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/lib/libpython3.7m.so.1.0
-- Python includes : /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/include/python3.7m
-- Python site-packages: lib/python3.7/site-packages
-- BUILD_CAFFE2_OPS : True
-- BUILD_SHARED_LIBS : ON
-- BUILD_TEST : True
-- USE_ASAN : OFF
-- USE_CUDA : True
-- CUDA static link : False
-- USE_CUDNN : ON
-- CUDA version : 9.1
-- cuDNN version : 7.0.2
-- CUDA root directory : /cm/shared/apps/cuda91/toolkit/9.1.85
-- CUDA library : /cm/shared/apps/cuda91/toolkit/9.1.85/lib64/stubs/libcuda.so
-- cudart library : /cm/shared/apps/cuda91/toolkit/9.1.85/lib64/libcudart.so
-- cublas library : /cm/shared/apps/cuda91/toolkit/9.1.85/lib64/libcublas.so;/cm/shared/apps/cuda91/toolkit/9.1.85/lib64/libcublas_device.a
-- cufft library : /cm/shared/apps/cuda91/toolkit/9.1.85/lib64/libcufft.so
-- curand library : /cm/shared/apps/cuda91/toolkit/9.1.85/lib64/libcurand.so
-- cuDNN library : /cm/shared/package/cuDNN/cuda90rc/7.0.2/cudnn/cuda/lib64/libcudnn.so.7
-- nvrtc : /var/node433/local/pdas/anaconda3/lib/libnvrtc.so
-- CUDA include path : /cm/shared/apps/cuda91/toolkit/9.1.85/include
-- NVCC executable : /cm/shared/apps/cuda91/toolkit/9.1.85/bin/nvcc
-- CUDA host compiler : /var/node433/local/pdas/anaconda3/bin/cc
-- USE_TENSORRT : OFF
-- USE_ROCM : False
-- USE_EIGEN_FOR_BLAS :
-- USE_FBGEMM : ON
-- USE_FFMPEG : False
-- USE_GFLAGS : OFF
-- USE_GLOG : OFF
-- USE_LEVELDB : False
-- USE_LITE_PROTO : OFF
-- USE_LMDB : False
-- USE_METAL : OFF
-- USE_MKL : ON
-- USE_MKLDNN : ON
-- USE_NCCL : True
-- USE_SYSTEM_NCCL : False
-- USE_NNPACK : True
-- USE_NUMPY : ON
-- USE_OBSERVERS : ON
-- USE_OPENCL : OFF
-- USE_OPENCV : False
-- USE_OPENMP : ON
-- USE_OPENMP : ON [847/1953]
-- USE_PROF : OFF
-- USE_QNNPACK : True
-- USE_REDIS : OFF
-- USE_ROCKSDB : OFF
-- USE_ZMQ : OFF
-- USE_DISTRIBUTED : True
-- USE_MPI : ON
-- USE_GLOO : ON
-- USE_GLOO_IBVERBS : OFF
-- Public Dependencies : Threads::Threads;caffe2::mkl;caffe2::mkldnn
-- Private Dependencies : qnnpack;nnpack;cpuinfo;fbgemm;/usr/lib64/libnuma.so;fp16;/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi_cxx.so;/cm/shared/apps/openmpi/gcc/64/1.10.3/lib64/libmpi.so;gloo;aten_op_header_gen;foxi_loader;rt;g
cc_s;gcc;dl
-- Configuring done
-- Generating done
-- Build files have been written to: /var/scratch/pdas/pytorch/build
[4/604] Building CXX object caffe2/CMakeFiles/cuda_cudnn_test.dir//aten/src/ATen/test/cuda_cudnn_test.cpp.o
FAILED: caffe2/CMakeFiles/cuda_cudnn_test.dir/
/aten/src/ATen/test/cuda_cudnn_test.cpp.o
/var/node433/local/pdas/anaconda3/bin/c++ -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMAGMA_V2 -DONNX_NAMESPACE=onnx_torch -DTH_BLAS_MKL -DUSE_GCC_ATOMICS=1 -D_FILE_OFFSET_BITS=64 -I../aten/src -I.
-I../ -I../third_party/protobuf/src -I../cmake/../third_party/benchmark/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -Icaffe2/aten/src/TH -I../aten/src/TH -Icaffe2/aten/s
rc -Iaten/src -I../aten/../third_party/catch/single_include -I../aten/src/ATen/.. -Icaffe2/aten/src/ATen -I../caffe2/core/nomnigraph/include -I../third_party/miniz-2.0.8 -I../c10/.. -I../c10/cuda/../.. -isystem /var/node433/local/pdas/an
aconda3/include -isystem ../third_party/ideep/include -isystem ../third_party/ideep/mkl-dnn/include -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -isystem ../third_party/gemmlowp -isystem ../third_party/neon2sse -isyste
m ../cmake/../third_party/googletest/googletest/include -isystem ../third_party -isystem ../cmake/../third_party/eigen -isystem /var/node433/local/pdas/anaconda3/envs/sourcePytorch1/include/python3.7m -isystem /var/node433/local/pdas/ana
conda3/envs/sourcePytorch1/lib/python3.7/site-packages/numpy/core/include -isystem ../cmake/../third_party/pybind11/include -isystem /cm/shared/apps/openmpi/gcc/64/1.10.3/include -isystem ../cmake/../third_party/cub -isystem /cm/shared/a
pps/cuda91/toolkit/9.1.85/include -isystem /cm/shared/package/cuDNN/cuda90rc/7.0.2/cudnn/cuda/include -isystem ../third_party/googletest/googletest/include -isystem ../third_party/googletest/googletest -Wno-deprecated -fvisibility-inline
s-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-functi
on -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized -DHAVE_AVX_
CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -fPIE -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -std=c++11 -pthread -std=gnu++11 -MD -MT caffe2/CMakeFiles/cuda_cudnn_test.dir//aten/
src/ATen/test/cuda_cudnn_test.cpp.o -MF caffe2/CMakeFiles/cuda_cudnn_test.dir/
/aten/src/ATen/test/cuda_cudnn_test.cpp.o.d -o caffe2/CMakeFiles/cuda_cudnn_test.dir/__/aten/src/ATen/test/cuda_cudnn_test.cpp.o -c ../aten/src/ATen/test/cud
a_cudnn_test.cpp
In file included from ../aten/src/ATen/cudnn/Descriptors.h:6:0,
from ../aten/src/ATen/test/cuda_cudnn_test.cpp:5:
../aten/src/ATen/cudnn/cudnn-wrapper.h:9:198: note: #pragma message: CuDNN v5 found, but need at least CuDNN v6. You can get the latest version of CuDNN from https://developer.nvidia.com/cudnn or disable CuDNN with NO_CUDNN=1
#pragma message ("CuDNN v" STRING(CUDNN_MAJOR) " found, but need at least CuDNN v6. You can get the latest version of CuDNN from https://developer.nvidia.com/cudnn or disable CuDNN with NO_CUDNN=1")
^
../aten/src/ATen/cudnn/cudnn-wrapper.h:10:17: note: #pragma message: We strongly encourage you to move to 6.0 and above.
#pragma message "We strongly encourage you to move to 6.0 and above."
^
../aten/src/ATen/cudnn/cudnn-wrapper.h:11:17: note: #pragma message: This message is intended to annoy you enough to update.
#pragma message "This message is intended to annoy you enough to update."
^
In file included from ../aten/src/ATen/cuda/CUDAContext.h:6:0,
from ../aten/src/ATen/test/cuda_cudnn_test.cpp:4:
../aten/src/ATen/cudnn/Descriptors.h: In member function ‘void at::native::ConvolutionDescriptor::set(cudnnDataType_t, int, int*, int*, int*, int)’:
../aten/src/ATen/cudnn/Descriptors.h:157:68: error: ‘cudnnSetConvolutionGroupCount’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetConvolutionGroupCount(mut_desc(), groups));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
../aten/src/ATen/cudnn/Descriptors.h:159:60: error: ‘CUDNN_DEFAULT_MATH’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetConvolutionMathType(mut_desc(), CUDNN_DEFAULT_MATH));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
../aten/src/ATen/cudnn/Descriptors.h:159:78: error: ‘cudnnSetConvolutionMathType’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetConvolutionMathType(mut_desc(), CUDNN_DEFAULT_MATH));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
../aten/src/ATen/cudnn/Descriptors.h:161:62: error: ‘CUDNN_TENSOR_OP_MATH’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetConvolutionMathType(mut_desc(), CUDNN_TENSOR_OP_MATH));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR; \ [795/1967]
^
../aten/src/ATen/cudnn/Descriptors.h:161:82: error: ‘cudnnSetConvolutionMathType’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetConvolutionMathType(mut_desc(), CUDNN_TENSOR_OP_MATH));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
../aten/src/ATen/cudnn/Descriptors.h: In member function ‘void at::native::DropoutDescriptor::set(cudnnHandle_t, float, at::Tensor)’:
../aten/src/ATen/cudnn/Descriptors.h:204:114: error: ‘cudnnRestoreDropoutDescriptor’ was not declared in this scope
AT_CUDNN_CHECK(cudnnRestoreDropoutDescriptor(mut_desc(), handle, dropout, state_ptr, state_size, 0 /* seed /));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
In file included from ../aten/src/ATen/test/cuda_cudnn_test.cpp:5:0:
../aten/src/ATen/cudnn/Descriptors.h: At global scope:
../aten/src/ATen/cudnn/Descriptors.h:226:87: error: ‘cudnnRNNAlgo_t’ has not been declared
cudnnRNNMode_t mode, cudnnDataType_t datatype, cudnnDataType_t input_type, cudnnRNNAlgo_t algo) {
^
In file included from ../aten/src/ATen/cuda/CUDAContext.h:6:0,
from ../aten/src/ATen/test/cuda_cudnn_test.cpp:4:
../aten/src/ATen/cudnn/Descriptors.h: In member function ‘void at::native::RNNDescriptor::set(cudnnHandle_t, int, int, at::native::DropoutDescriptor&&, cudnnRNNInputMode_t, cudnnDirectionMode_t, cudnnRNNMode_t, cudnnDataType_t, cudnnData
Type_t, int)’:
../aten/src/ATen/cudnn/Descriptors.h:238:19: error: ‘cudnnSetRNNDescriptor_v6’ was not declared in this scope
datatype));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
In file included from ../aten/src/ATen/test/cuda_cudnn_test.cpp:5:0:
../aten/src/ATen/cudnn/Descriptors.h: At global scope:
../aten/src/ATen/cudnn/Descriptors.h:255:23: error: ‘cudnnCTCLossStruct’ was not declared in this scope
: public Descriptor<cudnnCTCLossStruct,
^
../aten/src/ATen/cudnn/Descriptors.h:257:53: error: wrong number of template arguments (2, should be 3)
&cudnnDestroyCTCLossDescriptor>
^
../aten/src/ATen/cudnn/Descriptors.h:66:19: note: provided for ‘template<class T, cudnnStatus_t (
ctor)(T**), cudnnStatus_t (* dtor)(T*)> class at::native::Descriptor’
class AT_CUDA_API Descriptor
^
In file included from ../aten/src/ATen/cuda/CUDAContext.h:6:0,
from ../aten/src/ATen/test/cuda_cudnn_test.cpp:4:
../aten/src/ATen/cudnn/Descriptors.h: In member function ‘void at::native::CTCLossDescriptor::set(cudnnDataType_t)’:
../aten/src/ATen/cudnn/Descriptors.h:260:55: error: ‘mut_desc’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetCTCLossDescriptor(mut_desc(), datatype));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
../aten/src/ATen/cudnn/Descriptors.h:260:66: error: ‘cudnnSetCTCLossDescriptor’ was not declared in this scope
AT_CUDNN_CHECK(cudnnSetCTCLossDescriptor(mut_desc(), datatype));
^
../aten/src/ATen/cuda/Exceptions.h:9:28: note: in definition of macro ‘AT_CUDNN_CHECK’
cudnnStatus_t status = EXPR;
^
[6/604] Building CXX object caffe2/CMakeFiles/event_gpu_test.dir/core/event_gpu_test.cc.o
In file included from ../caffe2/core/context_gpu.h:20:0,
from ../caffe2/core/event_gpu_test.cc:3:
../caffe2/core/common_cudnn.h:23:17: note: #pragma message: CUDNN version under 6.0 is supported at best effort.
#pragma message "CUDNN version under 6.0 is supported at best effort."
^
../caffe2/core/common_cudnn.h:24:17: note: #pragma message: We strongly encourage you to move to 6.0 and above.
#pragma message "We strongly encourage you to move to 6.0 and above."
^
../caffe2/core/common_cudnn.h:25:17: note: #pragma message: This message is intended to annoy you enough to update.
#pragma message "This message is intended to annoy you enough to update." [729/1967]
^
[20/604] Building CXX object caffe2/CMakeFiles/roi_align_op_gpu_test.dir/operators/roi_align_op_gpu_test.cc.o
In file included from ../caffe2/core/context_gpu.h:20:0,
from ../caffe2/operators/roi_align_op_gpu_test.cc:4:
../caffe2/core/common_cudnn.h:23:17: note: #pragma message: CUDNN version under 6.0 is supported at best effort.
#pragma message "CUDNN version under 6.0 is supported at best effort."
^
../caffe2/core/common_cudnn.h:24:17: note: #pragma message: We strongly encourage you to move to 6.0 and above.
#pragma message "We strongly encourage you to move to 6.0 and above."
^
../caffe2/core/common_cudnn.h:25:17: note: #pragma message: This message is intended to annoy you enough to update.
#pragma message "This message is intended to annoy you enough to update."
^
[21/604] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/mpi/mpi_ops_gpu.cc.o
In file included from ../caffe2/core/context_gpu.h:20:0,
from ../caffe2/mpi/mpi_ops_gpu.cc:2:
../caffe2/core/common_cudnn.h:23:17: note: #pragma message: CUDNN version under 6.0 is supported at best effort.
#pragma message "CUDNN version under 6.0 is supported at best effort."
^
../caffe2/core/common_cudnn.h:24:17: note: #pragma message: We strongly encourage you to move to 6.0 and above.
#pragma message "We strongly encourage you to move to 6.0 and above."
^
../caffe2/core/common_cudnn.h:25:17: note: #pragma message: This message is intended to annoy you enough to update.
#pragma message "This message is intended to annoy you enough to update."
^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "setup.py", line 710, in
build_deps()
File "setup.py", line 282, in build_deps
build_dir='build')
File "/var/scratch/pdas/pytorch/tools/build_pytorch_libs.py", line 252, in build_caffe2
check_call(ninja_cmd, cwd=build_dir, env=my_env)
File "/var/node433/local/pdas/anaconda3/envs/sourcePytorch1/lib/python3.7/subprocess.py", line 341, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja', 'install']' returned non-zero exit status 1.

Metadata

Metadata

Labels

module: buildBuild system issuestriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions