Cannot import the Cpp/CUDA extensions in NVIDIA pytorch docker containers #2350
Description
Describe the bug
I cannot import the MONAI Cpp/CUDA extensions using last NVIDIA pytorch docker containers. I tried with nvcr.io/nvidia/pytorch:21.02-py3, nvcr.io/nvidia/pytorch:21.03-py3,nvcr.io/nvidia/pytorch:21.04-py3,nvcr.io/nvidia/pytorch:21.05-py3.
To Reproduce
Steps to reproduce the behavior:
- Pull the docker container, e.g.
nvcr.io/nvidia/pytorch:21.04-py3 - Install MONAI:
BUILD_MONAI=1 pip -q install -U git+https://github.com/Project-MONAI/MONAI#egg=monai - Run commands
python -c 'import monai._C;'
Error message:
File "<string>", line 1, in <module>
ImportError: /workspace/home/locals/del_moi/.local/lib/python3.8/site-packages/monai/_C.so: undefined symbol: _ZN2at3eluERKNS_6TensorEN3c106ScalarES4_S4_
Expected behavior
Being able to import the Cpp/CUDA extensions.
Environment
python -c 'import monai; monai.config.print_debug_info()'
================================
Printing MONAI config...
================================
MONAI version: 0.5.3+101.gcbf2521
Numpy version: 1.20.1
Pytorch version: 1.9.0a0+2ecb2c7
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: cbf252199a8a1aa48672012f48d18dfdbb94301c
Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: NOT INSTALLED or UNKNOWN VERSION.
scikit-image version: 0.15.0
Pillow version: 7.0.0.post3
Tensorboard version: 1.15.0+nv
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.9.0a0
ITK version: NOT INSTALLED or UNKNOWN VERSION.
tqdm version: 4.53.0 [0/1831]
lmdb version: 1.2.1
psutil version: 5.8.0
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.2 LTS
Platform: Linux-4.15.0-117-generic-x86_64-with-glibc2.10
Processor: x86_64
Machine: x86_64
Python version: 3.8.8
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 48
Num logical CPUs: 96
Num usable CPUs: 96
CPU usage (%): [55.1, 100.0, 100.0, 44.4, 54.9, 100.0, 47.0, 100.0, 68.2, 48.2, 39.8, 100.0, 90.2, 100.0, 35.8, 100.0, 61.2, 48.8, 100.0, 45.1, 42.7, 46.3, 87.8, 69.9, 100.0, 69.5, 95.2, 100.0, 78.0, 68.3, 71.4, 100.0, 100.0, 50.0, 57.8, 65.1, 63.8, 52.4, 60.5, 66.3, 95.2, 56.8, 74.7, 54.3, 78.0, 56.1, 49.4, 68.3, 75.9, 43.4, 58.5, 80.7, 100.0, 58.5, 48.2, 61.0, 50.0, 50.6, 97.5, 41.0, 39.0, 55.4, 54.2, 40.7, 54.3, 100.0, 59.8, 100.0, 45.1, 100.0, 62.7, 79.8, 49.4, 67.1, 58.0, 65.9, 53.7, 58.5, 59.0, 63.4, 64.6, 57.8, 57.3, 100.0, 100.0, 66.3, 92.6, 97.6, 54.3, 94.0, 55.4, 56.8, 50.6, 100.0, 51.2, 55.0]
CPU freq. (MHz): 3300
Load avg. in last 1, 5, 15 mins (%): [131.6, 102.8, 97.1]
Disk usage (%): 51.5
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 1510.6
Available memory (GB): 1388.3
Used memory (GB): 108.6
================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.3
cuDNN enabled: True
cuDNN version: 8200
Current device: 0
Library compiled for CUDA architectures: ['sm_52', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_86']
GPU 0 Name: Tesla V100-SXM3-32GB
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 80
GPU 0 Total memory (GB): 31.7
GPU 0 CUDA capability (maj.min): 7.0