Skip to content

Segmentation fault on large concat operation #21526

@pinkhamr-fb-zz

Description

@pinkhamr-fb-zz

🐛 Bug

When concatenating two large tensors a 'Segmentation fault (core dumped)' is thrown. Tested on an older version of pytorch(1.0.1) and bug was not present.

To Reproduce

Steps to reproduce the behavior:

import torch
concat_list = []

concat_list.append(torch.ones((6500,1024*512), dtype=torch.uint8))
concat_list.append(torch.ones((4500,1024*512), dtype=torch.uint8))

ccat = torch.cat(concat_list)

Output:
Segmentation fault (core dumped)

Expected behavior

Expected: a concatenated tensor with lots of ones

Environment

Tried it on 2 computers:
Build 1:
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Fedora release 29 (Twenty Nine)
GCC version: (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
CMake version: version 3.14.4

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.2.148
GPU models and configuration:
GPU 0: Quadro P4000
GPU 1: TITAN RTX

Nvidia driver version: 418.43
cuDNN version: /usr/lib64/libcudnn.so.7.2.1

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.2.2.post3
[conda] Could not collect

Build 2
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti

Nvidia driver version: 384.130
cuDNN version: Probably one of the following:
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6.0.21
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.7.0.3
/usr/local/cuda-9.0/lib64/libcudnn.so.6.0.21
/usr/local/cuda-9.0/lib64/libcudnn.so.7.3.1

Versions of relevant libraries:
[pip] numpy==1.14.0
[pip] torch==1.0.0
[pip] torchvision==0.1.9
[conda] Could not collect

Additional context

Both machines have 64GB of RAM, so I expect the whole tensor should easily fit

Metadata

Metadata

Assignees

Labels

triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions