🐛 Bug
I am deploying a PyTorch model to production using uwsgi + Flask. I am using uwsgi config option lazy-apps = true: this loads the Flask application one time per worker, instead of forking from a master worker (read here for further details). It's needed to avoid deadlocks on the model. The problem is that when uwsgi kills or cheaps running workers, Segmentation Fault errors arise, such as:
!!! uWSGI process 14479 got Segmentation Fault !!!
malloc(): unsorted double linked list corrupted
!!! uWSGI process 11199 got Segmentation Fault !!!
corrupted double-linked list
Or:
!!! uWSGI process 11212 got Segmentation Fault !!!
*** backtrace of 11212 ***
uwsgi(uwsgi_backtrace+0x2e) [0x55632cfc3e7e]
uwsgi(uwsgi_segfault+0x27) [0x55632cfc4267]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fb23bc85210]
/lib/x86_64-linux-gnu/libc.so.6(+0x499f6) [0x7fb23bc889f6]
/lib/x86_64-linux-gnu/libc.so.6(on_exit+0) [0x7fb23bc88be0]
uwsgi(+0x3c925) [0x55632cf77925]
uwsgi(end_me+0x2b) [0x55632cfc0e2b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fb23c5ec3c0]
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0x216) [0x7fb23c5e7376]
/home/flucantoni/Projects/segfault/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-ae94cfde.3.9.dev.so(+0x30da4b) [0x7fb2289cba4b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fb23c5e0609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fb23bd61103]
*** end of backtrace ***
Just importing torch in the main Flask file is sufficient to cause this issue
To Reproduce
Steps to reproduce the behavior:
- Set up virtualenv with
uWSGI==2.0.19.1
Flask==1.1.2
torch==1.5.1+cpu
torchvision==0.6.1+cpu
- Create
uwsgi.ini config file
[uwsgi]
cheaper = 1
processes = 2
module = main
callable = app
show-config = true
lazy-apps = true
- Create a basic Flask main.py
import torch
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
return "Hello world!"
- Run uwsgi with
wsgi --http 127.0.0.1:3031 uwsgi.ini
- After loading is completed stop uwsgi Ctrl + C and see Segmentation Fault errors be raised
Expected behavior
I would expect workers being killed gracefully without errors
Environment
Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).
You can get the script and run it with:
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
- PyTorch Version (e.g., 1.0): 1.5.1+cpu
- OS (e.g., Linux): OS: Ubuntu 20.04 LTS
- GCC version: (Ubuntu 9.3.0-10ubuntu2) 9.3.0
- How you installed PyTorch (
conda, pip, source): pip
- Build command you used (if compiling from source):
- Python version: 3.8.2
- CUDA/cuDNN version: None
- GPU models and configuration: None
- Any other relevant information:
[pip3] numpy==1.19.1
[pip3] torch==1.5.1+cpu
[pip3] torchvision==0.6.1+cpu
uWSGI==2.0.19.1
Flask==1.1.2
Additional context
cc @ezyang @gchanan @zou3519
🐛 Bug
I am deploying a PyTorch model to production using
uwsgi+Flask. I am usinguwsgiconfig optionlazy-apps = true: this loads the Flask application one time per worker, instead of forking from a master worker (read here for further details). It's needed to avoid deadlocks on the model. The problem is that whenuwsgikills or cheaps running workers, Segmentation Fault errors arise, such as:Or:
Just importing torch in the main Flask file is sufficient to cause this issue
To Reproduce
Steps to reproduce the behavior:
uwsgi.iniconfig filewsgi --http 127.0.0.1:3031 uwsgi.iniExpected behavior
I would expect workers being killed gracefully without errors
Environment
Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).
You can get the script and run it with:
conda,pip, source): pip[pip3] numpy==1.19.1
[pip3] torch==1.5.1+cpu
[pip3] torchvision==0.6.1+cpu
uWSGI==2.0.19.1
Flask==1.1.2
Additional context
cc @ezyang @gchanan @zou3519