interplay with multiprocessing #841
Description
Describe the bug
I'm trying to run parallel tasks with a timeout per task (using multiprocessing) inside an API method. On trying to terminate the child processes post the time limit, the server process shuts down and disconnects.
To Reproduce
- Create a file:
repro.py
import os
import time
import uvicorn
from fastapi import FastAPI
from concurrent.futures import ProcessPoolExecutor
app = FastAPI(title='multiprocessing-issue')
def simple_routine(sleep_for):
print(f"PID {os.getpid()} has run time: {sleep_for}")
time.sleep(sleep_for)
return "done"
@app.post("/test-endpoint/?")
def test_endpoint():
print(f"main process: {os.getpid()}")
START_TIME = time.time()
with ProcessPoolExecutor(max_workers=3) as pool:
futures = [
pool.submit(simple_routine, 1),
pool.submit(simple_routine, 1),
pool.submit(simple_routine, 20),
]
results = []
for fut in futures:
try:
results.append(fut.result(timeout=2))
except:
results.append("not done")
# terminate the processes which are still running
for pid, proc in pool._processes.items():
print("terminating pid ", pid)
proc.terminate()
print("\n", "exiting at: ", int(time.time() - START_TIME))
return True
if __name__=="__main__":
uvicorn.run(app, host="0.0.0.0", port=5000)- Run it as
python repro.py. - Open another python interpreter and make this web request.
import requests
for _ in range(20):
print(requests.post("http://localhost:5000/test-endpoint").text)- The server process shuts down after the first request.
Expected behavior
The server shouldn't shut down and continue serving requests. Interestingly, the server logs a shutdown signal when I try to terminate the subprocesses but doesn't actually exit until the long running process is complete.
INFO: Started server process [615]
INFO: Uvicorn running on http://0.0.0.0:5900 (Press CTRL+C to quit)
INFO: Waiting for application startup.
INFO: Application startup complete.
main process: 615
PID 637 has run time: 1
PID 638 has run time: 1
PID 639 has run time: 10
terminating pid 637
terminating pid 638
terminating pid 639
INFO: Shutting down
INFO: Finished server process [615]
exiting at: 10Environment
- OS: [Ubuntu 18.04.1 LTS]
- FastAPI Version: 0.45.0
- Python version: 3.6.8
Additional context
I'm trying to move a working wsgi API written with Flask to FastAPI, really for the brevity and automatic documentation. The parallel tasks are cpu intensive methods and the termination works as expected with Flask. For context, the output from the Flask logger which is the expected behavior.
The issue likely has something to do with how FastAPI runs sync methods but I can't quite figure it.
main process: 1015
PID 1035 has run time: 1
PID 1039 has run time: 1
PID 1038 has run time: 10
terminating pid 1035
terminating pid 1038
terminating pid 1039
exiting at: 3
127.0.0.1 - - [09/Jan/2020 08:51:37] "POST /test-endpoint HTTP/1.1" 200 -Thank you for looking.