Memory leak when connection pool is constantly overflown? (asyncpg) #7058
Unanswered
0xD09CD196D188D0B0
asked this question in
Usage Questions
Replies: 1 comment 4 replies
-
|
Hi, from looking at your script I can't see anything amiss. Did you try recreating it by using asyncpg directly? |
Beta Was this translation helpful? Give feedback.
4 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
We were using default settings of the connection pool (
pool_size=5andoverflow=10) and after several hours of higher load, we encountered OOM's. It was constantly processing more than 5 tasks concurrently and the pool constantly was creating/removing extra connections allowed byoverflow. During the debugging, we found out thatasyncpg/connect_utils.py:705:is constantly eating more and more memory, reaching hundreds of megabytes.We attach a simple script to simulate a constant pool overflow by printing some stats.
after we run this dummy script locally, we end up with stats like
/asyncio/sslproto.py:545:andasyncpg/connect_utils.py:705takes 2.5 MBs even after the engine disposal and manual garbage collection.If set
num_jobsto 5, the pool won't be overflown and these are disappearing from top tracemalloc stats.So it is kind of reproduces the issue we have.
We may increase pool settings, but we can't be sure that there won't be a bigger load and it won't start leaking again.
I'm not sure whether it is SQLAlchemy's or asyncpg's issue, but we do not use asyncpg directly so I'm posting it here.
our versions:
Beta Was this translation helpful? Give feedback.
All reactions