Accumulating memory usage with 0.10.x #430
Description
Following on from a discussion in #419, I am seeing increased memory usage over time with version 0.10.x.
In particular, I was hitting out-of-memory conditions with a long-running Celery task that uses the Django ORM.
I used tracemalloc to take snapshots at the beginning and end of the task.
Under 0.10.1, here are the top 20 differences between the snapshots taken at the end and start of the task (with it running for four minutes):
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:746: size=33.6 MiB (+33.6 MiB), count=67076 (+67076), average=525 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:199: size=11.4 MiB (+11.3 MiB), count=73026 (+72307), average=164 B
/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py:472: size=10.7 MiB (+10.7 MiB), count=66784 (+66784), average=168 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:192: size=10.8 MiB (+10.7 MiB), count=67265 (+66702), average=168 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:123: size=9453 KiB (+9453 KiB), count=67215 (+67215), average=144 B
/venv/lib/python3.7/site-packages/django/db/models/lookups.py:159: size=8197 KiB (+8197 KiB), count=131148 (+131148), average=64 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:307: size=6432 KiB (+6107 KiB), count=3093 (+2840), average=2129 B
/venv/lib/python3.7/site-packages/django/db/models/lookups.py:19: size=4957 KiB (+4948 KiB), count=71804 (+71636), average=71 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:2197: size=4868 KiB (+4863 KiB), count=69230 (+69166), average=72 B
/venv/lib/python3.7/site-packages/django/db/models/expressions.py:159: size=4815 KiB (+4782 KiB), count=68264 (+67877), average=72 B
/venv/lib/python3.7/site-packages/sentry_sdk/hub.py:406: size=4710 KiB (+4710 KiB), count=66977 (+66977), average=72 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:89: size=4254 KiB (+4254 KiB), count=67019 (+67018), average=65 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:753: size=3669 KiB (+3669 KiB), count=67072 (+67072), average=56 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:194: size=2620 KiB (+2620 KiB), count=67069 (+67069), average=40 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:99: size=2618 KiB (+2618 KiB), count=67019 (+67018), average=40 B
/venv/lib/python3.7/site-packages/sentry_sdk/tracing.py:186: size=2618 KiB (+2618 KiB), count=67018 (+67018), average=40 B
/usr/lib/python3.7/uuid.py:204: size=2521 KiB (+2521 KiB), count=46067 (+46066), average=56 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:736: size=2097 KiB (+2097 KiB), count=67068 (+67068), average=32 B
/venv/lib/python3.7/site-packages/sentry_sdk/utils.py:723: size=2096 KiB (+2096 KiB), count=67067 (+67067), average=32 B
/venv/lib/python3.7/site-packages/django/db/utils.py:96: size=1738 KiB (+1738 KiB), count=26978 (+26978), average=66 B
Under 0.9.5, it looks like this:
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:307: size=6432 KiB (+6107 KiB), count=3086 (+2833), average=2134 B
/usr/lib/python3.7/uuid.py:204: size=2511 KiB (+2511 KiB), count=45919 (+45918), average=56 B
/venv/lib/python3.7/site-packages/django/db/utils.py:96: size=1738 KiB (+1738 KiB), count=26976 (+26976), average=66 B
/venv/lib/python3.7/site-packages/django/db/models/base.py:411: size=1696 KiB (+1696 KiB), count=11438 (+11438), average=152 B
/venv/lib/python3.7/site-packages/psycopg2/extras.py:678: size=1256 KiB (+1256 KiB), count=22969 (+22967), average=56 B
/usr/lib/python3.7/uuid.py:161: size=987 KiB (+987 KiB), count=22966 (+22966), average=44 B
/venv/lib/python3.7/site-packages/django/db/models/sql/datastructures.py:46: size=984 KiB (+886 KiB), count=13368 (+12044), average=75 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:310: size=882 KiB (+840 KiB), count=6025 (+5661), average=150 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:1231: size=870 KiB (+818 KiB), count=9018 (+8493), average=99 B
/venv/lib/python3.7/site-packages/django/db/models/base.py:408: size=750 KiB (+750 KiB), count=7837 (+7837), average=98 B
/venv/lib/python3.7/site-packages/django/db/models/sql/datastructures.py:116: size=754 KiB (+732 KiB), count=8880 (+8509), average=87 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:199: size=747 KiB (+664 KiB), count=6387 (+5668), average=120 B
/venv/lib/python3.7/site-packages/django/db/models/query.py:190: size=717 KiB (+664 KiB), count=6143 (+5666), average=119 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:337: size=702 KiB (+663 KiB), count=5999 (+5661), average=120 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:354: size=652 KiB (+620 KiB), count=2980 (+2832), average=224 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:311: size=645 KiB (+620 KiB), count=2950 (+2832), average=224 B
/venv/lib/python3.7/site-packages/django/db/models/sql/query.py:309: size=618 KiB (+580 KiB), count=3844 (+3537), average=165 B
/venv/lib/python3.7/site-packages/django/utils/deconstruct.py:17: size=762 KiB (+517 KiB), count=8924 (+4472), average=87 B
/venv/lib/python3.7/site-packages/django/db/models/fields/related.py:664: size=555 KiB (+494 KiB), count=9637 (+8572), average=59 B
/venv/lib/python3.7/site-packages/django/db/models/base.py:513: size=479 KiB (+479 KiB), count=7142 (+7142), average=69 B
Django is on version 2.2.2, Python on 3.7.3.
Are all the executed SQL statements being kept in memory?