Poetry relies on tomlkit 0.11.{2/3} which has a memory leak #6144
Description
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- OS version and name: >>>cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal - Poetry version: Multiple
- Link of a Gist with the contents of your pyproject.toml file:
Issue
TL;DR: poetry export now seems to use 5x+ more memory than it used to.
In a CircleCI workflow, we run poetry export. This has worked for months, but sometime between Friday (8/5/2022) and Monday (8/8/2022) morning, something changed and caused poetry export to use significantly more memory.
Using this cimg/python:3.9 docker image which has poetry 1.1.13 installed, we install poetry 1.1.14 using curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 - --version 1.1.14.
Then running poetry export produces the following allocations (using https://github.com/bloomberg/memray):
⠋ Calculating high watermark... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Memray ERROR: Failed to process memory record
Allocation metadata
-------------------
Command line arguments: '/home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/memray/__main__.py run -o output.bin /home/circleci/.local/bin/poetry export'
Peak memory size: 1.878GB
Number of allocations: 55887
Biggest 10 allocations:
-----------------------
📂 1.078GB (100.00 %) <ROOT>
└── [[20 frames hidden in 14 file(s)]]
└── 📂 1.078GB (100.00 %) <listcomp> /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/source.py:71
├── [[2 frames hidden in 2 file(s)]]
│ └── 📄 86.750MB (7.86 %) update_wrapper /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
├── 📂 219.125MB (19.86 %) __init__ /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/toml_char.py:9
│ ├── 📄 128.375MB (11.63 %) decorating_function /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:519
│ └── [[1 frames hidden in 1 file(s)]]
│ └── 📄 90.750MB (8.22 %) update_wrapper /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
├── 📂 215.750MB (19.55 %) __init__ /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/toml_char.py:11
│ ├── 📄 122.500MB (11.10 %) decorating_function /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:519
│ └── [[1 frames hidden in 1 file(s)]]
│ └── 📄 93.250MB (8.45 %) update_wrapper /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
├── 📂 212.250MB (19.24 %) __init__ /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/toml_char.py:13
│ ├── 📄 120.500MB (10.92 %) decorating_function /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:519
│ └── [[1 frames hidden in 1 file(s)]]
│ └── 📄 91.750MB (8.32 %) update_wrapper /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
├── [[2 frames hidden in 2 file(s)]]
│ └── 📄 97.500MB (8.84 %) update_wrapper /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
└── [[2 frames hidden in 2 file(s)]]
└── 📄 86.750MB (7.86 %) update_wrapper /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
Workaround attempts:
- Running the same workflow without installing poetry 1.1.13 runs fine but suffers from Poetry not passing through SHAs for certain package versions. #5967.
- Updating poetry using
poetry update selfresults in the same OOM. - Uninstalling poetry using the info from this issue, Cannot Uninstall Poetry on #644, does not work. Therefore, I can't test uninstalling
1.1.13and installing1.1.14.
Running the same workflow on this cimg/python:3.10 docker image which has poetry 1.1.14 runs which much smaller allocation:
Allocation metadata
-------------------
Command line arguments: '/home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/memray/__main__.py run -o output.bin /home/circleci/.local/bin/poetry export'
Peak memory size: 209.743MB
Number of allocations: 488785
Biggest 10 allocations:
-----------------------
📂 170.559MB (100.00 %) <ROOT>
└── [[3 frames hidden in 2 file(s)]]
└── 📂 170.559MB (100.00 %) _run_code /home/circleci/.pyenv/versions/3.10.6/lib/python3.10/runpy.py:86
├── [[5 frames hidden in 4 file(s)]]
│ └── 📂 169.304MB (99.26 %) wrap_handle /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/commands/command.py:92
│ ├── [[2 frames hidden in 2 file(s)]]
│ │ └── 📂 141.291MB (82.84 %) read /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/core/toml/file.py:32
│ │ ├── [[2 frames hidden in 2 file(s)]]
│ │ │ └── 📂 140.117MB (82.15 %) parse /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/api.py:82
│ │ │ ├── [[3 frames hidden in 2 file(s)]]
│ │ │ │ └── 📄 36.000MB (21.11 %) __init__ /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/toml_char.py:8
│ │ │ └── [[10 frames hidden in 2 file(s)]]
│ │ │ └── 📄 14.087MB (8.26 %) _to_linecol /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/source.py:179
│ │ └── [[1 frames hidden in 1 file(s)]]
│ │ └── 📄 1.173MB (0.69 %) decode /home/circleci/.pyenv/versions/3.10.6/lib/python3.10/codecs.py:322
│ ├── [[7 frames hidden in 5 file(s)]]
│ │ └── 📂 25.014MB (14.67 %) parse /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/api.py:82
│ │ ├── 📂 16.000MB (9.38 %) parse /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/parser.py:157
│ │ │ ├── [[9 frames hidden in 2 file(s)]]
│ │ │ │ └── 📄 14.000MB (8.21 %) _to_linecol /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/source.py:179
│ │ │ └── [[6 frames hidden in 3 file(s)]]
│ │ │ └── 📄 2.000MB (1.17 %) <genexpr> /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/items.py:412
│ │ └── [[3 frames hidden in 2 file(s)]]
│ │ └── 📄 5.000MB (2.93 %) __init__ /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/toml_char.py:8
│ └── [[13 frames hidden in 11 file(s)]]
│ └── 📄 3.000MB (1.76 %) <listcomp> /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/source.py:71
└── [[75 frames hidden in 14 file(s)]]
└── 📄 1.255MB (0.74 %) _compile_bytecode <frozen importlib._bootstrap_external>:672