Skip to content

Modern installer eats a lot of RAM when installing big wheels #7983

@ralbertazzi

Description

@ralbertazzi
  • Poetry version: 1.5.0
  • Python version: 3.9.8
  • OS version and name: Ubuntu 20.04 (also observed on other platforms)
  • pyproject.toml:
[tool.poetry]
authors = ["Author <[email protected]>"]
description = "Project"
name = "project"
version = "1.0.0"

[tool.poetry.dependencies]
python = "~3.9"

torch = { url = "https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp39-cp39-linux_x86_64.whl" }
  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Poetry's modern installer is using a lot of RAM when installing big wheels, such as the PyTorch one (>1 GB) that I provided in the example. The RAM usage starts increasing while Poetry is Installing... the wheel, and it spikes over 10 GB of RAM used while performing the operation. The RAM usage goes back to acceptable values before the Installing... operation is done.

Deactivating the modern installer (poetry config installer.modern-installation false) fixes the problem. The RAM usage does not go beyond 1 GB there.

I haven't debugged deeply enough to tell if it's a Poetry issue or an pypa/installer issue, but for sure Poetry is affected by it. This is bad for CI environments where resources are more limited and currently cause poetry install to go OOM.

cc #6409

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/installerRelated to the dependency installerkind/bugSomething isn't working as expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions