Skip to content

Comments

Prefer compatible wheels over source distribution for legacy repository dependency resolution#5385

Closed
gpfv wants to merge 2 commits intopython-poetry:masterfrom
gpfv:master
Closed

Prefer compatible wheels over source distribution for legacy repository dependency resolution#5385
gpfv wants to merge 2 commits intopython-poetry:masterfrom
gpfv:master

Conversation

@gpfv
Copy link

@gpfv gpfv commented Mar 30, 2022

Pull Request Check List

Resolves: #4439 and #3464

  • Added tests for changed code.
  • Updated documentation for changed code. (not needed)

PyPi repositories use the pypi.org JSON api for fetching dependencies, where for an external repo (legacy repo) the dependencies are determined by going through the files available in the repo and inspecting the metadata (actually PyPi repositories also use this method as fallback when requires_dist is None). Universal wheels are prefered, then source distribution and only after that a desperate attempt (# Pick the first wheel available and hope for the best) is made to extract the dependencies from a platform specific wheel.

It is cumbersome to retrieve this from source, the file needs to be extracted, which takes a long time, and even after it is not always certain the correct ones can be determined, due to lack of standardization and arbitrary code execution in setup.py. So either this step has to be improved, implying all past and future build systems need to be supported, or another option is to use a well defined package distribution format, which already exist. The wheel does not need to be reinvented, it just needs to be used.

As dependencies could potentially differ (although could not find an example of this easily) depending on the platform and python version, only the appropriate wheels are considered by matching with compatibility tags.

-> Using compatible wheels over sdist will result in a more correct dependency resolution at a greater speed.

for example PyQt5 (5.15.4) requires_dist

  • pypi ['PyQt5-sip (>=12.8, <13)', 'PyQt5-Qt5 (>=5.15)']
  • bdist ['PyQt5-sip (>=12.8, <13)', 'PyQt5-Qt5 (>=5.15)']
  • sdist None

wheel : ~3s not cached, ~0.2s cached
tar.gz: ~10s both not cached and cached (needs to be extracted everytime)

@abn
Copy link
Member

abn commented Apr 6, 2022

@gpfv this might need some TLC. :)

@gpfv
Copy link
Author

gpfv commented Apr 19, 2022

Testing Linux Capabilities added
will now only mock on Windows the two platforms win32 and win-amd64

@abn abn mentioned this pull request May 10, 2022
Copy link
Member

@neersighted neersighted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after a rebase.

@neersighted
Copy link
Member

Superseded by #6547

@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dependency resolution differences (wrong) when using custom (i.e. not pypi) repository

3 participants