Skip to content

Using pip download to fetch package sources seems to trigger building wheels for some packages. #8387

Closed

Description

@tvalentyn

Environment

  • pip version: 20.1.1
  • Python version: 3.7
  • OS: Debian Linux

Description

In numpy/numpy#14053 we observed that

python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary :all: --no-use-pep517

finishes immediately, while:

python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary :all:

takes a while to complete.

We originally noticed the slowdown in https://issues.apache.org/jira/browse/ARROW-8983.
The command python -m pip download --dest /tmp pyarrow==0.17.0 --no-binary :all: also gets stuck for a few minutes with "Installing build dependencies ... ", and increased CPU usage when pip downloads sources of numpy, a dependency of pyarrow. Interestingly, in case of pyarrow, adding --no-use-pep517 does not help.

Is there an inefficiency in pip download, or it's a known behavior and is considered WAI?

Thank you.

Activity

  1. ghost added
    S: needs triageIssues/PRs that need to be triaged
    on Jun 2, 2020
  2. sbidoul commented on Jun 3, 2020

    @sbidoul
    Member

    I made a quick test in verbose mode, and it appears that part of the slowdown comes from --no-binary :all: which extends to build dependencies. Since build dependencies include cython, pip first builds cython from source before attempting to obtain metadata.

    python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary numpy (or --no-binary :all: --only-binary cython) is faster because it installs cython from binary wheel.

    Nevertheless, that still involves some C compilation which does not occur with --no-use-pep517. I'm not sure why yet.

    There is ongoing discussion on how/whether --no-binary should apply to build dependencies. I don't think that has been conclusive yet.

  3. jorisvandenbossche commented on Jun 3, 2020

    @jorisvandenbossche

    Naive question: why is it needed to build numpy or create its build environment when you only want to download the source code? Is it to get some metadata that is only available from a wheel?

  4. pfmoore commented on Jun 3, 2020

    @pfmoore
    Member

    Is it to get some metadata that is only available from a wheel?

    Precisely that.

  5. jorisvandenbossche commented on Jun 3, 2020

    @jorisvandenbossche

    And how do you get that metadata in case of --no-use-pep517 ?

  6. sbidoul commented on Jun 3, 2020

    @sbidoul
    Member

    And how do you get that metadata in case of --no-use-pep517 ?

    setup.py egg_info

  7. sbidoul commented on Jun 7, 2020

    @sbidoul
    Member

    I looked into this a bit further and it appears that setuptools' implementation of PEP 517 prepare_metadata_for_build_wheel() uses setup.py dist_info. In the case of numpy 1.18.0, this involves cython and some compilation (of _configtest.c).

    setup.py egg_info runs instantly.

    That explains the performance difference, although I can't explain myself why dist_info and egg_info behave differently, the resulting metadata (METADATA vs PKG-INFO) being very similar.

  8. uranusjr commented on Jun 7, 2020

    @uranusjr
    Member

    Numpy has a highly customised setup.py that triggers said compilation, unless explicitly excluded. egg_info is one of those commands.

    See parse_setuppy_commands() here: https://github.com/numpy/numpy/blob/v1.18.0/setup.py

  9. zeha commented on Jul 7, 2020

    @zeha

    Why does downloading an sdist need metadata from a wheel in the first place?

  10. zeha commented on Jul 7, 2020

    @zeha

    Appears to be related to (duplicate of?) #7995.

  11. uranusjr commented on Jul 9, 2020

    @uranusjr
    Member

    Closing this in favour of #7995.

  12. locked as resolved and limited conversation to collaborators on Oct 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions