You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Using pip download to fetch package sources seems to trigger building wheels for some packages. #8387
We originally noticed the slowdown in https://issues.apache.org/jira/browse/ARROW-8983.
The command python -m pip download --dest /tmp pyarrow==0.17.0 --no-binary :all: also gets stuck for a few minutes with "Installing build dependencies ... ", and increased CPU usage when pip downloads sources of numpy, a dependency of pyarrow. Interestingly, in case of pyarrow, adding --no-use-pep517 does not help.
Is there an inefficiency in pip download, or it's a known behavior and is considered WAI?
I made a quick test in verbose mode, and it appears that part of the slowdown comes from --no-binary :all: which extends to build dependencies. Since build dependencies include cython, pip first builds cython from source before attempting to obtain metadata.
python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary numpy (or --no-binary :all: --only-binary cython) is faster because it installs cython from binary wheel.
Nevertheless, that still involves some C compilation which does not occur with --no-use-pep517. I'm not sure why yet.
There is ongoing discussion on how/whether --no-binary should apply to build dependencies. I don't think that has been conclusive yet.
Naive question: why is it needed to build numpy or create its build environment when you only want to download the source code? Is it to get some metadata that is only available from a wheel?
I looked into this a bit further and it appears that setuptools' implementation of PEP 517 prepare_metadata_for_build_wheel() uses setup.py dist_info. In the case of numpy 1.18.0, this involves cython and some compilation (of _configtest.c).
setup.py egg_info runs instantly.
That explains the performance difference, although I can't explain myself why dist_info and egg_info behave differently, the resulting metadata (METADATA vs PKG-INFO) being very similar.
Activity
sbidoul commentedon Jun 3, 2020
I made a quick test in verbose mode, and it appears that part of the slowdown comes from
--no-binary :all:which extends to build dependencies. Since build dependencies includecython, pip first builds cython from source before attempting to obtain metadata.python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary numpy(or--no-binary :all: --only-binary cython) is faster because it installscythonfrom binary wheel.Nevertheless, that still involves some C compilation which does not occur with
--no-use-pep517. I'm not sure why yet.There is ongoing discussion on how/whether
--no-binaryshould apply to build dependencies. I don't think that has been conclusive yet.jorisvandenbossche commentedon Jun 3, 2020
Naive question: why is it needed to build numpy or create its build environment when you only want to download the source code? Is it to get some metadata that is only available from a wheel?
pfmoore commentedon Jun 3, 2020
Precisely that.
jorisvandenbossche commentedon Jun 3, 2020
And how do you get that metadata in case of
--no-use-pep517?sbidoul commentedon Jun 3, 2020
setup.py egg_infosbidoul commentedon Jun 7, 2020
I looked into this a bit further and it appears that setuptools' implementation of PEP 517
prepare_metadata_for_build_wheel()usessetup.py dist_info. In the case of numpy 1.18.0, this involves cython and some compilation (of_configtest.c).setup.py egg_inforuns instantly.That explains the performance difference, although I can't explain myself why dist_info and egg_info behave differently, the resulting metadata (METADATA vs PKG-INFO) being very similar.
uranusjr commentedon Jun 7, 2020
Numpy has a highly customised
setup.pythat triggers said compilation, unless explicitly excluded.egg_infois one of those commands.See
parse_setuppy_commands()here: https://github.com/numpy/numpy/blob/v1.18.0/setup.pyzeha commentedon Jul 7, 2020
Why does downloading an sdist need metadata from a wheel in the first place?
zeha commentedon Jul 7, 2020
Appears to be related to (duplicate of?) #7995.
uranusjr commentedon Jul 9, 2020
Closing this in favour of #7995.