Latest setuptools breaks installing packages with native code. #152
Description
I noticed this today, when some of our weekly automated Docker-Image-Builds randomly failed.
For example, trying to install numpy after updating setuptools results in the following:
Details
# pip install --no-cache-dir numpy
Collecting numpy
Downloading numpy-1.21.5.zip (10.7 MB)
|████████████████████████████████| 10.7 MB 4.1 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: numpy
Building wheel for numpy (pyproject.toml) ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/pyenv/versions/pyston-2.3.1/bin/python /usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp5il3akyy
cwd: /tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9
Complete output (95 lines):
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/mtrand.pyx
Processing numpy/random/_sfc64.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/_philox.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_common.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/bit_generator.pyx
Cythonizing sources
creating /tmp/tmpficqc93o/tmp
creating /tmp/tmpficqc93o/tmp/tmpficqc93o
non-existing path in 'numpy/distutils': 'site.cfg'
running bdist_wheel
running build
running config_cc
running config_fc
running build_src
creating build
creating build/src.linux-x86_64-3.8
creating build/src.linux-x86_64-3.8/numpy
creating build/src.linux-x86_64-3.8/numpy/distutils
creating build/src.linux-x86_64-3.8/numpy/core
creating build/src.linux-x86_64-3.8/numpy/core/src
creating build/src.linux-x86_64-3.8/numpy/core/src/npymath
creating build/src.linux-x86_64-3.8/numpy/core/src/multiarray
Running from numpy source directory.
/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/system_info.py:2026: UserWarning:
Optimized (vendor) Blas libraries are not found.
Falls back to netlib Blas library which has worse performance.
A better performance should be easily gained by switching
Blas library.
if self._calc_info(blas):
Warning: attempted relative import with no known parent package
/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
warnings.warn(msg)
Traceback (most recent call last):
File "/usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/pyenv/versions/pyston-2.3.1/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 230, in build_wheel
return self._build_with_temp_dir(['bdist_wheel'], '.whl',
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 267, in run_setup
super(_BuildMetaLegacyBackend,
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/build_meta.py", line 158, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 448, in <module>
setup_package()
File "setup.py", line 440, in setup_package
setup(**metadata)
File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/core.py", line 169, in setup
return old_setup(**new_attr)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/core.py", line 148, in setup
dist.run_commands()
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build.py", line 61, in run
old_build.run(self)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-yaf4w4eu/overlay/lib/python3.8-pyston2.3/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 144, in run
self.build_sources()
File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 161, in build_sources
self.build_extension_sources(ext)
File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 318, in build_extension_sources
sources = self.generate_sources(sources, ext)
File "/tmp/pip-install-xkdkddw0/numpy_d7c39e368b80449c84a5556ec68ea3b9/numpy/distutils/command/build_src.py", line 378, in generate_sources
source = func(extension, build_dir)
File "numpy/core/setup.py", line 434, in generate_config_h
moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
File "numpy/core/setup.py", line 44, in check_types
out = check_types(*a, **kw)
File "numpy/core/setup.py", line 289, in check_types
raise SystemError(
SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
----------------------------------------
ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
Trying to install pandas results in a similar error, though then during an invocation of gcc, which likewise can't find Python.h.
Looking at the gcc commandline reveals that it gets passed the following include path: -I/usr/local/pyenv/versions/pyston-2.3.1/include/python3.8
The correct path would have been /usr/local/pyenv/versions/pyston-2.3.1/include/python3.8-pyston2.3.
This only very recently broke. I first traced it back to setuptools>=60.
And then looking at individual commits and some help on #pypa IRC, I ended up at pypa/setuptools@b6fcbbd.
What this does is change the default distutils setuptools uses to a local copy of it, overriding and replacing the stdlib one.
This behaviour can temporarily be reverted via export SETUPTOOLS_USE_DISTUTILS=stdlib before updating setuptools.
But in the long run, due to distutils being deprecated, that option will be removed from setuptools.
I'm not sure I fully understand what is happening, but my assumption is that Pyston has some custom changes in distutils to make the custom version tags work, which in turn get ignored/overwritten by setuptools local copy of distutils.