Skip to content

21.3: Breaking editable installation with No module named pip error #10573

Closed

Description

@Panaetius

Description

Our CI is failing due to version 21.3 release, in the pip wheel subprocess call during installation, with seemingly random packages being the culprit.

E.g.

  subprocess.CalledProcessError: Command '['/opt/hostedtoolcache/Python/3.7.12/x64/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpiq0bznxg', '--quiet', 'cryptography<3.5,>=3.4.1']' returned non-zero exit status 1.

Full Log attached below

The specific dependency it fails for seems completely random, in 5 subsequent runs it's been pathspec, cryptography, tabulate, rdflib and pyasn1.

I can reproduce the issue locally as well with Arch Linux, 21.3 fails, 21.2.4 works.
This only happens in editable mode (using -e flag), without it everything install fine.

Expected behavior

Renku Package to be installed without problem

pip version

21.3

Python version

3.7.8

OS

Arch Linux/Github Actions(Ubuntu)

How to Reproduce

  1. clone https://github.com/SwissDataScienceCenter/renku-python
  2. upgrade to pip 21.3
  3. do pip install -e .[all]

Output

$ python -m pip install -e .[all]
/home/user/.pyenv/versions/renku-python/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/home/user/.pyenv/versions/renku-python/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
Obtaining file:///home/user/DEV/ETH/renku-python
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
  Build backend does not support editables, falling back to setup.py egg_info.
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /home/user/.pyenv/versions/renku-python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/user/DEV/ETH/renku-python/setup.py'"'"'; __file__='"'"'/home/user/DEV/ETH/renku-python/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-0jfl5x91
       cwd: /home/user/DEV/ETH/renku-python/
  Complete output (31 lines):
  /home/user/.pyenv/versions/renku-python/bin/python: No module named pip
  Traceback (most recent call last):
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
      subprocess.check_call(cmd)
    File "/home/user/.pyenv/versions/3.7.8/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/home/user/.pyenv/versions/renku-python/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpppr5o_gk', '--quiet', 'zc.relation<1.2,>=1.1']' returned non-zero exit status 1.
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/home/user/DEV/ETH/renku-python/setup.py", line 266, in <module>
      "Development Status :: 4 - Beta",
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 152, in setup
      _install_setup_requires(attrs)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 809, in fetch_build_eggs
      replace_conflicting=True,
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 768, in resolve
      replace_conflicting=replace_conflicting
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1051, in best_match
      return self.obtain(req, installer)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1063, in obtain
      return installer(requirement)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 877, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 77, in fetch_build_egg
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['/home/user/.pyenv/versions/renku-python/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpppr5o_gk', '--quiet', 'zc.relation<1.2,>=1.1']' returned non-zero exit status 1.
  ----------------------------------------
WARNING: Discarding file:///home/user/DEV/ETH/renku-python. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Code of Conduct

Activity

  1. added
    S: needs triageIssues/PRs that need to be triaged
    type: bugA confirmed bug or unintended behavior
    on Oct 12, 2021
  2. added
    C: editableEditable installations
    and removed
    S: needs triageIssues/PRs that need to be triaged
    on Oct 12, 2021
  3. teije01 commented on Oct 12, 2021

    @teije01

    Experiencing the same problems, also

    /opt/conda/envs/vdsdb/bin/python3.7: No module named pip
    

    Same with other python versions. Downgrading pip resolved our problem

  4. sbidoul commented on Oct 12, 2021

    @sbidoul
    Member

    Ok, this seems to be related to the PEP 660 implementation.

    Since this project has a pyproject.toml without build-system table, it seems that pip<21.3 did first prepare the metadata using the setuptools.build_meta:__legacy__ build backend, then called setup.py develop, so doing a mix of PEP 517 and legacy installation.

    Now pip first tries a PEP 660 editable metadata preparation and, since the build backend does not support PEP 660, it falls back to a legacy metadata preparation (setup.py egg_info), which should in theory be equivalent but, for some reason I don't fully understand, does not work for your project.

  5. self-assigned this
    on Oct 12, 2021
  6. changed the title [-]21.3 breaking installation of package from source with random error on subprocess call to wheel[/-] [+]21.3 breaking editable installation with `No module named pip` error[/+] on Oct 12, 2021
  7. added this to the 21.3.1 milestone on Oct 12, 2021
  8. sbidoul commented on Oct 12, 2021

    @sbidoul
    Member

    @Panaetius can you test if #10577 fixes the issue ?

    You can install it with pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/10577/head"

  9. sbidoul commented on Oct 12, 2021

    @sbidoul
    Member

    Note I tentatively added this to a 21.3.1 bugfix milestone, since it is a regression, but this does not mean doing an actual bugfix release has been decided yet.

  10. Panaetius commented on Oct 12, 2021

    @Panaetius
    Author

    It gets a lot further in the install process but then unfortunately has the same error when trying to install things:

    https://gist.github.com/Panaetius/5d9b29b3694b310e567bc803fb9878a8

    Thank you for looking into this!

  11. sbidoul commented on Oct 12, 2021

    @sbidoul
    Member

    @Panaetius Could you retry in a fresh environment ? Also removing any .eggs and .egg-info directories that might have been created by previous attempts.

  12. ceelian commented on Oct 12, 2021

    @ceelian

    Having a similar problem on our CI System since the new release. Our CI does a fresh setup of the environment in each build. So IMO it looks like the "freshness" of the environment isn't the cause. Using the release versions pip-21.3 setuptools-58.2.0.

    Our workaround for now: pinning versions to pip==21.2.4 setuptools==57.4.0

  13. sbidoul commented on Oct 12, 2021

    @sbidoul
    Member

    @ceelian my comment about retrying in a fresh environment was specifically for @Panaetius who is testing the linked PR.
    If you'd like to test #10577 too, this is welcome.

  14. 37 remaining items

  15. layday commented on Oct 18, 2021

    @layday
  16. added a commit that references this issue on Oct 20, 2021
  17. added a commit that references this issue on Oct 22, 2021
  18. locked as resolved and limited conversation to collaborators on Nov 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

C: editableEditable installationsPEP implementationInvolves some PEPtype: bugA confirmed bug or unintended behavior

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions