Skip to content

Add support for PyPy#1

Merged
jaraco merged 5 commits intomasterfrom
feature/integrate-pypy
Jul 2, 2020
Merged

Add support for PyPy#1
jaraco merged 5 commits intomasterfrom
feature/integrate-pypy

Conversation

@jaraco
Copy link
Copy Markdown
Member

@jaraco jaraco commented Jul 2, 2020

Thanks to the work of @mattip in pypa/setuptools#2221, support for PyPy in distutils is coming along nicely. This branch cherry-picks the distutils-related commits from that PR and addresses other test-related failures. Now the tests are passing (or skipped) except for one on PyPy:

$ tox -e pypy3
pypy3 installed: attrs==19.3.0,cffi==1.14.0,greenlet==0.4.13,importlib-metadata==1.7.0,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pyparsing==2.4.7,pytest==5.4.3,readline==6.2.4.1,six==1.15.0,wcwidth==0.2.5,zipp==3.1.0
pypy3 run-test-pre: PYTHONHASHSEED='3646014116'
pypy3 run-test: commands[0] | pytest
============================= test session starts ==============================
platform darwin -- Python 3.6.9[pypy-7.3.1-final], pytest-5.4.3, py-1.9.0, pluggy-0.13.1
cachedir: .tox/pypy3/.pytest_cache
rootdir: /Users/jaraco/code/public/pypa/distutils, inifile: pytest.ini
collected 286 items

distutils/tests/test_archive_util.py ....................                [  6%]
distutils/tests/test_bdist.py ...                                        [  8%]
distutils/tests/test_bdist_dumb.py ..                                    [  8%]
distutils/tests/test_bdist_msi.py s.                                     [  9%]
distutils/tests/test_bdist_rpm.py ss.                                    [ 10%]
distutils/tests/test_bdist_wininst.py s.                                 [ 11%]
distutils/tests/test_build.py ..                                         [ 11%]
distutils/tests/test_build_clib.py ......                                [ 13%]
distutils/tests/test_build_ext.py s.............s..............          [ 24%]
distutils/tests/test_build_py.py .......                                 [ 26%]
distutils/tests/test_build_scripts.py ....                               [ 27%]
distutils/tests/test_check.py .s.ss.                                     [ 30%]
distutils/tests/test_clean.py ..                                         [ 30%]
distutils/tests/test_cmd.py ........                                     [ 33%]
distutils/tests/test_config.py ....                                      [ 34%]
distutils/tests/test_config_cmd.py .....                                 [ 36%]
distutils/tests/test_core.py ......                                      [ 38%]
distutils/tests/test_cygwinccompiler.py ....                             [ 40%]
distutils/tests/test_dep_util.py ....                                    [ 41%]
distutils/tests/test_dir_util.py ........                                [ 44%]
distutils/tests/test_dist.py ........s......................             [ 55%]
distutils/tests/test_extension.py ...                                    [ 56%]
distutils/tests/test_file_util.py ......                                 [ 58%]
distutils/tests/test_filelist.py .............                           [ 62%]
distutils/tests/test_install.py .....s..                                 [ 65%]
distutils/tests/test_install_data.py ..                                  [ 66%]
distutils/tests/test_install_headers.py ..                               [ 67%]
distutils/tests/test_install_lib.py ......                               [ 69%]
distutils/tests/test_install_scripts.py ...                              [ 70%]
distutils/tests/test_log.py ..                                           [ 70%]
distutils/tests/test_msvc9compiler.py ssss.                              [ 72%]
distutils/tests/test_msvccompiler.py ssss.                               [ 74%]
distutils/tests/test_register.py .....s..s.                              [ 77%]
distutils/tests/test_sdist.py ......s.........                           [ 83%]
distutils/tests/test_spawn.py ...                                        [ 84%]
distutils/tests/test_sysconfig.py ..........s.s..                        [ 89%]
distutils/tests/test_text_file.py ..                                     [ 90%]
distutils/tests/test_unixccompiler.py F...                               [ 91%]
distutils/tests/test_upload.py .......                                   [ 94%]
distutils/tests/test_util.py ...........                                 [ 98%]
distutils/tests/test_version.py ....                                     [ 99%]
distutils/tests/test_versionpredicate.py .                               [100%]

=================================== FAILURES ===================================
_____________ UnixCCompilerTestCase.test_osx_cc_overrides_ldshared _____________

self = <distutils.tests.test_unixccompiler.UnixCCompilerTestCase testMethod=test_osx_cc_overrides_ldshared>

    @unittest.skipUnless(sys.platform == 'darwin', 'test only relevant for OS X')
    def test_osx_cc_overrides_ldshared(self):
        # Issue #18080:
        # ensure that setting CC env variable also changes default linker
        def gcv(v):
            if v == 'LDSHARED':
                return 'gcc-4.2 -bundle -undefined dynamic_lookup '
            return 'gcc-4.2'
        sysconfig.get_config_var = gcv
        with EnvironmentVarGuard() as env:
            env['CC'] = 'my_cc'
            del env['LDSHARED']
            sysconfig.customize_compiler(self.cc)
>       self.assertEqual(self.cc.linker_so[0], 'my_cc')
E       AssertionError: 'gcc' != 'my_cc'
E       - gcc
E       + my_cc

distutils/tests/test_unixccompiler.py:118: AssertionError
=========================== short test summary info ============================
FAILED distutils/tests/test_unixccompiler.py::UnixCCompilerTestCase::test_osx_cc_overrides_ldshared
================== 1 failed, 261 passed, 24 skipped in 4.12s ===================
ERROR: InvocationError for command /Users/jaraco/code/public/pypa/distutils/.tox/pypy3/bin/pytest (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   pypy3: commands failed

Matti, can you suggest the right fix for this issue? Once I have the tests passing on PyPy3 here, I'll fold that into pypa/setuptools@distutils.

@jaraco
Copy link
Copy Markdown
Member Author

jaraco commented Jul 2, 2020

When I look at sysconfig_pypy, I find it has this diff for customize_compiler:

diff --git a/distutils/sysconfig.py b/distutils/sysconfig.py
index 879b6981ed..39e4a848a2 100644
--- a/distutils/sysconfig.py
+++ b/distutils/sysconfig.py
@@ -205,8 +205,8 @@ def customize_compiler(compiler):
                 _osx_support.customize_compiler(_config_vars)
                 _config_vars['CUSTOMIZED_OSX_COMPILER'] = 'True'
 
-        (cc, cxx, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
-            get_config_vars('CC', 'CXX', 'CFLAGS',
+        (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+            get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
                             'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
 
         if 'CC' in os.environ:
@@ -229,7 +229,7 @@ def customize_compiler(compiler):
         if 'LDFLAGS' in os.environ:
             ldshared = ldshared + ' ' + os.environ['LDFLAGS']
         if 'CFLAGS' in os.environ:
-            cflags = cflags + ' ' + os.environ['CFLAGS']
+            cflags = opt + ' ' + os.environ['CFLAGS']
             ldshared = ldshared + ' ' + os.environ['CFLAGS']
         if 'CPPFLAGS' in os.environ:
             cpp = cpp + ' ' + os.environ['CPPFLAGS']

But if I apply that diff to this branch, not only does it not address the failure, it creates a new one.

@jaraco
Copy link
Copy Markdown
Member Author

jaraco commented Jul 2, 2020

Aha, so looking at the history of sysconfig_pypy, the lingering presence of the use of opt there seems to be an artifact of the original copy (50b42f56b6c5). So the form as found in sysconfig_cpython is probably more correct, but that still doesn't explain why the test is failing.

@jaraco
Copy link
Copy Markdown
Member Author

jaraco commented Jul 2, 2020

It seems the issue happens here when:

(Pdb) cc
'gcc -pthread -arch x86_64'
(Pdb) ldshared 
'gcc -pthread -shared -undefined dynamic_lookup'

Because ldshared doesn't start with 'gcc -pthread -arch x86_64', the override doesn't happen.

On CPython, cc = 'gcc', so the override does happen.

@jaraco
Copy link
Copy Markdown
Member Author

jaraco commented Jul 2, 2020

I'm now all but convinced that test is a poorly-written test. It monkey-patches get_config_var, even though the underlying implementation relies on get_config_vars.

@jaraco
Copy link
Copy Markdown
Member Author

jaraco commented Jul 2, 2020

If I update sysconfig with this diff:

diff --git a/distutils/sysconfig.py b/distutils/sysconfig.py
index 879b6981ed..110ff3a8b4 100644
--- a/distutils/sysconfig.py
+++ b/distutils/sysconfig.py
@@ -205,9 +205,12 @@ def customize_compiler(compiler):
                 _osx_support.customize_compiler(_config_vars)
                 _config_vars['CUSTOMIZED_OSX_COMPILER'] = 'True'
 
-        (cc, cxx, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
-            get_config_vars('CC', 'CXX', 'CFLAGS',
-                            'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
+        (cc, cxx, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = (
+            get_config_var(var)
+            for var in (
+                'CC', 'CXX', 'CFLAGS',
+                'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
+        )
 
         if 'CC' in os.environ:
             newcc = os.environ['CC']

The test passes.

@jaraco
Copy link
Copy Markdown
Member Author

jaraco commented Jul 2, 2020

So the issue isn't pypy-specific, but the flaw is triggered by the conditions that happen to be present in my environment (darwin, Homebrew PyPy build).

@jaraco
In test_unixcompiler.test_osx*, also patch sysconfig.get_config_vars …
c3a052a 
…following the patterns of prior implementations.
@jaraco jaraco merged commit 76d53a8 into master Jul 2, 2020
@jaraco jaraco deleted the feature/integrate-pypy branch July 2, 2020 09:11
@mattip mattip mentioned this pull request Jul 2, 2020
Merged
@jaraco jaraco mentioned this pull request Sep 4, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

Support platform-specific behaviors

Development

Successfully merging this pull request may close these issues.

2 participants

@jaraco @mattip