Skip to content

Python updates 2022-02-15#160067

Merged
mweinelt merged 17 commits intostagingfrom
python-updates
Mar 2, 2022
Merged

Python updates 2022-02-15#160067
mweinelt merged 17 commits intostagingfrom
python-updates

Conversation

@mweinelt
Copy link
Member

@mweinelt mweinelt commented Feb 14, 2022

This cycle is about creating a cheap baseline evaluation for the expected setuptools breakage to have a comparison against master.

https://hydra.nixos.org/eval/1744066

In the meantime it has been rebased and retargeted onto staging.

This is closely related to #155516 and we will start updating or removing packages that rely on this feature.

Previous python-unstable: #154969

@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Feb 14, 2022
@mweinelt
Copy link
Member Author

mweinelt commented Feb 14, 2022

Basic evals checks fail in pytest. Bisected to setuptools 60.9.0 bump, so I stuck with 60.8.2 for now and reported the test failures upstream. pytest-dev/pytest#9688

_____________________________ test_version_verbose _____________________________

pytester = <Pytester PosixPath('/build/pytest-of-nixbld/pytest-0/test_version_verbose0')>
pytestconfig = <_pytest.config.Config object at 0x7ffff6a56580>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7ffff39d9ca0>

    def test_version_verbose(pytester: Pytester, pytestconfig, monkeypatch) -> None:
        monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
        result = pytester.runpytest("--version", "--version")
        assert result.ret == 0
        result.stderr.fnmatch_lines([f"*pytest*{pytest.__version__}*imported from*"])
        if pytestconfig.pluginmanager.list_plugin_distinfo():
>           result.stderr.fnmatch_lines(["*setuptools registered plugins:", "*at*"])
E           Failed: nomatch: '*setuptools registered plugins:'
E               and: 'This is pytest version 6.2.5, imported from /nix/store/jfn7276vsd1l8daa1g14g5qdhdjzn419-python3.9-pytest-6.2.5/lib/python3.9/site-packages/pytest/__init__.py'
E           remains unmatched: '*setuptools registered plugins:'

/build/pytest-6.2.5/testing/test_helpconfig.py:12: Failed
----------------------------- Captured stderr call -----------------------------
This is pytest version 6.2.5, imported from /nix/store/jfn7276vsd1l8daa1g14g5qdhdjzn419-python3.9-pytest-6.2.5/lib/python3.9/site-packages/pytest/__init__.py
=========================== short test summary info ============================
FAILED testing/test_helpconfig.py::test_version_verbose - Failed: nomatch: '*...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!

@dotlambda
Copy link
Member

Where did the pip update go?

@mweinelt
Copy link
Member Author

mweinelt commented Feb 15, 2022

Force pushed it away when I downgraded from 60.9.0 to 60.8.2, it wasn't really relevant and can be added back again later. Want to get one eval so we see what breaks.

@ofborg ofborg bot added 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Feb 15, 2022
@ofborg ofborg bot requested a review from alunduil February 15, 2022 03:14
@github-actions github-actions bot added 6.topic: golang Go is a high-level general purpose programming language that is statically typed and compiled. 6.topic: printing Drivers, CUPS & Co. 6.topic: qt/kde Object-oriented framework for GUI creation 6.topic: vim Advanced text editor labels Feb 15, 2022
@mweinelt mweinelt changed the base branch from master to staging February 15, 2022 14:55
@github-actions github-actions bot removed 6.topic: qt/kde Object-oriented framework for GUI creation 6.topic: printing Drivers, CUPS & Co. 6.topic: golang Go is a high-level general purpose programming language that is statically typed and compiled. 6.topic: vim Advanced text editor labels Feb 15, 2022
@mweinelt mweinelt marked this pull request as ready for review February 15, 2022 16:01
@ofborg ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. 8.has: clean-up This PR removes packages or removes other cruft and removed 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. labels Feb 15, 2022
@risicle
Copy link
Contributor

risicle commented Feb 20, 2022

Have built quite a lot of packages with this on macos 10.15 and am now only getting pre-existing failures.

mweinelt and others added 17 commits March 2, 2022 20:57
The library was renamed and this package didn't follow … yet?
Including pytest-xdist without configuring it has no effect...
setuptools 60.x (re-)introduces a bundled distutils, which we need
to patch with C++ support in the same way as we do with the stdlib's
distutils (mostly to placate clang on macos)
This is the tool that manages our certificate store.
@mweinelt
Copy link
Member Author

mweinelt commented Mar 2, 2022

The state of things right now is looking pretty good.
https://shells.darmstadt.ccc.de/~hexa/1744686.html

Will rebase now, to drop the portpicker commit that conflicts.

@mweinelt mweinelt merged commit 718c9d6 into staging Mar 2, 2022
@mweinelt
Copy link
Member Author

mweinelt commented Mar 2, 2022

Thanks @risicle for coming up with that patch. That was a big chunk of the heavy-lifting to get this PR merged!

@ofborg ofborg bot requested a review from jonringer March 2, 2022 22:25
@trofi
Copy link
Contributor

trofi commented Mar 3, 2022

Bisect claims that 5c09870 python3Packages.setuptools: 57.2.0 -> 60.8.2 broke tests for python3Packages.scikit-build in staging:

$ nix build -f. python3Packages.scikit-build -L
...
python3.9-scikit-build> tests/test_command_line.py::test_hide_listing[True-sdist] FAILED         [ 26%]
python3.9-scikit-build> tests/test_command_line.py::test_hide_listing[True-bdist_wheel] FAILED   [ 27%]
...
python3.9-scikit-build> =================================== FAILURES ===================================
python3.9-scikit-build> ________________________ test_hide_listing[True-sdist] _________________________
python3.9-scikit-build> action = 'sdist', hide_listing = True
python3.9-scikit-build> capfd = <_pytest.capture.CaptureFixture object at 0x7ffff495b1f0>
python3.9-scikit-build>     @pytest.mark.parametrize("action", ['sdist', 'bdist_wheel'])
python3.9-scikit-build>     @pytest.mark.parametrize("hide_listing", [True, False])
python3.9-scikit-build>     def test_hide_listing(action, hide_listing, capfd):
python3.9-scikit-build>         cmd = [action]
python3.9-scikit-build>         if hide_listing:
python3.9-scikit-build>             cmd.insert(0, "--hide-listing")
python3.9-scikit-build>         @project_setup_py_test("test-hide-listing", cmd, verbose_git=False, disable_languages_test=True)
python3.9-scikit-build>         def run():
python3.9-scikit-build>             pass
python3.9-scikit-build>         run()
python3.9-scikit-build>         out, _ = capfd.readouterr()
python3.9-scikit-build>         if hide_listing:
python3.9-scikit-build> >           assert to_platform_path("bonjour/__init__.py") not in out
python3.9-scikit-build> E           AssertionError: assert 'bonjour/__init__.py' not in 'running sdi... under it)\n'
python3.9-scikit-build> E             'bonjour/__init__.py' is contained here:
python3.9-scikit-build> E               running sdist
python3.9-scikit-build> E               running generate_source_manifest
python3.9-scikit-build> E               running check
python3.9-scikit-build> E               reading manifest file 'MANIFEST'
python3.9-scikit-build> E               creating hello-1.2.3
python3.9-scikit-build> E               creating hello-1.2.3/bonjour...
python3.9-scikit-build> E
python3.9-scikit-build> E             ...Full output truncated (19 lines hidden), use '-vv' to show
python3.9-scikit-build> _          = "hint: Using 'master' as the name for the initial branch. This default branch name\nhint: is subject to change. To con...d meta-data: url\n\nwarning: check: missing meta-data: if 'author' supplied, 'author_email' should be supplied too\n\n"
python3.9-scikit-build> action     = 'sdist'
python3.9-scikit-build> capfd      = <_pytest.capture.CaptureFixture object at 0x7ffff495b1f0>
python3.9-scikit-build> cmd        = ['--hide-listing', 'sdist']
python3.9-scikit-build> hide_listing = True
python3.9-scikit-build> out        = "running sdist\nrunning generate_source_manifest\nrunning check\nreading manifest file 'MANIFEST'\ncreating hello-1.2....nd adding 'hello-1.2.3' to it\ncreating dist\nCreating tar archive\nremoving 'hello-1.2.3' (and everything under it)\n"
python3.9-scikit-build> run        = <function test_hide_listing.<locals>.run at 0x7ffff46da310>
python3.9-scikit-build> tests/test_command_line.py:176: AssertionError
...
python3.9-scikit-build> FAILED tests/test_command_line.py::test_hide_listing[True-sdist] - AssertionE...
python3.9-scikit-build> FAILED tests/test_command_line.py::test_hide_listing[True-bdist_wheel] - asse...

@risicle
Copy link
Contributor

risicle commented Mar 3, 2022

scikit-build/scikit-build#623

Looks like we're quite behind on scikit-build anyway.

@risicle
Copy link
Contributor

risicle commented Mar 3, 2022

In fact the next "Python updates" includes a bump that may well fix it ada1d70

@mweinelt
Copy link
Member Author

mweinelt commented Mar 3, 2022

I see two different tests failing with that update, maybe we should just disable those.

@nixos-discourse
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: clean-up This PR removes packages or removes other cruft 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants