Skip to content

Python updates 2022-03-02#162538

Merged
mweinelt merged 403 commits intostagingfrom
python-updates
Mar 14, 2022
Merged

Python updates 2022-03-02#162538
mweinelt merged 403 commits intostagingfrom
python-updates

Conversation

@mweinelt
Copy link
Member

@mweinelt mweinelt commented Mar 2, 2022

This is a major python update cycle that we do every once in a while. The last major update cycle started around mid-january (#154969).

https://hydra.nixos.org/job/nixpkgs/python-updates/tested#tabs-constituents

Things to watch out for:

  • AWS packages (awscli, boto3, botocore, moto) package have very narrow version constraints
  • Updates of hypothesis can be fragile for a huge number of package tests
    • 6.39.0 caused issues with pytest-asyncio, downgraded to 6.38.0
  • slackclient before version 3.0, after that it's called slack-sdk, they're two distinct packages
    • false update dropped

Evaluations will appear shortly over at https://hydra.nixos.org/jobset/nixpkgs/python-updates.

The previous cycle can be found in #160067.

@mweinelt mweinelt requested review from FRidh and jonringer as code owners March 2, 2022 21:46
@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Mar 2, 2022
@ofborg ofborg bot added 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Mar 2, 2022
@mweinelt mweinelt force-pushed the python-updates branch 2 times, most recently from 5803fbf to ac9342a Compare March 2, 2022 23:50
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 3, 2022
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 3, 2022
@mweinelt
Copy link
Member Author

mweinelt commented Mar 3, 2022

https://shells.darmstadt.ccc.de/~hexa/1746506.html

  • sphinx fixed by dropping docutils upgrade
  • aiohttp fixed by disabling tests that trigger deprecation warnings in pytest 7
  • pytest on python3.10 by dropping a patch that wasn't required anymore

will trigger another eval shortly

@mweinelt
Copy link
Member Author

mweinelt commented Mar 3, 2022

  • Dropped the astroid==2.10.0 bump to fix pylint and asttokens

@mweinelt mweinelt force-pushed the python-updates branch 3 times, most recently from 9505ffe to 30a47ea Compare March 3, 2022 15:14
@mweinelt
Copy link
Member Author

mweinelt commented Mar 3, 2022

  • Fixed httpcore by relaxing h11 constraint and introducing socksio

@mweinelt
Copy link
Member Author

mweinelt commented Mar 3, 2022

@DavHau Can you please check on scikit-learn? There's some history of this build failing over in #156578.

@ofborg ofborg bot added the 8.has: package (new) This PR adds a new package label Mar 3, 2022
@mweinelt
Copy link
Member Author

mweinelt commented Mar 3, 2022

@ofborg ofborg bot added the ofborg-internal-error Ofborg encountered an error label Mar 3, 2022
@jonringer
Copy link
Contributor

@GrahamcOfBorg eval

@risicle
Copy link
Contributor

risicle commented Mar 3, 2022

Hmm. On nixos x86_64 joblib fails:

________ test_nested_parallel_warnings[threading-multiprocessing-True] _________

parent_backend = 'threading', child_backend = 'multiprocessing', expected = True

    @with_multiprocessing
    @parametrize('parent_backend,child_backend,expected', [
        ('loky', 'multiprocessing', True), ('loky', 'loky', False),
        ('multiprocessing', 'multiprocessing', True),
        ('multiprocessing', 'loky', True),
        ('threading', 'multiprocessing', True),
        ('threading', 'loky', True),
    ])
    def test_nested_parallel_warnings(parent_backend, child_backend, expected):
    
        # no warnings if inner_n_jobs=1
>       Parallel(n_jobs=2, backend=parent_backend)(
            delayed(_assert_warning_nested)(
                backend=child_backend, inner_n_jobs=1,
                expected=False)
            for _ in range(5))

joblib/test/test_parallel.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
joblib/parallel.py:1056: in __call__
    self.retrieve()
joblib/parallel.py:935: in retrieve
    self._output.extend(job.get(timeout=self.timeout))
/nix/store/adnh88kc6cf2dc8fqfyvjmyyd86v52qn-python3-3.9.10/lib/python3.9/multiprocessing/pool.py:771: in get
    raise self._value
/nix/store/adnh88kc6cf2dc8fqfyvjmyyd86v52qn-python3-3.9.10/lib/python3.9/multiprocessing/pool.py:125: in worker
    result = (True, func(*args, **kwds))
joblib/_parallel_backends.py:595: in __call__
    return self.func(*args, **kwargs)
joblib/parallel.py:262: in __call__
    return [func(*args, **kwargs)
joblib/parallel.py:262: in <listcomp>
    return [func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

backend = 'multiprocessing', inner_n_jobs = 1, expected = False

    def _assert_warning_nested(backend, inner_n_jobs, expected):
        with warns(None) as records:
            parallel_func(backend=backend, inner_n_jobs=inner_n_jobs)
    
        if expected:
            # with threading, we might see more that one records
            if len(records) > 0:
                return 'backed parallel loops cannot' in records[0].message.args[0]
            return False
        else:
>           assert len(records) == 0
E           assert 2 == 0
E            +  where 2 = len(WarningsChecker(record=True))

joblib/test/test_parallel.py:202: AssertionError

preventing scikit-learn building.

@risicle
Copy link
Contributor

risicle commented Mar 3, 2022

Wait, scratch that - looks like it was just flakiness.

@ofborg ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 2.status: merge conflict This PR has merge conflicts with the target branch labels Mar 3, 2022
mweinelt and others added 12 commits March 14, 2022 00:29
- use pyproject build
- disable flaky test case
- prune dependencies
- create documentation output
- update meta
We don't have that dependency packaged.
This reverts commit 940af4b.

Causes problems with scikit-learn, details can be found over in
#156578.
This reverts commit 6cc7b47.

Disabling these tests hides crashing issues that are encountered in the
test suite. They are ultimately caused by an update to numpy>=1.22.0
that was reverted so that these tests can be reenabled.
@mweinelt mweinelt merged commit a39a3e6 into staging Mar 14, 2022
@risicle
Copy link
Contributor

risicle commented Mar 14, 2022

Urgh pytest-mpl giving grief here, even once provided with jinja2 as it asks. Its tests get very meta and dependent on renderer details. Would say we should call in the maintainer but I know he's awol.

@SuperSandro2000
Copy link
Member

@akaWolf ?

@ofek
Copy link
Contributor

ofek commented Mar 14, 2022

How often does staging get merged into master?

@jonringer
Copy link
Contributor

How often does staging get merged into master?

I try to help with staging, and I don't even know. There's a lot of potential blockers. On average though, every 2 weeks. give or take a week

@ofek
Copy link
Contributor

ofek commented Mar 14, 2022

Got it, thanks!

@mofrim
Copy link
Contributor

mofrim commented Apr 10, 2022

hello! after upgrading python3Packages.ipympl to v0.8.8 with this merge build of this package fails because it lacks matplotlib as buildinput. i already opened a pr (#168148) to this.

@Shados
Copy link
Member

Shados commented Apr 13, 2022

@mweinelt The updated python3Packages.levenshtein version fails to build, and that shows on the hydra jobs for it... why was it included in the merge?

EDIT: Looks like even with that one reverted, puddletag fails to build due having an exact version req on lxml==4.7.1, hahh.

@mweinelt
Copy link
Member Author

why was it included in the merge?

Because this is a bulk update run, and we're trying our best to keep the package set stable and up to date. That doesn't work out in all cases sadly.

Let's continue working on this in #168547 or revert it that fails.

@NorfairKing
Copy link
Contributor

Looks like this PR was relevant to an issue that I've had: nix-community/poetry2nix#605
I've made an issue about backporting it: #171447

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python3-11-tensorflow-2-12-how-to/30156/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

None yet

Development

Successfully merging this pull request may close these issues.