Skip to content

Python packages set: major updates#105368

Merged
FRidh merged 398 commits intostagingfrom
python-unstable
Dec 1, 2020
Merged

Python packages set: major updates#105368
FRidh merged 398 commits intostagingfrom
python-unstable

Conversation

@FRidh
Copy link
Member

@FRidh FRidh commented Nov 29, 2020 •
edited
Loading

Motivation for this change

It's been a long time since we did major updates in the whole set. Applications were not automatically updated, but likely need some fixing as well.

https://hydra.nixos.org/jobset/nixpkgs/python-unstable

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@FRidh FRidh changed the base branch from master to staging-next November 29, 2020 14:59
@FRidh FRidh requested a review from jonringer November 29, 2020 14:59
@ofborg ofborg bot added 6.topic: haskell General-purpose, statically typed, purely functional programming language 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: rust General-purpose programming language emphasizing performance, type safety, and concurrency. 6.topic: stdenv Standard environment 8.has: changelog This PR adds or changes release notes 8.has: documentation This PR adds or changes documentation 8.has: module (update) This PR changes an existing module in `nixos/` and removed 6.topic: haskell General-purpose, statically typed, purely functional programming language 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: rust General-purpose programming language emphasizing performance, type safety, and concurrency. 6.topic: stdenv Standard environment 8.has: changelog This PR adds or changes release notes 8.has: module (update) This PR changes an existing module in `nixos/` labels Nov 29, 2020
@FRidh FRidh mentioned this pull request Nov 29, 2020
Closed
@FRidh
Copy link
Member Author

FRidh commented Nov 29, 2020

numpy needs numpy/numpy#17776

@jonringer
Copy link
Contributor

jonringer commented Nov 29, 2020

the use of flit_core is making gimp fail because python2Packages.flit_core is disabled, which failing the eval as gimp is a darwin constituent https://gist.github.com/GrahamcOfBorg/f0aa782e418d7d426a7bf7a5a7372186

@jonringer
Copy link
Contributor

jonringer commented Nov 29, 2020

maybe we should revert the packaging bump until they figure out their existence.

@FRidh
Copy link
Member Author

FRidh commented Nov 30, 2020 •
edited
Loading

the use of flit_core is making gimp fail because python2Packages.flit_core is disabled, which failing the eval as gimp is a darwin constituent https://gist.github.com/GrahamcOfBorg/f0aa782e418d7d426a7bf7a5a7372186

In principle the Python 3 flit can make a wheel for Python 2 (they're universal wheels). I've been avoiding this but can use this fact.

Easiest here is to just keep another package for Python 2.

@FRidh
Copy link
Member Author

FRidh commented Nov 30, 2020

In principle the Python 3 flit can make a wheel for Python 2 (they're universal wheels). I've been avoiding this but can use this fact.

I had a quick look at this approach. Fun stuff! So, in principle we could run the flit command, Python 3 version, to build the wheel. However, e.g. packaging uses PEP 517, so pip will attempt to use that method to find the backend. Then, it fails, because it cannot find the Python 3 version. Sigh.

Really the best approach here, if we were caring sufficiently for Python 2, would be to separate the wheel building part into a separate derivation. The installation function could then simply pick the universal wheel built with Python 3. Now I would still like to do that some day, and not for Python 2, but for other reasons, but not anytime soon.

So for now I suggest we keep the older packaging for Python 2 around.

@ofborg ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Nov 30, 2020
@jonringer
Copy link
Contributor

jonringer commented Dec 1, 2020

what do you want to do about:

DEPRECATION: The -b/--build/--build-dir/--build-directory option is deprecated. pip 20.3 will remove support for this functionality. A possible replacement is use the TMPDIR/TEMP/TMP environment variable, possibly combined with --no-clean. You can find discussion regarding this at pypa/pip#8333.

TMPDIR is already set during build, so we could just remove this option if I'm realing this correctly

Jonathan Ringer and others added 17 commits December 1, 2020 14:44
@FRidh
python3Packages.pytest_xdist: use pytest_xdist_2 by default
e0cbbc6 
meant to be used with pytest_6
@FRidh
libgit2: 1.0.0 -> 1.1.0
eb7aa55
@FRidh
python3Packages.jsonrpc-websocket: fix tests
d4f96dd
@FRidh
python3Packages.Pyro4: fix tests
3d3a388
@FRidh
python3Packages.portend: fix build
7308561
@FRidh
python3Packages.requests-unixsocket: fix tests
60671ec
@FRidh
python3Packages.cheroot: fix tests by removing pytest-testmon
f616eaf
@FRidh
python3Packages.secretstorage: 3.1.1 -> 3.3.0
74d73c7
@FRidh
python3Packages.pikepdf: fix tests
7bd3696
@FRidh
python3Packages.setproctitle: ignore impure tests
9cb6244
@FRidh
python3Packages.dask: fix tests
c68fdc1 
Tests cannot be ran concurrently
@FRidh
pgcli: allow for sqlparse 4.x
eb8d121
@FRidh
python3Packages.pyproj: fix tests
7c0fe73
@FRidh
python3Packages.faker: 4.17.1 -> 4.18.0
9a88538
@FRidh
python3Packages.faker: fix tests, remove random2
df11a28
@FRidh
pythonPackages.pip-install-hook: remove --build option
76966f8 
Pip had the option --build to build in a custom or temporary directory.
Nowadays, pip just listens to TMPDIR, which we already set.

This option was deprecated and is removed in pip 20.3.
@FRidh
python3Packages.pip: 20.2.4 -> 20.3
b0e3728
@FRidh FRidh merged commit 9750653 into staging Dec 1, 2020
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Dec 1, 2020
@nixos-discourse
Copy link

nixos-discourse commented Dec 1, 2020

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/upcoming-python-breakage/7554/19

@FRidh
Copy link
Member Author

FRidh commented Dec 1, 2020

Thank you @jonringer. There are about 900 failures left, which is about the same as it was before. I've updated to the latest pip.

@FRidh FRidh mentioned this pull request Dec 1, 2020
Merged
10 tasks
@TredwellGit
Copy link
Member

TredwellGit commented Dec 3, 2020

After this was merged on staging:

$ nixos-rebuild build -I nixpkgs=.
building Nix...
building the system configuration...
error: pytest-xdist-2.1.0 not supported for interpreter python2.7
(use '--show-trace' to show detailed location information)

@TredwellGit
Copy link
Member

TredwellGit commented Dec 3, 2020

Seems to be caused by TeX Live depending on Python 2.

@TredwellGit
Copy link
Member

TredwellGit commented Dec 3, 2020 •
edited
Loading

I have TeX Live working; will request pull later.

@FRidh FRidh mentioned this pull request Dec 4, 2020
Open
@TredwellGit TredwellGit mentioned this pull request Dec 4, 2020
Merged
4 tasks
@meatcar meatcar mentioned this pull request Dec 13, 2020
Merged
10 tasks
@collares collares mentioned this pull request Dec 14, 2020
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SuperSandro2000 SuperSandro2000 SuperSandro2000 left review comments

@jonringer jonringer Awaiting requested review from jonringer

Assignees

No one assigned

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: clean-up This PR removes packages or removes other cruft 8.has: documentation This PR adds or changes documentation 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@FRidh @jonringer @groodt @nixos-discourse @TredwellGit @teto @SuperSandro2000