Skip to content

Python: autogenerate PyPI source data#14927

Closed
FRidh wants to merge 13 commits intoNixOS:masterfrom
FRidh:pypi
Closed

Python: autogenerate PyPI source data#14927
FRidh wants to merge 13 commits intoNixOS:masterfrom
FRidh:pypi

Conversation

@FRidh
Copy link
Member

@FRidh FRidh commented Apr 23, 2016

Recently PyPI changed their url scheme. Therefore, when a package is updated the entire url needs to be updated. That was it for me. I think we need a way to autogenerate PyPI source data. See also issue #11587. I recommend we discuss autogenerating PyPI data in this issue, and discuss in this PR only implementation details.

To do:

Possible improvements:

  • add a pypi ? False option to determine whether to use PyPI or not. Currently it is determined automatically.
  • parse licenses

Regressions:

  • packages that specify both name and version, and have the version already included in the name will show the version twice now.

@FRidh FRidh added 6.topic: python Python is a high-level, general-purpose programming language. 9.needs: reporter feedback This issue needs the person who filed it to respond 2.status: work-in-progress 9.needs: changelog This PR needs a changelog entry 9.needs: documentation This needs to be documented well. 9.needs: clean-up Somebody please clean up this mess! 6.topic: policy discussion Discuss policies to work in and around Nixpkgs labels Apr 23, 2016
@FRidh FRidh self-assigned this Apr 23, 2016
@FRidh
Copy link
Member Author

FRidh commented Apr 23, 2016

@FRidh
Copy link
Member Author

FRidh commented Apr 23, 2016

So far it only creates the JSON data. License parsing is something for later.

@FRidh FRidh force-pushed the pypi branch 2 times, most recently from d9fe0c3 to 774e101 Compare April 23, 2016 16:43
@FRidh
Copy link
Member Author

FRidh commented Apr 23, 2016

buildPythonPackage now uses by default PyPI. If only a name is given, the latest version is used. If also a version is given, then that version is used. When a src is given, that is used instead.

@FRidh FRidh force-pushed the pypi branch 2 times, most recently from f9d8007 to 4317177 Compare April 23, 2016 17:01
@FRidh FRidh force-pushed the pypi branch 6 times, most recently from b4cac8e to 7edbd75 Compare April 23, 2016 20:04
@FRidh FRidh mentioned this pull request Apr 23, 2016
7 tasks
@FRidh FRidh force-pushed the pypi branch 2 times, most recently from cedaefb to 73a0897 Compare April 24, 2016 09:37
FRidh added 5 commits April 24, 2016 12:06
- `name` should not include the version.
- `version` is optional. When given, the derivation is made for this
specific version. Otherwise the latest version is used.
- Custom `src` can still be given.
@copumpkin
Copy link
Member

If this proves to be too gnarly, I was thinking of a more lightweight option, which is to introduced a new fetchPyPI fixed-output derivation that we can use for all our src attributes, without changing anything else. The fetcher would know how to figure out the correct URL and but would otherwise still produce the same tarballs as we were getting before.

@FRidh
Copy link
Member Author

FRidh commented Apr 25, 2016

A function like fetchPyPI that obtains the url and hash via the JSON API as I do now is indeed a much lighter-weight option. The only downside is having to set the version manually. And it would be a bit more verbose.

Or we could have fetchPyPI and also still generate a list with meta-data and available versions per package. I can also leave buildPythonPackage untouched and create instead say buildPyPIPackage that uses buildPythonPackage.

What we need is to find an agreement on what kind of data we want to have in NIxpkgs and what not.
And as I think @copumpkin wrote somewhere else as well: do we want to have all packages, or a curated list. In case of the latter it would make sense having to explicitly set the version.

@copumpkin
Copy link
Member

@FRidh explicitly picking the version doesn't seem overly onerous, since we do it everywhere else. The other question is a much bigger one though 😄

@FRidh
Copy link
Member Author

FRidh commented Apr 25, 2016

@copumpkin actually, having a fetchPyPI would solve the url scheme change of PyPI, but you would still need to use say nix-prefetch-pypi to obtain the hash and put it in your code. That's something I would like to get rid of.

(fetchPyPI could obtain the hash as I wrote in my previous post, but that would take away most of the use of the hash. A certain Nixpkgs revision wouldn't be unique anymore then either, so let's not do that.)

@FRidh
Copy link
Member Author

FRidh commented Apr 26, 2016

fetchPyPI:
triton/triton@ffcbbf7

@FRidh
Copy link
Member Author

FRidh commented Apr 28, 2016

#15007 is an alternative to the PR where I use a separate function, buildPyPIPackage, to build a PyPI package. I think that is a better solution.

@FRidh FRidh closed this Apr 28, 2016
@domenkozar domenkozar removed the 9.needs: changelog This PR needs a changelog entry label Sep 30, 2016
@FRidh FRidh deleted the pypi branch March 4, 2017 07:43
@FRidh FRidh restored the pypi branch March 4, 2017 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: policy discussion Discuss policies to work in and around Nixpkgs 6.topic: python Python is a high-level, general-purpose programming language. 9.needs: clean-up Somebody please clean up this mess! 9.needs: documentation This needs to be documented well. 9.needs: reporter feedback This issue needs the person who filed it to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants