Skip to content

fetchdata and fetchmulti#19540

Closed
zimbatm wants to merge 6 commits intoNixOS:masterfrom
zimbatm:fetchmulti
Closed

fetchdata and fetchmulti#19540
zimbatm wants to merge 6 commits intoNixOS:masterfrom
zimbatm:fetchmulti

Conversation

@zimbatm
Copy link
Member

@zimbatm zimbatm commented Oct 14, 2016

Motivation for this change

Moving the fetch data into pure NIXON code means we can more easily write update scripts. The updating part is still missing though.

The second motivation is that it's currently painful to update sha256s with multi-arch sources (binary packages). With fetchmulti it's possible to do nix-prefetch-url . -A mypackage.src.all and get all the hashes out.

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • OS X
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

/cc @garbas @kamilchm @domenkozar

@mention-bot
Copy link

@zimbatm, thanks for your PR! By analyzing the history of the files in this pull request, we identified @krgn, @wkennington and @peterhoeg to be potential reviewers.

@garbas
Copy link
Member

garbas commented Oct 14, 2016

👍

i imagine there are many other situations where fetchmulti could be used. is there a way to grep for those cases?

also do we need to document this? are other fetch* functions documented?

@edolstra
Copy link
Member

The name fetchmulti implies that it fetches multiple files. But if I understand it correctly, it fetches just one file from a selection. So it should be something like fetchOneOf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> no function calls.

Copy link
Member

@edolstra edolstra Oct 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing functions as strings seems really ugly to me...

Also, there is nothing in the fetchdata function specific to fetching. Rather it's a generic "call a function specified by the 'fetcher' argument". But such a function doesn't sound like a good idea to me. Why write f { fetcher = "g"; args... } when I can write g args directly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, not very nice. Clearly the goal is to be able to write in "nixon", i.e. without function calls.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be more compelling if I replaced it with a type attribute such that type = "git" would map to the fetchgit fetcher and so on?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whooaa! Cool!!

@zimbatm
Copy link
Member Author

zimbatm commented Oct 15, 2016

Thanks for the reviews! I realize my vision isn't complete here and it's more of a 20% review.

The problem I am trying to solve is that it takes a lot of time to update packages. Most of the time it's enough to bump the version and update the sha256 but right now it's hard to automate that because the data is mixed with the mkDerivation bits. First bump the version, then run nix-prefetch-url and copy-and-paste the sha256 back into the file. My thinking is that if that data would be externalized then it would make things easier to automate.

Right now all I am doing in this PR is to move all the data into a pure nix (NIXON) declaration (which explain why fetcher is also a string) with the idea that in a further PRs I will be able to move that out to a separate file. I also converted a couple of files to fetchOneOf to demonstrate how it would work but there are a lot left to do.

I want us to get into a situation where the source data can live in a separate file (eg: fetchdata import ./source.nix) and have a tool that's able to update that external file either automatically by finding the list of upstream releases or manually with the user giving the version that he wants. If we have that we can also notify the maintainers of new releases and get a better sense of which parts of nixpkgs need more work.

If we can agree on the fetcher as a key and that interface in general I am happy to just propose the fetchOneOf fetcher in this PR and wait for the more complete picture to form before introducing the fetchdata function into nixpkgs.

What do you think?

@zimbatm
Copy link
Member Author

zimbatm commented Oct 15, 2016

@garbas for now my approach to finding multi-arch binaries is to grep for x86_64.*\.tar but it's not perfect

@zimbatm zimbatm mentioned this pull request Oct 15, 2016
7 tasks
@zimbatm
Copy link
Member Author

zimbatm commented Oct 15, 2016

Moving the discussion to #19582. I will come back with a more complete solution with my updating script idea.

@zimbatm zimbatm closed this Oct 15, 2016
@Mic92 Mic92 mentioned this pull request Jan 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants