@garbas, thanks for your PR! By analyzing the history of the files in this pull request, we identified @taku0, @mythmon and @edolstra to be potential reviewers.

Do you want to call this passthru.updateScript or meta.updateScript? It is package metadata, after all.

@ttuegel 👍 makes sense (i'll batch suggestions and then implement them)

Good idea to have a common interface to updating packages.

While you said the Firefox example is still WIP, I wonder whether it is good practice to add dependencies of the update script to the function that builds the derivation. How about using a script with a nix-shell shebang instead?

how about turning the update scripts into own derivations with own files, they could be a separate nix file, be called as a package and have the executable outputting the current data as version

i wonder if there is way to evaluate actual nix paths without putting them into store (so the target/source file can be given as a file

in addition it makes sense to be able to have update scripts as general packages
think of the dozens of packages coming from a apt repository for example (steam + deps, enpass).

also for example python/lua/ruby packages and others do share a very common way to obtain update metadata
it would be really nice to be able to just drop in a generic helpers to most of those and be done with it

Last time I have tried to put a lot of stuff for an auto-updater into expressions, it started annoying people really quickly… So now I put stuff in a separate file near the expression. Maybe the size of the updater means it should be in a separate file. Of course, there are many reusable snippets around that could help reducing it size.

This means that I will just write the updater as «call the existing update script» for my expressions.

Also, do I understand correctly that every update-enabled expression needs its file path, and the example stores it in a non-standard format (pushd in the middle of the code)? If we want to ever refactor the layout (and we probably do), we need to pay attention to separating the «path-to-self» part in a sane way. Or just ask Nix to evaluate it.

@FRidh 👎 on nix-shell shebang. and i tried to use them initially but i quickly needed more then what nix-shell shebang offered.

@RonnyPfannschmidt @7c6f434c (update script in a separate file) as said firefox updateScript is still WIP, right now it was just easier for me to put it all in one file. Otherwise I would expect that if custom update script, and just one liner to create meta.updateScript which calls the updater.

@RonnyPfannschmidt (general update script) I'm sure as we go we will have many common update scripts. example from my current code would be a script which "looks for latest revision in a branch" but i'm sure we'll invent some more

@RonnyPfannschmidt (update multiple derivation) for those derivations where you need to manage versions "in harmony" a namespace should already exists. majority of packages already works this way, eg. pythonPackages, haskelPackages, nodePackages... then you register updateScript on the namespace itself. here are 2 examples update script which updates node and elm packages and update script which updates python packages

@7c6f434c (every updateScript needs to call pushd) I don't like this part either, ideally i would do this pushd/popd from an update.nix but I'm not sure it is possible to know the path of the expression. but i didn't look much into it. I would love to be proven wrong.

i hope i answered all the questions / concerns

@FRidh 👎 on nix-shell shebang. and i tried to use them initially but i quickly needed more then what nix-shell shebang offered.

How so? If I understand the update.nix correctly, all passthru.updateScript (or meta.updateScript but won't that cause a rebuild?) needs to be is a script that knows where the file that src = ... uses is.

Regarding the snippets, we could attach the updateFromGitHub to fetchFromGitHub.update and so have something like passthru.updateScript = src.update;. This we could do with most fetchers I suppose. By default they would write to src.json or src.nix.

mkDerivation {
  src = fetchFromGitHub { }; # The arguments are in `src.nix`. `fetchFromGitHub` has `src ? src.nix`. Or do we still have issues with finding the path?
  passthru.updateScript = src.update; 
};

It is easy to update sha256 in-place (it is unique and it has no
interesting structure inside, after all), and I think source URLs should
refer to version, together with the name. Most of the packages contain
just one version = line. I remember that having src.nix got people
annoyed last when I tried it, and updating the version in-place seems to
work just fine in my current experience with auto-updaters.

Can you link to this discussion? I use a src.nix or srcs.nix for some of my packages. These annoyed people can pry them from my cold, dead hands! 😃

how about having something like fetchFrom* support passing a nix/json file instead of a attributeset
then the default update script for those could use that nix file as metadata source

an alternativ would a surrounding tool that can edit fetchFromGithub expressions that are "simple?

Not sure how to look it up easily… Maybe my need to store both update meta and source data in separate files was the most annoying part.

As I said on the ML, I’d reserve the update attribute until we have found a nice interface for these scripts that integrates into CI in a uniform way. Until then the update scripts could call themselves something else and we can update gracefully once the interface exists.

@ttuegel @FRidh i tried to switch to meta.scriptUpdate but that will require a rebuild. if we stick to passthru.scriptUpdate then build of package is not required in order to run the nix-shell update.nix .... I do think updateScript fits more into meta then passthru but we would have to change something else to get this working in as meat attribute. I think this is out of scope for this PR, but if somebody feels very strongly that this should be in meta then that person should create a PR. I currently care getting this into nixpkgs without much change to existing infrastructure.

@Profpatsch update attribute is not going to be used but passthru.updateScript.

@7c6f434c @FRidh this PR doesn't restrict none of your ideas about having update script with fetchFrom* functions. it is good to have this discussion, i'm just making it clear that nothing blocks this PR.

the code has been rebased on current master and suggestions from @Mic92 were added.

if no further concerns are raised i will merge this on Monday evening hawaii timezone. after that i plan to move my currently used update scripts to nixpkgs (in a separate PR).

Why changing meta would lead to a rebuild? Of course, adding attributes to meta would need to write something in documentation, otherwise it will get randomly removed by people trying to do cleanups…

Could you do something about the pushd part?

I think the script should be attached to the passthru of src instead of the passthru of the derivation we're eventually building, and not meta. What do you think @garbas ? Is this a trivial change to make?

Edit: anyway, it's not that important

update attribute is not going to be used but passthru.updateScript.

Yeah, then I’d reserve both update and updateScript.

@7c6f434c (meta) no idea what is happening to meta, but for the purpose of this PR i'm going with passthru. we can easily change it afterwards.

@7c6f434c (pushd) i don't see this as a blocker for this PR. while i would wish this could be done differently. It looks like instrumenting callPackage or even nix itself would be the way to go, but that would delay this PR unnecessary.

@FRidh I would like it at the expression level, because sometimes you would want to update whole attribute set of packages (eg: using pypi2nix, elm2nix, ...). I already do this in few places outside nixpkgs and would like to bring this to nixpkgs.

@Profpatsch sry, not sure I understand what you mean. this PR uses updateScript. I think it clearly states the intent, and this word does not appear in nixpkgs at all, that means it would be trivial to change it.

I think the script should be attached to the passthru of src instead of the passthru of the derivation we're eventually building, and not meta.

mkDerivation should provide a default updateScript that delegates to the derivation's src attribute. That way if we provide one for fetchFromGitHub then all the related derivations will support it.

Could you do something about the pushd part?

The easiest way would be to split the fetchers so they import from a separate file. Then that file's path can be passed to the updater script. That's something I was exploring over at #19582

finally got around to merge this.

since we are still figuring out how to do update i wont add anything to the manual (at this point) but only write to the mailing list explaining the functionality and show few examples.

i assume before next release we might have something that we could document.

nix-shell update.nix --argstr package firefox-bin results in error: Package with an attribute name `firefox-bin` does have an `passthru.updateScript` defined. while nix-shell update.nix --argstr maintainer garbas works fine. Did I miss something?

I should have passed firefox-bin-unwrapped instead of firefox-bin.

The wrapper script should probably propagate the updateScript though

@zimbatm propagating updateScript results in double update when nix-shell update.nix --argstr maintainer garbas is executed, i.e., both firefox-bin and firefox-bin-unwrapped will be updated.

Good point. Those would be uniquified away in the update.nix file.

• update.nix has executable permission for some reason.

• Can update.nix be moved to a subdirectory (probably maintainers/scripts) in order not to clutter the root directory?