Skip to content

treewide: migrate to fetchCargoVendor, batch 1#376038

Merged
mweinelt merged 1 commit intoNixOS:stagingfrom
alyssais:fetchCargoVendor-1
Jan 23, 2025
Merged

treewide: migrate to fetchCargoVendor, batch 1#376038
mweinelt merged 1 commit intoNixOS:stagingfrom
alyssais:fetchCargoVendor-1

Conversation

@alyssais
Copy link
Member

Cargo 1.84.0 seems to have changed the output format of cargo vendor again, once again invalidating fetchCargoTarball FOD hashes. It's time to fix this once and for all, switching across the board to fetchCargoVendor, which is not dependent on cargo vendor's output format.

It should be possible to reproduce this diff. To do so, get the list of files changed by this commit, e.g. with git diff --name-only, then run the following two commands, each with that list of files as their standard input:

xargs sed -i 's/^\(. *\)\(cargoHash\)\b/\1useFetchCargoVendor = true;\n\1cargoHash/'
cut -d / -f 4 | xargs -n 1 nix-update --version=skip

This will take a long time. It might be possible to parallelize it using xargs' -P option. I haven't tested it.

When reviewing the diff by eye, I think the biggest thing to watch out for is additions of useFetchCargoVendor that don't correspond to a single cargoHash change. I think I filtered all the packages where that would happen out via manual review, but given there are more than a thousand in this batch it's possible I didn't have a 100% accuracy rate.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@alyssais alyssais requested review from K900, emilazy and mweinelt January 23, 2025 07:53
@github-actions github-actions bot added the 6.topic: xen-project Issues and PRs related to the Xen Project Hypervisor. label Jan 23, 2025
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 23, 2025
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 23, 2025
@alyssais
Copy link
Member Author

This is obviously very prone to merge conflicts. When merge conflicts appear, I'll just drop them from this patch and we can revisit them in a future one.

@github-actions github-actions bot added 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Jan 23, 2025
Cargo 1.84.0 seems to have changed the output format of cargo vendor
again, once again invalidating fetchCargoTarball FOD hashes.  It's
time to fix this once and for all, switching across the board to
fetchCargoVendor, which is not dependent on cargo vendor's output
format.

It should be possible to reproduce this diff.  To do so, get the list
of files changed by this commit, e.g. with git diff --name-only, then
run the following two commands, each with that list of files as their
standard input:

	xargs sed -i 's/^\(. *\)\(cargoHash\)\b/\1useFetchCargoVendor = true;\n\1cargoHash/'
	cut -d / -f 4 | xargs -n 1 nix-update --version=skip

This will take a long time.  It might be possible to parallelize it
using xargs' -P option.  I haven't tested it.
@mweinelt mweinelt merged commit 220bc76 into NixOS:staging Jan 23, 2025
25 of 27 checks passed
@alyssais alyssais deleted the fetchCargoVendor-1 branch January 23, 2025 14:32
@emilazy
Copy link
Member

emilazy commented Jan 24, 2025

The script is eminently reasonable and trivial to review; thank you.

Planning a reproduction attempt on x86_64-darwin (probably won’t help that much to catch platform differences, sadly, as I use a case‐sensitive volume for Nix):

  1. jj new master (I don’t want to base this on staging just for nix-update to build rustc etc. – hopefully enough stuff will match.)
  2. nix shell nixpkgs#gnused
  3. jj diff --name-only -r 63bff8c132d17b960f98d3878d10a410ec584472 | xargs sed -i 's/^\(. *\)\(cargoHash\)\b/\1useFetchCargoVendor = true;\n\1cargoHash/'
  4. jj show – looks sensible. (Not going to dig into the details here, I’m only interested in reproducing the diff.)
  5. jj new
  6. jj diff --name-only -r 63bff8c132d17b960f98d3878d10a410ec584472 | cut -d / -f 4 | xargs -P 8 -n 1 nix-update --version=skip

(and actually I don’t have time to dedicate to babysitting the process right now but I wanted to record my steps for when I do tomorrow or something; it has at least successfully bumped a few hashes that match the commit)

@TomaSajt
Copy link
Contributor

Huh, didn't expect cargo to "break" its own implementation this quick after fetchCargoVendor was added...
fetchCargoVendor should be much more resilient in terms of FOD hashes, hopefully it is an OK "final" solution to this problem.

btw, is there / will there be a tracking issue for discoverability for out-of-tree consumers of nixpkgs?

@alyssais
Copy link
Member Author

There will be a deprecation of fetchCargoTarball.

@eljamm eljamm mentioned this pull request Jan 26, 2025
13 tasks
@alyssais alyssais mentioned this pull request Jan 27, 2025
13 tasks
@CathalMullan CathalMullan mentioned this pull request Feb 6, 2025
13 tasks
@amesgen amesgen mentioned this pull request Feb 7, 2025
13 tasks
@pluiedev pluiedev mentioned this pull request May 30, 2025
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: xen-project Issues and PRs related to the Xen Project Hypervisor. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants