Allow customizing fetchurl hashedMirrors by jherland · Pull Request #409010 · NixOS/nixpkgs

jherland · 2025-05-20T09:32:19Z

Nix users behind a firewall may have difficulties reaching the default mirrors referenced from pkgs/build-support/fetchurl/mirrors.nix. Additionally, Nix users inside corporations may need to prefer company-internal mirrors.

This PR exposes the set of mirrors used by fetchurl via the nixpkgs config, allowing users to customize it via their ~/.config/nixpkgs/config.nix. The default mirror list is made available in config.mirrors, and users may remove/add/replace/adjust the mirrors via the module system, as shown in the documented example.

@infinisil edit: This PR is now changed to only allow customisation of hashedMirrors

Things done

Add a 👍 reaction to pull requests you find important.

emilazy · 2025-05-20T12:41:14Z

I would vaguely prefer to drop the whole mirrors machinery: they’re not very useful in a CDN‐oriented 2025 and it’s not just firewalls that cause issues, since sometimes mirrors lag behind or have other issues. Especially the ftp:// and http:// mirrors are no good to have. Besides which, the vast majority of users are just getting these FODs from the official cache and never touching the original sources anyway, so it doesn’t meaningfully decrease load on the origin servers.

How would you feel about just dropping the mirrors that have caused you issues? I think that for cases where there is one obvious primary mirror, especially one that doesn’t have an explicit warning against using it or that is behind a CDN or load‐balancing redirect, we should just list that mirror. Then when all of these entries have only one URL we can think about dropping the mechanism entirely. If you wanted to make that happen, that would be great, but otherwise maybe you can just drop specific problematic ones?

emilazy · 2025-05-20T12:48:56Z

Okay I just looked at the mirrors list again and got annoyed so I’m going to tackle trimming it down myself. Hopefully you can look at the resulting PR and let me know if it will fix your issue here.

jherland · 2025-05-20T13:47:22Z

How would you feel about just dropping the mirrors that have caused you issues?

Although dropping problematic mirrors would help some cases, allowing customizations to the set of mirrors remains a primary concern for us: We're working in a company scenario where access to external urls is deliberately blocked by firewalls, and package sources must be fetched through company-internal mirrors.

The customizations we want to apply to the mirror list include e.g. replacing hashedMirrors = ["https://tarballs.nixos.org"] with hashedMirrors = [COMPANY_INTERNAL_MIRROR]. In short, changes that would not be appropriate to merge into nixpkgs.

Adding this customization point allows us to get away with a configuration change rather than having to patch nixpkgs in our deployment.

Then when all of these entries have only one URL we can think about dropping the mechanism entirely.

Do you mean dropping the mirror:// scheme altogether? Even if each mirror only has one URL associated with it, I still appreciate the abstraction it offers, especially for mirrors that are used by a lot of packages.

emilazy · 2025-05-20T16:07:03Z

I’ve put up #409110.

Although dropping problematic mirrors would help some cases, allowing customizations to the set of mirrors remains a primary concern for us: We're working in a company scenario where access to external urls is deliberately blocked by firewalls, and package sources must be fetched through company-internal mirrors.

The customizations we want to apply to the mirror list include e.g. replacing hashedMirrors = ["https://tarballs.nixos.org"] with hashedMirrors = [COMPANY_INTERNAL_MIRROR]. In short, changes that would not be appropriate to merge into nixpkgs.

Right. But the vast majority of packages don’t use the mirror:// mechanism at all; we’re increasingly preferring to fetch sources from GitHub and other source repositories over release tarballs, for instance. So you need a solution that works for packages that don’t use mirror:// anyway.

hashedMirrors is a different kind of thing IMO; a customization point for that would make sense to me. For the rest, I’m not sure I see the value: you can’t avoid downloading external URLs even if you override all of these mirrors. I think the best solution there is to set up a selective caching HTTPS proxy, which is something Nix/Nixpkgs should support quite well at this point.

The remaining value I see in mirror:// is in templating mirrors that are used across the tree and that either might change or need fallbacks for things that get moved to an archive, yeah. But I don’t think there’s all too much of that; it’s hard to imagine ftp.gnu.org will ever move, and if it did we could just do a tree‐wide find and replace. The intersection of “fairly widely used throughout the tree” and “might require a fallback archive URL” is pretty much Linux distros, and even for Debian that’s only some ~40 uses throughout the tree, and they’d probably be happier with a fetchurl wrapper handling it anyway given the fairly baroque URL format. So honestly I don’t think there’s much reason to keep it around.

oxij · 2025-05-21T06:04:26Z

To quote #409110 (comment) which is relevant here:

Also, philosophically, I disagree with #409010 (comment) and #409010 (comment). The fact that CDNs are common in 2025 is not an argument for making Nixpkgs less resilient to censorship and networking issues. IMHO, with CDNs and GitHub centralization we should be adding more mirrors and implement the ability to specify alternative non-GitHub srcs for fetchFromGitHub, when such things are available. Which, for many projects, they are.

The fact that several companies can single-handedly prevent you from updating your machines does not seem that nice of a thing to me.

balsoft · 2025-05-22T11:30:16Z

hashedMirrors is a different kind of thing IMO; a customization point for that would make sense to me. For the rest, I’m not sure I see the value: you can’t avoid downloading external URLs even if you override all of these mirrors. I think the best solution there is to set up a selective caching HTTPS proxy, which is something Nix/Nixpkgs should support quite well at this point.

It's currently handled by the same mechanism, and adding an override specifically for it is more work than adding an override for the entire mirror list, all for less functionality. If in the future we want to get rid of non-hashed mirrors entirely, this PR doesn't make it any more difficult to do.

philiptaron · 2025-07-23T20:25:29Z

I'm loathe to merge this since it's my belief that #410186 allows your usecases to complete without this complication of the mirror system.

balsoft · 2025-09-19T10:20:55Z

@philiptaron

I'm loathe to merge this since it's my belief that #410186 allows your usecases to complete without this complication of the mirror system.

Actually, it's not possible to use rewriteURL for hashedMirrors; the only way to override them right now is with an impure environment variable, which doesn't play well with multi-user Nix and is in general quite hacky. I still think this PR should be merged, as it allows for customization of hashedMirrors; even if we drop non-hashed mirrors in the future, this would still be useful for multi-user Nix setups (or otherwise to avoid setting the env variable).

Another possibility is to change rewriteURL to allow rewriting URLs "inside" the mirror list; this would be a bit more complex than the implementation in this PR (which is pretty much just taking the mirrors attrset from a config option instead of a hardcoded file), and potentially more difficult to change in the future if we get rid of mirrors entirely.

infinisil

It's currently handled by the same mechanism, and adding an override specifically for it is more work than adding an override for the entire mirror list, all for less functionality. If in the future we want to get rid of non-hashed mirrors entirely, this PR doesn't make it any more difficult to do.

I have to disagree with that:

Making only hashedMirrors overridable is not substantially harder than making all mirrors overridable, see suggestion below
The goal should not be to have the most functionality, but to have functionality according to what we need. And @jherland confirmed that they only need hashedMirrors to be overridable:

The customizations we want to apply to the mirror list include e.g. replacing hashedMirrors = ["https://tarballs.nixos.org"] with hashedMirrors = [COMPANY_INTERNAL_MIRROR].
This PR in its current state does make it harder to get rid of non-hashed mirrors in the future, because it introduces a public interface for them, which comes with expectation of stability and deprecation periods, which does require maintenance and consideration to handle properly.

So, I think we should just expose an override interface for hashedMirrors, which is also a great step towards avoiding conflation with mirrors. The fact that those are specified with the same attribute set is really just an implementation detail anyways, it's not fit for an interface.

If there is also a requirement to override non-hashed mirrors (now or in the future), a separate PR would be more appropriate to not conflate the discussions of the two any more.

PS: I'd like to refer to C4, which I hope to see used more, point 2.3.2:

A patch SHOULD be a minimal and accurate answer to exactly one identified and agreed problem.

infinisil · 2025-09-23T23:08:35Z

pkgs/top-level/config.nix

+    mirrors = mkOption {
+      type = types.attrsOf (types.listOf types.str);


Only make hashedMirrors customisable, and set the default using the module system. Other code needs adjustments.

Suggested change

mirrors = mkOption {

type = types.attrsOf (types.listOf types.str);

hashedMirrors = mkOption {

type = types.listOf types.str;

default = [ "https://tarballs.nixos.org" ];

Description should mention https://github.com/NixOS/nixpkgs/blob/master/maintainers/scripts/copy-tarballs.pl

infinisil · 2025-09-23T23:26:06Z

pkgs/top-level/all-packages.nix

+  mirrors = config.mirrors;
+


There shouldn't be a need to expose this as a top-level attribute.

Suggested change

mirrors = config.mirrors;

infinisil · 2025-09-23T23:28:54Z

pkgs/build-support/fetchurl/default.nix

 }:

 let
-
-  mirrors = import ./mirrors.nix;


This will be just

mirrors = import ./mirrors.nix // { inherit hashedMirrors; };

I also suggest setting hashedMirrors in mirrors.nix to:

hashedMirrors = throw "Use config.hashedMirrors instead of (import ./pkgs/build-support/fetchurl/mirrors.nix).hashedMirrors"

Hopefully nobody did that, but if they did it's a free good error message, and at the same time prevents people from trying to make mirror://hashedMirrors/... work.

Hopefully nobody did that, but if they did it's a free good error message, and at the same time prevents people from trying to make mirror://hashedMirrors/... work.

Unless I'm missing something, this change would have no effect on mirror://hashedMirrors; we would have to separate the two entirely (which I'm in favor of).

Was missing from abda866 While the native stdenv's eval seems to be broken anyways, the freebsd one at least evaluates again with this

Allows having alternate hashed mirrors as fallbacks. Useful in case the default hashed mirror is not accessible or doesn't have everything needed. Co-authored-by: Johan Herland <[email protected]> Co-authored-by: Yuriy Taraday <[email protected]> Co-authored-by: Alexander Bantyev <[email protected]>

infinisil · 2025-10-06T12:16:56Z

I now pushed this to change it to just supporting hashedMirrors customisation, along with some other minor fixes. The subsequent force push is for rebasing after conflicts with #445592, since I also added a test case.

philiptaron · 2025-10-06T16:22:29Z

pkgs/build-support/fetchurl/default.nix

My memory from #414769 is that there's an equivalent change needed in pkgs/build-support/fetchurl/booter.nix and minimal-bootstrap for any change in pkgs/build-support/fetchurl/default.nix.

rewriteURL made sense to apply to fetchurlBoot because it's an eval-time feature and fetchurlBoot supported mirror:// too. hashedMirrors however is a build-time feature that fetchurlBoot doesn't support, so can't be applied here. I tested the command in the PR, works without issues.

github-actions bot added 6.topic: fetch Fetchers (e.g. fetchgit, fetchsvn, ...) 6.topic: stdenv Standard environment labels May 20, 2025

jherland force-pushed the jherland/allow-custom-fetchurl-mirrors branch from 61f2eca to c80c516 Compare May 20, 2025 09:32

github-project-automation bot added this to Stdenv May 20, 2025

github-actions bot added the 8.has: documentation This PR adds or changes documentation label May 20, 2025

nix-owners bot requested review from Artturin, Ericson2314, RossComputerGuy, emilazy, philiptaron, reckenrode and toonn May 20, 2025 09:33

github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels May 20, 2025

NixOSInfra added the 12.first-time contribution This PR is the author's first one; please be gentle! label May 20, 2025

emilazy mentioned this pull request May 20, 2025

fetchurl: remove most mirrors #409110

Open

13 tasks

jherland mentioned this pull request May 23, 2025

fetchurl: Add hook for rewriting/filtering URLs #410186

Merged

14 tasks

jherland force-pushed the jherland/allow-custom-fetchurl-mirrors branch 4 times, most recently from 2168363 to 3a64d1f Compare June 6, 2025 09:56

nixpkgs-ci bot removed the 12.first-time contribution This PR is the author's first one; please be gentle! label Jun 25, 2025

jherland assigned balsoft Jul 31, 2025

infinisil requested changes Sep 23, 2025

View reviewed changes

github-project-automation bot moved this to In Progress in Stdenv Sep 23, 2025

infinisil force-pushed the jherland/allow-custom-fetchurl-mirrors branch from 3a64d1f to 3b1e78c Compare October 6, 2025 12:12

infinisil changed the title ~~Allow customizing fetchurl mirrors~~ Allow customizing fetchurl hashedMirrors Oct 6, 2025

infinisil and others added 2 commits October 6, 2025 14:14

stdenv: Pass rewriteURL to freebsd and native stdenvs

c0004e6

Was missing from abda866 While the native stdenv's eval seems to be broken anyways, the freebsd one at least evaluates again with this

infinisil force-pushed the jherland/allow-custom-fetchurl-mirrors branch from 3b1e78c to 56f680d Compare October 6, 2025 12:15

infinisil approved these changes Oct 6, 2025

View reviewed changes

nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Oct 6, 2025

balsoft approved these changes Oct 6, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Oct 6, 2025

philiptaron reviewed Oct 6, 2025

View reviewed changes

philiptaron approved these changes Oct 6, 2025

View reviewed changes

philiptaron added this pull request to the merge queue Oct 6, 2025

Merged via the queue into NixOS:master with commit 5457a56 Oct 6, 2025
33 of 35 checks passed

github-project-automation bot moved this from In Progress to Done in Stdenv Oct 6, 2025

		mirrors = mkOption {
		type = types.attrsOf (types.listOf types.str);

-    mirrors = mkOption {
-      type = types.attrsOf (types.listOf types.str);
+    hashedMirrors = mkOption {
+      type = types.listOf types.str;
+      default = [ "https://tarballs.nixos.org" ];

Uh oh!

Conversation

jherland commented May 20, 2025 • edited by infinisil Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

emilazy commented May 20, 2025

Uh oh!

emilazy commented May 20, 2025

Uh oh!

jherland commented May 20, 2025

Uh oh!

emilazy commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oxij commented May 21, 2025

Uh oh!

balsoft commented May 22, 2025

Uh oh!

philiptaron commented Jul 23, 2025

Uh oh!

balsoft commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

infinisil left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

infinisil Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

infinisil Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

infinisil Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

balsoft Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

infinisil commented Oct 6, 2025

Uh oh!

philiptaron Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

infinisil Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jherland commented May 20, 2025 •

edited by infinisil

Loading

emilazy commented May 20, 2025 •

edited

Loading

balsoft commented Sep 19, 2025 •

edited

Loading

infinisil left a comment •

edited

Loading