I'm very much in favor of this change, but it's a complete PITA to test it. @oxij made a stab at this in #391262 and I found it really hard to review -- I had to basically test all the users through nix-build --check whatever.src.

“fetchurl: remove unused mirrors” should be trivial after the relevant PRs are merged.

“fetchurl: remove most secondary mirrors” should be simple to review, since it only drops mirrors and clearly leaves canonical ones, unless you’re concerned that some of them may be out of date in a way that happens to preserve an old package. “fetchurl: remove unencrypted SageMath mirrors” similarly.

“fetchurl: use working mirror for OSDN” should also not be too bad, since you can see that the canonical mirror is dead anyway (and there are barely any users).

So it’s only the commits to use canonical mirrors that I can see causing an issue. I observed that the directory structure was clearly identical as far as I could see between the sites getting replaced, and that they’re what the relevant upstreams point to on their sites. In some cases there are even redirects. If there’s a specific change you’re worried about I can make suggestions on how to test it.

Some thoughts on why we should probably remove this mechanism entirely in the future in #409010 (comment), by the way.

My concern (and it may be that it's a silly one) is that some mirrors deliver different contents, and that I, the merger, need to be able to certify that this isn't the case, for each derivation which uses the mirror mechanism. Since these are FODs, I need to use --check, and I have to use grepping to find the users. That's the painful part: the manual and one-by-one nature of the business.

I've been tracking the "remove the mirrors" discourse and agree with your take.

some mirrors deliver different contents, and that I, the merger, need to be able to certify that this isn't the case

How does this apply to cases where we’re only removing mirrors?

How does this apply to cases where we’re only removing mirrors?

My reading of the code in pkgs/build-support/fetchurl/builder.sh says that the first mirror which works will be used. So when removing mirrors, we're exposing a mirror which previously wasn't used to use.

My reading of the code in pkgs/build-support/fetchurl/builder.sh says that the first mirror which works will be used. So when removing mirrors, we're exposing a mirror which previously wasn't used to use.

I guess I don’t really understand. We have multiple mirrors in the first place based on the idea that connections to one of the earlier ones might fail, so I don’t think “previously wasn’t used” is strictly correct.

Bu this worry only applies to cases where we’re deleting ones other than the first mirror, right? In that case, are you worried about accidental broken links, or supply chain attacks? Everything pins a hash – are you worried about things we have hashes based on mirrors previously returning incorrect data, or are you worried about the newly canonical mirrors returning incorrect hashes? (I’m not really sure what the latter would mean conceptually given that I’ve tried to prefer the most upstream canonical URLs.)

I guess it's not achievable as of now with the way fetchurl works but in theory, if mirrors of any kind will be kept in the long term, it's maybe best to make sure there's always the "official canonical" repo listed first and then make sure that when it's the first fetch make sure we are actually fetching from official sources. But then again, github/gitlab/sr.ht is probably a better place to fetch from anyways.

Edit: maybe something simpler like WARNING: not fetching from first mirror in list could go a long way for nixpkgs contribs?

I guess I don’t really understand. We have multiple mirrors in the first place based on the idea that connections to one of the earlier ones might fail, so I don’t think “previously wasn’t used” is strictly correct.

Here's what I am (very slightly) worried about.

Before this patch, assuming that the first mirror in the list returned the FOD-hash-appropriate contents, a nix-build --check whatever.src would succeed.

Let's use as our example pig which is hosted at mirror://apache/pig/${pname}-${version}/${pname}-${version}.tar.gz. When I run nix-build --check pig.src, I'm not in fact checking whether every mirror returns this information -- just the first one.

So all I really know is whether https://dlcdn.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz works -- not whether ftp://ftp.funet.fi/pub/mirrors/apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz worked, let alone any of the others.

After this patch, I'm checking to see whether https://downloads.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz works.

The testing I want to perform is this:

1. For every derivation which uses any mirror modified in this PR
2. For every entry in the mirror list
3. Does that realized URL correctly download something that matches the given hash in the derivation

I'm willing to relax this and do a bunch of spot checks then merge. But I want to be able to do the full thing.

Here's what I am (very slightly) worried about.

Before this patch, assuming that the first mirror in the list returned the FOD-hash-appropriate contents, a nix-build --check whatever.src would succeed.

Let's use as our example pig which is hosted at mirror://apache/pig/${pname}-${version}/${pname}-${version}.tar.gz. When I run nix-build --check pig.src, I'm not in fact checking whether every mirror returns this information -- just the first one.

So all I really know is whether https://dlcdn.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz works -- not whether ftp://ftp.funet.fi/pub/mirrors/apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz worked, let alone any of the others.

After this patch, I'm checking to see whether https://downloads.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz works.

(I am talking here only about dropping mirrors, not about the other changes to mirrors in some of the commits.)

Semantically, I would not quite say that is true. You are checking the first one if it works. The only reason we have the backups is that it might not work. The mirror list only makes sense if we assume these properties:

1. The first link may not always work (because of transient conditions on either side of the connection, or because a link went 404).
2. Any links that do work will return the same hash.

Otherwise, if a link fails to work because of transient conditions, then you can fall back to a mirror that has the wrong hash. And therefore my response is: this problem already exists; it just might not be easy to observe.

Note that the vast majority of these FODs will already be cache. Some of them will likely have already died upstream without anyone noticing yet. So we do already have that kind of problem before this PR.

Anyway, let’s consider a few cases:

1. Previously we got a correct‐contents 200 from a non‐canonical mirror but the canonical mirrors return a 404 because the thing was removed and I either messed up leaving in archival URLs or the content was only surviving because a mirror lagged seriously behind. This is pretty bad but relying on a random mirror for preservation like this is also pretty bad anyway. I agree this case would be worth checking for but these kinds of 404s are common in Nixpkgs anyway because links die. I suspect there are few if any cases like this because it requires weird mirror stuff to happen on top of a normal 404 condition.

2. Previously we got an incorrect‐contents 200 from a non‐canonical mirror but the canonical mirrors return a correct‐contents 200. This would be bad – it means the mirror was wrong, maybe even in a security‐relevant way! – but this is an existing problem and removing the non‐canonical mirrors makes it strictly more exposed, because it’s now more likely that you will observe the hash mismatch (that was already existing, and possible to observe under certain conditions).

3. Previously we got a correct‐contents 200 from a non‐canonical mirror but the canonical mirrors return an incorrect‐contents 200. I don’t know what this would mean given the basic trust model of mirrors: if cdn.kernel.org returns a different hash to a mirror then the mirror is wrong or something is very bad. So I consider this case somewhat incoherent.

4. Previously we got a 404 from a non‐canonical mirror (that didn’t happen in the past, or we wouldn’t have a hash recorded for the FOD at all, I guess?) but now we get a 200 from a canonical mirror. Yay, this is good: we fixed the FOD.

So the only thing I really worry about here is (1), because (2) makes a pre‐existing problem strictly more exposed, (3) makes no sense to me, and (4) is good. But I also don’t think (1) is any worse than the general Nixpkgs link rot problem (that FOD caching protects somewhat against in practice), and I also think it’s less likely.

That’s why I am only worried about changing mirrors, not removing them (as long as the ones being removed are non‐canonical and canonical ones remain). We already do not know for sure which mirror will be selected (that’s the whole point of the mechanism), so every problem removing mirrors creates already exists stochastically; the only model that makes sense for maintaining lists of mirrors implies that removing them is safe except for the (1) case; the FOD cache buffers against any problem in practice; and any potential changes seem to me either desirable or both uncommon and pedestrian in nature.

The testing I want to perform is this:

1. For every derivation which uses any mirror modified in this PR
2. For every entry in the mirror list
3. Does that realized URL correctly download something that matches the given hash in the derivation

I'm willing to relax this and do a bunch of spot checks then merge. But I want to be able to do the full thing.

Sure, I don’t oppose doing this, but I think it’ll be somewhat of a pain. There’s 1124 files in by-name that use mirror://, so those would be pretty easy to check if you assume it’s always in src (not necessarily a safe assumption). But then there’s also a bunch of other stuff, the entire CPAN/CRAN set, etc.

Perhaps you could edit fetchurl to make some change in the .drv when a mirror:// URL is used, generate a list of rebuilds, and then build only the fetchurl FODs. I think that would be comprehensive.

also

3. The user is running nix-build on a host from a country that a particular mirror does not want to work with, or the user has all of its traffic routed through TOR.
   In this case, default mirrors frequently refuse to work, and thus alternatives are very useful.
   In fact, I know for certain that the removal of https://download.savannah.nongnu.org/releases/ line will break over-TOR builds for packages hosted on Savannah.

Well, sure. But countries have blocked GitHub in the past, as well, and fetchFromGitHub is unavoidable. Having reliable access to resources on the internet is a necessary precondition of building packages for a distribution that sources them from thousands of external URLs. If you don’t have that then you need to establish it to have any hope of rebuilding Nixpkgs package sets. Otherwise – well, we do run our own global caching mirror for all these FOD sources that you can get them from instead.

Tor is a different case, since it is of course one of the primary ways of obtaining less filtered internet access. However, I just successfully downloaded https://download-mirror.savannah.gnu.org/releases/freetype/freetype-2.13.3.tar.xz over Tor, so if there was any issue it seems to have been resolved? (And of course there’s no guarantee that the redirector wouldn’t point you to a Tor‐blocking mirror anyway.)

4. When the /etc/ssl/certs/ca-certificates.crt is outdated, like when updating a really old NixOS system to a new release, the HTTPS mirrors might not work.
   Thus, removal of unencrypted mirrors will break such updates (and tarballs.nixos.org won't help!).

That’s just unreasonable. There are many critical packages in Nixpkgs that have sources specified solely with HTTPS URLs. Getting an up‐to‐date certificates list is the one of the first steps in reviving such an ancient system. I cannot imagine any situation where working around it by randomly having HTTP and FTP URLs for some small semi‐random subset of packages solves this problem for anyone, since they will inevitably have to deal with HTTPS anyway.

Well, sure. But countries have blocked GitHub in the past, as well, and fetchFromGitHub is unavoidable. Having reliable access to resources on the internet is a necessary precondition of building packages for a distribution that sources them from thousands of external URLs. If you don’t have that then you need to establish it to have any hope of rebuilding Nixpkgs package sets. Otherwise – well, we do run our own global caching mirror for all these FOD sources that you can get them from instead.

tarballs.nixos.org is not a panacea, many things simply don't get cached there, other things vanish from it from time to time (when it gets out of disk space, I assume), and when they do, having a working mirror is useful. And vice versa, when things are blocked, tarballs.nixos.org frequently helps.

Thus, I don't see how the above is an argument for removing mirrors or why removal of working mirrors could be a good thing.

Tor is a different case, since it is of course one of the primary ways of obtaining less filtered internet access. However, I just successfully downloaded https://download-mirror.savannah.gnu.org/releases/freetype/freetype-2.13.3.tar.xz over Tor, so if there was any issue it seems to have been resolved? (And of course there’s no guarantee that the redirector wouldn’t point you to a Tor‐blocking mirror anyway.)

Well, TOR exit node is a bit of a lottery, isn't it?

Maybe I simply get unlucky, but on some of my build machines builds fail all the time because gnu.org drops all connections from some countries.
I added nongnu.org line there mainly for that reason.

That’s just unreasonable. There are many critical packages in Nixpkgs that have sources specified solely with HTTPS URLs. ...

Making updates to outdated systems easier does not seem unreasonable to me. In fact, the fact that pkgs.cacert can not be built without working certs at the moment feels like an issue to me. I would rather we fix it instead of making it worse.

Also, let's not forget

Whether the canonical mirror or the dispatcher, when available, should be the first element in the list is unclear to me. I assume that the dispatchers going first has a technical reason and I would be uncomfortable with changing it without knowing the effects of such a change.

Also, philosophically, I disagree with #409010 (comment) and #409010 (comment). The fact that CDNs are common in 2025 is not an argument for making Nixpkgs less resilient to censorship and networking issues. IMHO, with CDNs and GitHub centralization we should be adding more mirrors and implement the ability to specify alternative non-GitHub srcs for fetchFromGitHub, when such things are available. Which, for many projects, they are.

The fact that several companies can single-handedly prevent you from updating your machines does not seem that nice of a thing to me.

In fact, the fact that pkgs.cacert can not be built without working certs at the moment feels like an issue to me.

I think that, if your system is that horribly out of date (I don't know exactly how old it has to be, but I assume over 5 years or so for it to really matter a lot), you should've updated way earlier, or just "re-bootstrap" it:

• Get a more up-to-date system
• Download https://channels.nixos.org/nixos-$latestrelease/latest-nixos-minimal-$arch-linux.iso
• Flash it to USB
• Boot on old system
• Rebuild the system configuration on the channel of the live system (you will probably run into issues if you haven't touched your config for that long)

Also, how can building such a basic security-critical package be done without working certs, like at all? If you want to fallback to HTTP instead, you are ultimately trusting that you are lucky and you didn't get MITM'd. (there is no way to know if you have bogus certs)

Edit: I agree on you with the Tor stuff tho. No reason to make nixpkgs explicitly support "less of it"...

@dtomvan Re updates:

HTTP does not hurt there, after all, the hashes get checked anyway.

Last time I needed to do it, I did nix store query --outpath $(nix-instantiate -A pkgs.cacert), then nix copy'id that path from the Hydra cache, set security.pki.certificateFiles = [ path ] to it, nixos-rebuild switch to get an old system with up to date certs, and then updated Nixpkgs/configuration.nix and then nixos-rebuild switch to build the new system.

Which is simpler, IMHO.

But it would have been nicer if I could simply nixos-rebulid the new system instead.