Note that we are currently trying our best to remove as many older GHC versions as possible since keeping them around increases the maintenance burden not just for the Haskell subsystem, but also LLVM since some backends use pretty old versions. Currently, we are looking at removing everything < 9.4.

This change could possibly work as we could add the old GHCs purely for bootstrapping with the NCG backend of GHC for powerpc. However, I am shivering at the amount of code and patches added in this PR and not necessarily looking forward to keeping this working.

I think the most promising solution for increasing is using the fact that we can cross compile GHC 9.4 completely. The idea would be to create “haskell bootstrap tools” archives from that for other platforms. This would also solve the issue for riscv64-linux etc. However, this is an unproven theory at the moment.

I completely understand the concerns and shivers here, especially since this isn't a very popular architecture anymore 😅.

I'd be fine with keeping this a draft for now. I remember that I started looking into this because some Haskell package got pulled in somewhere while building things beyond the basic stdenv.

I think the most promising solution for increasing is using the fact that we can cross compile GHC 9.4 completely. The idea would be to create “haskell bootstrap tools” archives from that for other platforms. This would also solve the issue for riscv64-linux etc. However, this is an unproven theory at the moment.

Is there an issue / existing discussion for this somewhere? Would love to know if that idea ever gets somewhere, so I can adjust this accordingly.

You can try pkgsCross.ppc64-elfv1.haskell.compiler.ghc84 and see how far it gets. Since it has a native code generator, there’s a reasonable chance it’ll even produce a usable GHC that can be used to bootstrap later versions natively. (I suppose you’d probably want to arrange to give it natively‐compiled LLVM tools as a dependency, etc., to shrink the size of the closure that is cross‐compiled, but that’s not required for a proof‐of‐concept.)

(Came here via #440271 and can affirm that the idea of keeping all these older GHCs working scares me, especially if LLVM becomes involved. For POWER that shouldn’t matter, but then it doesn’t seem obvious that having this chain in‐tree is worth it for POWER over uploading a cross‐compiled GHC to tarballs.nixos.org or the like. I am sympathetic to source‐based bootstrapping concerns, but in this case I believe that nobody has ever compiled a GHC that can’t eventually be traced to some unexplained binary from the 90s, so IMO the version proliferation here is not worth tracing back to a slightly older binary (and even then, I somewhat prefer the approach used by our in‐tree minimal bootstrap of having separate stripped‐down derivations that do the minimum required to get to the next stage rather than burdening the wider package set). However, I am not a Haskell maintainer, so this is just an unsolicited opinion with no real weight.)

You can try pkgsCross.ppc64-elfv1.haskell.compiler.ghc84 and see how far it gets.

If the Ampere machine is still around at NixCon tomorrow, then I'll see if I can give that a shot on there. Otherwise that'll have to wait until after NixCon, cus my laptop is prolly too puny to handle a build like that in a timely manner ^^".

You can try pkgsCross.ppc64-elfv1.haskell.compiler.ghc84 and see how far it gets.

ghc94, not ghc98. Starting with GHC 9.6, the build system does no longer allow cross compiling GHC (though this may get fixed eventually).

The big question for you is of course whether the result can actually be used on power, testing the build can be done by almost anyone. We were successful with riscv64-linux in the past.

I am sympathetic to source‐based bootstrapping concerns, but in this case I believe that nobody has ever compiled a GHC that can’t eventually be traced to some unexplained binary from the 90s, so IMO the version proliferation here is not worth tracing back to a slightly older binary

The risk is also that these old binaries randomly stop working as happened with ppc64le, so a solution for getting fresh-ish binaries would be great. (Amjoseph determined that re-using other distribution's underexplained binaries is not an option since GHC is not sufficiently relocatable and patching in the /nix/store paths was not always feasible. However, this may have changed since.)

ghc94, not ghc98.

I actually write ghc84, so we both typo’d :)

FWIW, @OPNA2608, it shouldn’t be meaningfully more expensive than any other given GHC build, unless you mean that we don’t have the cross toolchain for it cached, in which case I think it would be reasonable to ensure that gets built on Hydra like we do many other cross toolchains. And yes, I successfully cross‐compiled an unregisterised GHC 9.4 for RISC‐V when testing #440271; since PowerPC shouldn’t even involve LLVM (something I forgot when talking about injecting dependencies) and was presumably, uh, registerised before 9.4, I expect it should hopefully go quite smoothly. (Also, consider applying for access to the community builders if you’d like time‐shared access to big machines to test Nixpkgs builds on? They have an Ampere, although of course one must be responsible with the shared resources.)

I guess what you’d actually want to cross‐build is a static haskell.compiler.ghc94, so that it can be fetched from a FOD without Nixpkgs dependencies? I’m not sure if GHC supports being built statically, though, especially since that would involve Musl. FODs that have Nix store dependencies are kind of awkward. I suppose you could treat it like the iSO or similar and just pack the entire closure in, and then do a bunch of splitting‐out and patchelf(1)ing. @sternenseemann probably has better ideas than me for how to productionize this.

The risk is also that these old binaries randomly stop working as happened with ppc64le, so a solution for getting fresh-ish binaries would be great. (Amjoseph determined that re-using other distribution's underexplained binaries is not an option since GHC is not sufficiently relocatable and patching in the /nix/store paths was not always feasible. However, this may have changed since.)

Yeah, I mean – I’d like to see everything fully bootstrapped from source, ideally. I’d just also like clean separation of “ancient versions for bootstrap paths” and “fully‐supported members of the main package set”. Unfortunately, my understanding is that a fresh fully‐bootstrapped GHC build has never been accomplished and that it’s unclear what the path to making it work would even be. My guess is that it would involve a reasonably competent Haskell implementation in a language that already has a reasonable bootstrapping path, but it seemed like getting Hugs to work was too much, and what Haskeller is going to willingly write a compiler in a normie language?

The other nice thing about a second‐sourced implementation of a reasonably modern version of a language is that you can skip the “compile 20 versions in a row” nonsense and jump to the latest compiler reasonably quickly. But it doesn’t seem to me like this will happen any time soon, so practical concerns dominate, and using existing bindists or creating them when they don’t exist sounds like the right medium‐term path to me.

I assume the plan is that if bindists bitrot you can just do the same thing you’d do for cross but for native and treat them as another set of “GHC bootstrap tools”, possibly even bumping them semi‐regularly like we do the existing bootstrap tarballs?

FWIW, @OPNA2608, it shouldn’t be meaningfully more expensive than any other given GHC build

(Also, consider applying for access to the community builders if you’d like time‐shared access to big machines to test Nixpkgs builds on? They have an Ampere, although of course one must be responsible with the shared resources.)

I have a decently beefy workstation at home that I usually do such things on, It just didn't fit into my backpack. Or it would've crushed my back :).

I only suggested Ampere specifically because there were mentions of one of those being at NixCon, and I imagine it'd be a fair lot beefier than my PineBook Pro without a battery to support higher power draw. I managed to sit down and mess around abit with that machine on Sunday, but most of that time went into trying to get networking... working, and we didn't get the build done in time.

I'll leave the community builders to community members with bigger needs, for more important things, and for more urgent stuff. Right now this was just bad timing for me, I just didn't expect to be asked to do that big of a build during NixCon ^^".

I'll be back home from NixCon later today, and give it a try on my workstation at home. My return trip has been quite a mess thanks to DB, so I'll try to empty my head in abit and get some more rest...

unless you mean that we don’t have the cross toolchain for it cached

Checked on nixpkgs-unstable a day or two ago, didn't seem cached - wanted to build powerpc64-unknown-linux-gnuabielfv1-gcc etc on my local machine. The toolchain for the ELFv2 target wasn't cached either. But I'm on an aarch64-linux machine, so that might not help... Cross-compiling from ARM to something else seems like abit of an unusual thing to do :)

in which case I think it would be reasonable to ensure that gets built on Hydra like we do many other cross toolchains

Do you happen to know what would need to be done to enable that? Or if those toolchains are only getting prebuilt for x86_64, and not ARM... I guess that's fine then.

I guess what you’d actually want to cross‐build is a static haskell.compiler.ghc94, so that it can be fetched from a FOD without Nixpkgs dependencies? I’m not sure if GHC supports being built statically, though, especially since that would involve Musl.

Gave pkgsStatic.haskell.compiler.ghc94 a try last night:

"inplace/bin/mkdirhier" ghc/stage2/build/tmp//.
/nix/store/bdxk8s5xs8zwf9ciimj5hv3kk0s3ygs8-python3.13-sphinx-8.2.3/bin/sphinx-build -b html -d docs/users_guide/.doctrees-html -w docs/users_guide/.log -n -D latex_paper_size=letter docs/users_guide docs/users_guide/build-html/users_guide
/nix/store/bdxk8s5xs8zwf9ciimj5hv3kk0s3ygs8-python3.13-sphinx-8.2.3/bin/sphinx-build -b man -d docs/users_guide/.doctrees-man docs/users_guide docs/users_guide/build-man
Traceback (most recent call last):
Traceback (most recent call last):
  File "/nix/store/bdxk8s5xs8zwf9ciimj5hv3kk0s3ygs8-python3.13-sphinx-8.2.3/bin/.sphinx-build-wrapped", line 9, in <module>
    sys.exit(main())
             ~~~~^^
  File "/nix/store/bdxk8s5xs8zwf9ciimj5hv3kk0s3ygs8-python3.13-sphinx-8.2.3/bin/.sphinx-build-wrapped", line 9, in <module>
    sys.exit(main())
             ~~~~^^
  File "/nix/store/bdxk8s5xs8zwf9ciimj5hv3kk0s3ygs8-python3.13-sphinx-8.2.3/lib/python3.13/site-packages/sphinx/cmd/build.py", line 476, in main
    locale.setlocale(locale.LC_ALL, '')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/nix/store/bdxk8s5xs8zwf9ciimj5hv3kk0s3ygs8-python3.13-sphinx-8.2.3/lib/python3.13/site-packages/sphinx/cmd/build.py", line 476, in main
    locale.setlocale(locale.LC_ALL, '')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/nix/store/bv3hgziq4mv07whsa3df1d5zh72lxrbk-python3-3.13.6/lib/python3.13/locale.py", line 615, in setlocale
    return _setlocale(category, locale)
  File "/nix/store/bv3hgziq4mv07whsa3df1d5zh72lxrbk-python3-3.13.6/lib/python3.13/locale.py", line 615, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
locale.Error: unsupported locale setting
make[1]: *** [docs/users_guide/ghc.mk:16: docs/users_guide/build-html/users_guide/index.html] Error 1

I almost have a fix for the hashable dontCheck, I'm just missing the spoons to fix that up rn (and I'm unsure if upstream would even accept it, given their responses on the bug report…)

All the crypton-related dontChecks are quite unfortunate indeed though.

I almost have a fix for the hashable dontCheck, I'm just missing the spoons to fix that up rn (and I'm unsure if upstream would even accept it, given their responses on the bug report…)

I'm fine with this being a follow up.

All the crypton-related dontChecks are quite unfortunate indeed though.

I think we'll just have to accept this for now. crypton's tests have been plagued by various issues even on something as mundane as aarch64… I'm not sure if there's a good way to make this more transparent to users, but oh well.

I still need to give a static build a try, but doing bootstrapping via cross-compiled & tarball'd static GHC builds might be doable after all. Though I should prolly not be the person that figures out all the details of that ^^".

I started working on this, but got a little stuck in terms of which direction to go and also don't have a ton of time at the moment, but hopefully I'll figure all of that out eventually.

@sternenseemann formatting was resolved. Does this need anything else, or is it waiting on smth?

(Please let me know if there's a better place to discuss this.)

I still need to give a static build a try, but doing bootstrapping via cross-compiled & tarball'd static GHC builds might be doable after all. Though I should prolly not be the person that figures out all the details of that ^^".

I started working on this, but got a little stuck in terms of which direction to go and also don't have a ton of time at the moment, but hopefully I'll figure all of that out eventually.

I am also interested in getting cross bootstrapping working and merged. My system is using musl as system libc (nixpkgs.hostPlatform = "powerpc64le-linux-musl"), so the Debian binary package will not work easily. I was able to successfully bootstrap a native GHC from aarch64-linux-musl on nixos-25.05 with the following steps:

1. Cross-compile an unregisterised GHC 9.2.1 (to avoid https://gitlab.haskell.org/ghc/ghc/-/issues/21914) with the following overlay:

ghc921 = let
  flake = config.nix.registry.nixpkgs.flake;
  pkgFunc = import "${flake}/pkgs/development/compilers/ghc/common-make-native-bignum.nix" {
    version = "9.2.1";
    sha256 = "sha256-9EQBL5ehNtmUD3fN/wP9pI+UdeLtD+yWbE01xN9V90Y=";
  };
in (self.callPackage pkgFunc {
  bootPkgs = self.pkgsBuildBuild.haskell.packages.ghc8107Binary;
  buildTargetLlvmPackages = self.pkgsBuildTarget.llvmPackages_12;
  llvmPackages = self.llvmPackages_12;
  python3 = self.buildPackages.python311;
  inherit (self.buildPackages.python311Packages) sphinx;
  inherit (self.buildPackages.darwin) autoSignDarwinBinariesHook xattr;

  enableDocs = false;
  enableNativeBignum = true;
  enableShared = false;
  enableUnregisterised = true;
  libffi = null;
}).overrideAttrs (prev: {
  # One patch doesn't apply to GHC 9.2.1
  patches = lib.filter (patch: ! lib.hasInfix "fcompact-unwind" patch) prev.patches;

  # Required for glibc->musl cross, fixed already in nixos-25.11
  preConfigure = prev.preConfigure + ''
    export LOCALE_ARCHIVE="${self.buildPackages.glibcLocales}/lib/locale/locale-archive"
  '';
});

2. Create a tarball from that store path and make it available to the powerpc64le system. The binaries in this tarball only link to musl and libiconv. I didn't try to force static linking because the destination system already had the musl dynamic linker available. I could try making the bootstrap binaries fully static, but maybe this isn't necessary since musl is only a small dependency?
3. Replace GHC 9.2.8 in the bootstrap chain with the binary distribution:

ghc928 = self.stdenv.mkDerivation rec {
  dontBuild = true;

  installPhase = ''
    lib=$out/lib/ghc-${version}
    mkdir $out
    cp -a * $out
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      $lib/bin/*
    substituteInPlace \
      $out/bin/{{ghc,ghc-pkg,ghci,runghc}-${version},hp2ps,hpc,hsc2hs} \
      --replace-fail /nix/store/6jpk1jfwk9q34rkbrqgsm51q6qrk2y25-bash-interactive-powerpc64le-unknown-linux-musl-5.2p37 ${self.bash}
    substituteInPlace \
      $out/bin/{{ghc,ghc-pkg,ghci,runghc}-${version},hp2ps,hpc,hsc2hs} \
      $lib/package.conf.d/*.conf \
      --replace-fail /nix/store/y3lyzyix0k5z261x3f037bp4hv3wvnfv-ghc-musl-native-bignum-powerpc64le-unknown-linux-musl-9.2.1 $out
    substituteInPlace \
      $lib/package.conf.d/base-*.conf \
      $out/nix-support/propagated-target-target-deps \
      --replace-fail /nix/store/2ziimb6c5bl6nxvvdnv98mvr9xmdg4g1-musl-iconv-1.2.5 ${self.libiconv}
    substituteInPlace \
      $lib/settings \
      --replace-fail /nix/store/fr65r5hmph1ppwsfmrcy2avpm0f3nzif-gcc-wrapper-14.3.0 ${self.stdenv.cc} \
      --replace-fail /nix/store/dvcs0j71x7rrsv22p4xjn2avyjyiymdl-binutils-wrapper-2.44 ${self.stdenv.cc.bintools}
    rm $lib/package.conf.d/package.cache
    $out/bin/ghc-pkg recache --global
  '';

  meta.license = lib.licenses.bsd3;

  passthru = {
    enableShared = false;
    haskellCompilerName = "ghc-${version}";
    targetPrefix = "";
  };

  pname = "ghc-binary-cross";
  sourceRoot = ".";

  src = self.fetchurl {
    hash = "sha256-XgDmtoWXPRPT5u717oSEqMep3VjbVyWYcff2DlIAYEY=";
    url = "https://distfiles.smaeul.xyz/nix/ghc-musl-native-bignum-powerpc64le-unknown-linux-musl-${version}.tar.xz";
  };

  version = "9.2.1";
};

4. Allow hadrian to build with an unregisterised GHC:

ghc963 = compiler.ghc963.override (prev: {
  hadrian = compiler.ghc963.passthru.hadrian.overrideAttrs (prev: {
    configureFlags = prev.configureFlags ++ [ "-f-threaded" ];
  });
});

I did not have any issues with failing tests. I plan to try to reproduce this on nixpkgs master, wight GHC 9.4.8 as the cross-compiled bootstrap version.

(Please let me know if there's a better place to discuss this.)

I'm not sure if there's a dedicated issue for this yet. Maybe https://matrix.to/#/#haskell:nixos.org for discussing ideas?

Cross-compile an unregisterised GHC 9.2.1

Cross-compiled registerised GHC also segfaults on ppc64le? Might make sense to just cross-compile it unregisterised by default for 64-bit POWER then.

Smth like:

  enableUnregisterised ?
    # Registerised RV64 compiler produces programs that segfault
    # See https://gitlab.haskell.org/ghc/ghc/-/issues/23957
    (stdenv.hostPlatform.isRiscV64 || stdenv.targetPlatform.isRiscV64)
    # GHC cross-compiled for 64-bit POWER segfaults
    || ((!lib.systems.equals stdenv.buildPlatform stdenv.hostPlatform) && stdenv.hostPlatform.isPower64),

Cross-compiled registerised GHC also segfaults on ppc64le?

Yes, in fact ghc-pkg recache also segfaults, so the derivation for the binary compiler itself fails.

Might make sense to just cross-compile it unregisterised by default for 64-bit POWER then.

Yes, that's a good idea. I'd maybe like to characterize the issue a bit better when submitting a PR, but I'm cross-compiling from aarch64, so it takes several hours to build GHC, and so I haven't spent time on building things I know are broken.

I have been successful with the GHC 9.4.8 on nixos-25.11 cross bootstrap. Here's the build machine side:

ghc948 = compiler.ghc948.override {
  enableDocs = false;
  enableNativeBignum = true;
  enableNuma = false;
  enableProfiledLibs = false;
  enableShared = false;
  enableUnregisterised = true;
  libffi = null;
  stdenv = self.makeStaticBinaries self.stdenv; # this is the new part
}

And here's the target machine side: https://github.com/smaeul/nixpkgs/commits/386472554a34ca5d14ad341c50f82fd3f9205868

The tarball's only library dependency is pkgsMusl.libiconv, which could probably be copied into the tarball, and then you could probably use ${pkgs.musl.dev}/bin/musl-gcc on the target instead of requiring a musl-targeting toolchain (pkgsMusl.stdenv).

My plan is:

1. Integrate nukeReferences so the build script can work with multiple tarballs
2. Build a big-endian tarball, which should cover both glibc and musl hostPlatform
3. Open a pull request
4. Try to figure out how to automate building of trusted tarballs (like make-bootstrap-tools-cross.nix)