Describe the bug

If I have a package that is configured with depsBuildBuild = [ buildPackages.stdenv.cc ], this package will fail to compile when building with the GCC stdenv and cross-compiling back to the same system. The simplest way to see this is to build something that uses gccStdenv in the pkgsLLVM set, such as pkgsLLVM.kexec-tools, but this will affect other packages too (such as ncurses) if I do a custom cross-compile setup like

crossSystem = {
  system = builtins.currentSystem;
  dummyValueForCrossCompiling = true;
};

The rationale for such a setup is I'm using crossOverlays to swap out glibc for the host system, and I need the crossSystem definition or stdenv just uses the stage 2 glibc instead.

The root cause seems to be that unwrapped GCC always includes target-prefixed binaries even when not cross-compiling.

A quick grep through nixpkgs suggests this issue will affect close to 100 derivations.

Steps To Reproduce

nix build nixpkgs/5ba549eafcf3e33405e5f66decd1a72356632b96#pkgsLLVM.kexec-tools

Build log

last 10 log lines:
> updateAutotoolsGnuConfigScriptsPhase
> Updating Autotools / GNU config script to a newer upstream version: ./config/config.sub
> Updating Autotools / GNU config script to a newer upstream version: ./config/config.guess
> configuring
> configure flags: --prefix=/nix/store/91xaishiyx0q54ljmlppxay2sv4kgn9v-kexec-tools-x86_64-unknown-linux-gnu-2.0.26 BUILD_CC=cc --build=x86_64-unknown-linux-
gnu --host=x86_64-unknown-linux-gnu
> checking for x86_64-unknown-linux-gnu-gcc... x86_64-unknown-linux-gnu-gcc
> checking whether the C compiler works... no
> configure: error: in `/build/kexec-tools-2.0.26':
> configure: error: C compiler cannot create executables
> See `config.log' for more details

If I keep the failed build and inspect the resulting config.log file it fails at

configure:2467: checking whether the C compiler works
configure:2489: x86_64-unknown-linux-gnu-gcc    conftest.c  >&5
/nix/store/rhhll3vwpj38ri72ahrrrvcbkhz4fhh6-binutils-2.40/bin/ld: cannot find crt1.o: No such file or directory
/nix/store/rhhll3vwpj38ri72ahrrrvcbkhz4fhh6-binutils-2.40/bin/ld: cannot find crti.o: No such file or directory
/nix/store/rhhll3vwpj38ri72ahrrrvcbkhz4fhh6-binutils-2.40/bin/ld: cannot find -lgcc_s: No such file or directory
collect2: error: ld returned 1 exit status
configure:2493: $? = 1
configure:2531: result: no

I dug into what's going on here and the problem is that the $PATH for such a derivation ends up looking like

"${buildPackages.stdenv.cc}/bin:${buildPackages.stdenv.cc.cc}/bin:[…]:${buildPackages.stdenv.cc.bintools}/bin:${buildPackages.stdenv.cc.bintools.bintools}/bin:[…]:${stdenv.cc}/bin:${stdenv.cc.cc}/bin:[…]:${stdenv.cc.bintools}/bin:${stdenv.cc.bintools.bintools}/bin:[…]"

Notice here how the buildPackages.stdenv wrapped compiler and unwrapped compiler show up before the stdenv wrapped compiler.

buildPackages.stdenv.cc contains only unprefixed binaries (e.g. gcc), because it only includes the prefix when hostPlatform != targetPlatform and buildPackages.stdenv uses buildPackages.buildPackages.gcc. However the unwrapped compiler does include the target-prefixed binaries. This means that when the package tries to look for $CC it finds the prefixed binary in the unwrapped build compiler instead of finding the expected cross-compiler.

It looks to me like this setup was designed with the expectation that cross-compiling will always go to another system and therefore the target-prefixed binary will always uniquely refer to the host compiler (is that the right name for it?), but when cross-compiling back to the same system it ends up being the unwrapped build compiler instead.

This isn't noticed for most of the packages in pkgsLLVM because the host compiler is clang instead of gcc, it's only packages that use gccStdenv that have the problem, or it's when cross-compiling without setting useLLVM.

Based on this, it seems the root cause is that the unwrapped GCC always includes target-prefixed binaries even when not cross-compiling. Clang doesn't do this (nor does bintools), so I don't understand why GCC does. Because it does that, it means that any derivation that looks for {target}-gcc is going to get a broken compiler.

I noticed in cc-wrapper/default.nix there's a comment suggesting that cc-wrapper should always include the target-prefixed binaries:

This would mask the issue but cause another one. It would mean that {target}-gcc resolves to the build compiler instead of the host compiler. It would be a wrapped compiler, so compilation would work, but it will use the build linker instead of the host linker (which matters for e.g. pkgsLLVM) and it will use the build libc instead of the host libc (which matters for me because I'm swapping out glibc with crossOverlays).

Notify maintainers

@Synthetica9 @vcunat @Ericson2314 @amjoseph-nixpkgs

Metadata

❯ nix run nixpkgs#nix-info -- -m
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.51, NixOS, 23.11 (Tapir), 23.11.20230919.5ba549e`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - nixpkgs: `/etc/nix/inputs/nixpkgs`