@edolstra @Ericson2314 Can one of you review this change? We're carrying a large-ish workaround in firefox to work around this issue.

I don't think the rest of our infrastructure (index-debuginfo.cc, dwarffs) handles non-standard build IDs, so this change is not enough to support those.

BTW, 64 bits seems rather insufficient as a unique identifier... I wonder why LLVM went with such short IDs?

Given the birthday paradox, this means we'll only need a few billion binaries (hardly unimaginable given the size of the binary cache) until we can expect collisions.

I don't expect that it's so bad. Existence of a collision shouldn't cause issues by itself, if the code is written reasonably. Probability of a collision for a hash that you need should still be fairly low, I think.

Existence of a collision shouldn't cause issues by itself, if the code is written reasonably.

The hash is used as a key to look up debug info, so a collision means that we get the wrong debug info.

Yes, of course. But existence of a collision in a huge repo only breaks info on (some of) the colliding files, nowhere else. There will still be very very few of the collisions.

I'd like to test this, how can I get an affected llvm stdenv ? I tried compiling sl with llvmPackages.stdenv but it still has long build ids, so presumably it's still using a partly gnu toolchain.

Perhaps not a solution but: you can get lld to produce longer build-ids with --build-id=md5, --build-id=uuid, --build-id=sha1:

[nix-shell:~]$ nix-shell --expr "with (import <nixpkgs> {}); (mkShell.override { stdenv = pkgsLLVM.stdenv; }) {}" --run '$CC -Wl,--build-id=fast -xc - <<<"int main() { }" -o test; $READELF -n test | grep Build'
    Build ID: 8903a1c09635bcb6

[nix-shell:~]$ nix-shell --expr "with (import <nixpkgs> {}); (mkShell.override { stdenv = pkgsLLVM.stdenv; }) {}" --run '$CC -Wl,--build-id=md5 -xc - <<<"int main() { }" -o test; $READELF -n test | grep Build'
    Build ID: f10ec65cd2789197314b9025851c6b18

[nix-shell:~]$ nix-shell --expr "with (import <nixpkgs> {}); (mkShell.override { stdenv = pkgsLLVM.stdenv; }) {}" --run '$CC -Wl,--build-id=sha1 -xc - <<<"int main() { }" -o test; $READELF -n test | grep Build'
    Build ID: f0793470c0198f9f167186a1a0e15b7c60cb2e68

[nix-shell:~]$ nix-shell --expr "with (import <nixpkgs> {}); (mkShell.override { stdenv = pkgsLLVM.stdenv; }) {}" --run '$CC -Wl,--build-id=uuid -xc - <<<"int main() { }" -o test; $READELF -n test | grep Build'
    Build ID: 8a00108ec7d22857a47b23eba1f481ee

(I haven't done any benchmarking but I believe fast uses xxhash so there's presumably some perf cost to using sha1, md5 instead of it; uuid is just random so we probably don't want to use it for the sake of reproducibility)

Maybe we can change this:

to use --build-id=sha1 (seems to be what GNU ld defaults to; it also accepts --build-id=sha1)

lld and GNU ld also let you specify an ID to use manually with --build-id=0x..... (with the requirement that you have an even number of digits), maybe we could use the derivation's out base hash here? Downside is that all binaries in a derivation will have the same build-id which seems unacceptable.

(thanks to @rrbutani I could obtain such an executable with short build id as follows:

with import ./. { };
(sl.override { stdenv = pkgsLLVM.stdenv; }).overrideAttrs (old:
{
  separateDebugInfo = true;
  nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [ pkgs/build-support/setup-hooks/separate-debug-info.sh ];
})

)

About tooling: nixseparatedebuginfod can handle such short buildids.

About collisions: I agree with @vcunat that even if hydra may one day contain collisions, the probability that the debuginfo you need is affected is quite low. To give concrete numbers I have currently 18618 distinct build ids on my system, and if I'm doing the math right, the probability that I'm affected if I asked nixseparatedebuginfod to download the debuginfo for them all is 1-product(1-k/(2^64), k, 1, 18618) so 9.39592755809e-12. This is low enough for me. For this reason we can accept such short build ids.

But it would be nice if we can have lld produce the same length of build ids as ld as @rrbutani suggests is possible.

Concretely I am in favor of this change, modulo the bash syntax for comparison, and the fact that we still disable separatedebuginfo for the llvm stdenv (

We can change the kind of buildids lld produces later if deemed desirable.

I personally prefer the alternative approach #218301 to make ld.ldd produce SHA1. This fits more with our existing infrastructures, and looks cleaner to me.