Skip to content

linux/bootstrap-tools: move libstdc++ out of default library search path#209054

Merged
trofi merged 1 commit intoNixOS:stagingfrom
trofi:stdcxx-out-of-bootstrap-lib
Jan 12, 2023
Merged

linux/bootstrap-tools: move libstdc++ out of default library search path#209054
trofi merged 1 commit intoNixOS:stagingfrom
trofi:stdcxx-out-of-bootstrap-lib

Conversation

@trofi
Copy link
Contributor

@trofi trofi commented Jan 4, 2023

This change allows building new gcc during bootstrap without fear of pulling in outdated libstdc++.so after g++ switched from bootstrapTools to freshly built g++.

Noticed when tried to add early bootstrap stage to rebuild gcc before glibc is fully untangled from bootstrapTools as a failure to built binutils:

ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
/nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'

The change survives existing bootstrap and unblockes early gcc bootstrap.

Description of changes
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

This change allows building new gcc during bootstrap without fear of
pulling in outdated libstdc++.so after g++ switched from bootstrapTools
to freshly built g++.

Noticed when tried to add early bootstrap stage to rebuild `gcc` before
`glibc` is fully untangled from `bootstrapTools` as a failure to built
`binutils`:

    ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
    /nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'

The change survives existing bootstrap and unblockes early `gcc` bootstrap.
@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Jan 4, 2023
@ofborg ofborg bot added 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Jan 4, 2023
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+2 Thank you for figuring this out! We should definitely merge this.

fi

# path to version-specific libraries, like libstdc++.so
LIBSTDCXX_SO_DIR=$(echo $out/lib/gcc/*/*)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked that for all 16 of our existing bootstrap-files tarballs there is only one subdirectory under lib/gcc, and only one subdirectory beneath that (unpack-bootstrap-tools.sh is a huge pain because every time we touch it we have to rebuild the whole universe).

@ghost ghost mentioned this pull request Jan 9, 2023
4 tasks
@trofi trofi requested a review from Ericson2314 January 9, 2023 21:19
@trofi trofi merged commit 4433093 into NixOS:staging Jan 12, 2023
@trofi trofi deleted the stdcxx-out-of-bootstrap-lib branch January 12, 2023 18:28
@trofi
Copy link
Contributor Author

trofi commented Jan 14, 2023

This regressed tests for bootstrap tarball building:

$ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
...
       > /nix/store/82k0qrddaa84i5nl3zv3f3bcadwwcxli-test-bootstrap-tools/bin/bar: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

I think it's a pure test issue. Test does not use any wrappers and assumes certain directory layout.

Looking.

trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 14, 2023
After NixOS#209054 we started moving
libstdc++.so out of default glibc's paths. This exposed bootstrap
tools build failure as:

    $ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
    ...
    >
    .../bin/bar: error while loading shared libraries: libstdc++.so.6:
       cannot open shared object file: No such file or directory

Note that bootstrap itself did not break. The change only expands
handcrafted `-rpath` entries.
@trofi
Copy link
Contributor Author

trofi commented Jan 14, 2023

Proposed the fix as #210752

ConnorBaker pushed a commit to nixos-cuda/cuda-legacy that referenced this pull request Apr 14, 2025
 #### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like NixOS/nixpkgs#132343

 #### Incorporates

- NixOS/nixpkgs#210004
- NixOS/nixpkgs#36948 (unreverted)
- NixOS/nixpkgs#210325
- NixOS/nixpkgs#210118
- NixOS/nixpkgs#210132
- NixOS/nixpkgs#210109
- NixOS/nixpkgs#213909
- NixOS/nixpkgs#216136
- NixOS/nixpkgs#216237
- NixOS/nixpkgs#210019
- NixOS/nixpkgs#216232
- NixOS/nixpkgs#216016
- NixOS/nixpkgs#217977
- NixOS/nixpkgs#217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @Ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: NixOS/nixpkgs#210004
[versioned directory]: NixOS/nixpkgs#209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253
[mentioned]: NixOS/nixpkgs#210112 (comment)
[crisis]: NixOS/nixpkgs#108305
[foreign]: NixOS/nixpkgs#170857 (comment)
[static lib{mpfr,mpc,gmp,isl}.a hack]: https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: stdenv Standard environment 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant