Skip to content

GCC version in manylinux_2_24 #1012

Closed as not planned
Closed as not planned

Description

@henryiii

The GCC version in manylinux1 is 4.8, painfully old, unable to build C++14 code (see #118). It’s actually possible to build GCC 9 on CentOS 5, though support was removed in GCC 10.

The GCC version in manylinux2010 is 8.3, new enough for basically full C++17 support minus a few library features.

The GCC version in manylinux2014 is 9.3, full support for C++17 and even bits of C++20.

The GCC version in manylinux_2_24 is a dismal 6.3, much worse than both the year based manylinuxes and not even new enough for C++17 language support (mostly added in GCC 7)! This is a huge step backward for an image with a higher GLIBC version. (PS: listing the GLIBC versions of the older manylinuxes would be nice, IMO)

Is there some way to replicate the “RHEL dev toolset” in the Debian based images?

Activity

  1. henryiii commented on Feb 27, 2021

    @henryiii
    CollaboratorAuthor

    Maybe https://stackoverflow.com/questions/61590804/how-to-install-gcc-9-on-debianlatest-docker

    as long as it doesn’t update glibc by upgrading to a new os (which it might...), this should work like the dev toolset, I think?

  2. mayeut commented on Feb 27, 2021

    @mayeut
    Member

    Once you install gcc-9, it will update at least libstdc++ which is a no go. RHEL dev toolset took care of that and also other libraries like libgcc_s so that binaries produced with devtoolset would still be compatible with the base image with no action.
    If you have time to craft a PR installing gcc-9 and patching it so that it's still compatible with the base image, I'd be happy to take a look and merge this.

    manylinux_2_24 is mostly beneficial for alternate architectures where some functions were fixed in glibc (reported by numpy).

  3. mattip commented on Apr 8, 2021

    @mattip
    Contributor

    Unfortunately, the 6.3.10 version of gcc is too old to compile OpenBLAS on arm64, I get a error: unknown value 'cortex-a73' for -mtune. It seems support for cortex-a73 was added in gcc7. So now we are stuck: NumPy needs both manylinux_2_24 and a modern gcc.

  4. henryiii commented on Apr 8, 2021

    @henryiii
    CollaboratorAuthor

    Not at all surprised. You want the most recent GCC possible for alternative archs.

  5. h-vetinari commented on Apr 9, 2021

    @h-vetinari

    Or go for 2_28 right away (at least for arm64)? Debian 10 has GCC 8.3.

  6. mayeut commented on Apr 9, 2021

    @mayeut
    Member

    @h-vetinari
    Debian 10 can't be used as a base image for manylinux_2_28. There are symbol conflicts with other 2.28 distros (at least Photon 3.0)
    Furthermore, it would leave Ubuntu 18.04 users without wheels and there are still a great number of them.

    @mattip, there are still a number of possibilities for OpenBLAS/numpy:

    • Update the manylinux_2_24 toolchain in manylinux images.

    This would be the preferred option but, as mentioned in a previous comment, I simply can't take on this one but will be happy to review a PR doing that as it would benefit the whole manylinux ecosystem (checking it respects original symbol requirements for manylinux_2_24 and reliability/maintainability of proposed packages)

    Allows to remove CORTEXA73 tuning.
    This would allow to build with the stock manylinux_2_24 image but would probably introduce a speed penalty (I don't know how much & there is probably something better to be done for dispatching over at OpenBLAS - fallback to ARMV8 where it should maybe fallback to CORTEXA72 ?)
    Also, the dynamic list seems to be built depending on compiler support for some architectures. Should probably be reported over at OpenBLAS to get that feature for aarch64 also.

    • Use OpenBLAS built on manylinux2014 images

    It should produce a shared library that's perfectly valid on manylinux_2_24.
    In fact, now that I think about this, maybe building NumPy/OpenBLAS (others?) on manylinux2014 but tagging it manylinux_2_24 would work ? (depending how the accuracy issues mentioned in #494 were fixed in glibc i.e. symbol version bumped or not ?)

    • Update to ppa:ubuntu-toolchain-r/test gcc-9

    While I would not recommend it as a generic toolchain for the manylinux_2_24 image (libstdc++ issue), this one might be well suited for NumPy/OpenBLAS. OpenBLAS built successfully on the aarch64 manylinux_2_24 image using this toolchain with no extra non-compliant symbols:

    apt-get update
    apt-get install dirmngr
    apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 1E9377A2BA9EF27F
    echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial main" >> /etc/apt/sources.list
    apt-get update
    apt-get install -y --no-install-recommends gcc-9 gfortran-9
    export CC=gcc-9
    export FC=gfortran-9
    git clone https://github.com/xianyi/OpenBLAS.git
    cd OpenBLAS/
    git checkout af2b0d0
    make BUFFERSIZE=20 DYNAMIC_ARCH=1 USE_OPENMP=0 NUM_THREADS=64 BINARY=64 TARGET=ARMV8
    
    • Create & use manylinux_2_27 image based on Ubuntu 18.04

    Despite Ubuntu 18.04 not being listed as a potential manylinux candidate over at https://github.com/mayeut/pep600_compliance, it's only because the symbol analysis is done on libstdc++ (gcc 8 based) rather than on the default compiler used (gcc 7)
    Users of amazonlinux:2 won't be able to use wheels from this one.

  7. henryiii commented on Apr 12, 2021

    @henryiii
    CollaboratorAuthor

    Or go for 2_28 right away (at least for arm64)? Debian 10 has GCC 8.3.

    We need newer compilers with older images; requiring the wheel to be based on an OS newer than the compiler doesn't work well. If you want to use GCC 9 or 10, now even 2_28 would be to old. This was available with RHEL based images, so it should(*) be possible here, too. CentOS 6 has GCC 8, and that's 2_12.

    *: No statement about how hard it is, just that it should be possible.

  8. h-vetinari commented on Apr 12, 2021

    @h-vetinari

    If you want to use GCC 9 or 10

    That's quite a wide gap to bridge. Scipy is just recently thinking about bumping their gcc lower bound from 4.8 to 5.5 (not least due to being buildable on more exotic platforms) - the point being that I think very few projects can assume such modern compilers (with all that entails) for their entire user base.

    This was available with RHEL based images, so it should(*) be possible here, too.

    RH presumably spent a non-trivial amount of time on that... Don't get me wrong, that'd be awesome to have, but I'm thinking that even "cutting-edge" packages should normally be fine with the compilers of a ~2 year old mainstream distro.

    That's part of the appeal of the perennial manylinux, IMO, because maintainers can choose themselves where they want to set their cutoff (assuming pypa would provide all those images).

    @mayeut:

    There are symbol conflicts with other 2.28 distros (at least Photon 3.0)

    Photon already has a number of compatibility issues, apparently. From the POV of a maintainer, I want to cover the largest possible portion of my user base (vis-à-vis trade-offs in terms of maintainability) - and it's IMO a legitimate choice not to care about fringe distros (where users can still build from source, and are presumably used to that anyway).

  9. mayeut commented on Apr 24, 2021

    @mayeut
    Member

    Photon already has a number of compatibility issues, apparently.

    Yes, it's fully headless and does not provide any libraries like X11. It's not a major issue but it's mentioned anyway. If one is using Photon, I guess they know it only works this way and won't rely on packages requiring these dependencies (e.g. use opencv-python-headless instead of opencv-python). It's more of a limitation than a compatibility issue.

    From the POV of a maintainer, I want to cover the largest possible portion of my user base (vis-à-vis trade-offs in terms of maintainability) - and it's IMO a legitimate choice not to care about fringe distros (where users can still build from source, and are presumably used to that anyway).

    Agreed, it's obviously up to the package maintainers to decide what they want to support. However, there's no reason for manylinux to knowingly provide images that are not compliant with PEP600.

  10. mayeut commented on Apr 24, 2021

    @mayeut
    Member

    I did some experiments with PPA in the openblas PR. It seems to be the way to go to update the toolchain here.
    I was able to rebuild binutils-2.36.1 & gcc-10 from the upcoming Ubuntu 21.04 but targeting Ubuntu 16.04 and that works well when installed on Debian 9.
    I do not have the proper skills to do that in a clean way I think and any help on the matter would be greatly appreciated.
    I will try and up my skills w.r.t. providing a PPA with an up-to-date binutils however, for gcc-10, there's still the libstdc++ compatibility issue that's really out of my league here.

    1st experiments:

  11. h-vetinari commented on May 3, 2021

    @h-vetinari

    Regarding the toolchain issues, you might be interested how https://github.com/symengine/symengine-wheels handles this (because it builds based on conda-packages, which are compiled with much newer compilers).

  12. isuruf commented on May 8, 2021

    @isuruf

    Is there some way to replicate the “RHEL dev toolset” in the Debian based images?

    Yes, build gcc 10 with the patches from https://git.centos.org/rpms/devtoolset-10-gcc/blob/c7/f/SOURCES.
    In devtoolset, libstdc++.so is a linker script that points to a static libstdc++ for the new parts and shared libstdc++ for the older parts and the C++ ABI is set to GCC-4 ABI.

  13. mayeut commented on May 8, 2021

    @mayeut
    Member

    @isuruf,

    would you be willing to contribute a PPA which provide such installable artefacts and maintain those ?

    PS, they do not only patch libstdc++.so, other bunch of *.so use the same trick (like libgcc_s.so) also, they patch some sources (for these tricks, for these tricks to actually work as well as generic bug-fix/red-hat specific fixes) and the build process. As you said, all the material is in the patches from RedHat, we would need to find someone willing to do the work and some support.

  14. 10 remaining items

  15. h-vetinari commented on Nov 25, 2021

    @h-vetinari

    From that table, at each manylinux click we would lose support for :

    I think you're off-by-one here, as in: the right column needs to be shifted one row down. In particular manylinux2_28 still supports centos 8 / debian 10.

  16. mattip commented on Nov 25, 2021

    @mattip
    Contributor

    @h-vetinari thanks, I rephrased the top line of the comment. The table column heading was correct.

  17. h-vetinari commented on Jun 6, 2022

    @h-vetinari

    Based on current data from the pep-compliance repo, the EOL of manylinux_2_24 is imminent. I've opened a separate issue to discuss this: #1332

  18. mayeut commented on Aug 28, 2022

    @mayeut
    Member

    #1369 will drop support for manylinux_2_24 images on January 1st, 2023. This issue will never be fixed.

  19. snnn commented on Apr 10, 2024

    @snnn

    If we split the problem in two parts: the compiler and the runtime(gcc_s and libstdc++), we could find a different solution that just solves the first part. We may do cross-compiling with a very new C/C++ compiler(either GCC or LLVM/Clang) and an old rootfs that contains old version of Glibc/libstdc++/gcc_s/ ... . Then we do not have to depend on RHEL's devtoolset, and do not have to bind ourselves to a specific Linux distro. You may still use the latest C++ grammar, if the new grammar doesn't require a new runtime support. For example, you can use C++17 and target Ubuntu 16.04 but you cannot use C++17's std::filesystem library. I think this limitation is acceptable because anyway you cannot make it work on Apple's operating systems. If an open source project supports Linux, most likely it supports macOS too. The same concern applies. With this approach, we still can get many benefits from the new compiler and toolchain that:

    1. Better security. The new compilers are better patched, therefore they are more secure.
    2. Broader hardware support. For example, if you need to target a new ARM CPU that was just release recently.
    3. New C++ grammars, like ranged for.
    4. Better diagnostic warnings at build time
    5. New security flags. Like control flow guard.
  20. h-vetinari commented on Apr 10, 2024

    @h-vetinari

    You may still use the latest C++ grammar, if the new grammar doesn't require a new runtime support. For example, you can use C++17 and target Ubuntu 16.04 but you cannot use C++17's std::filesystem library.

    Yeah, except then every project would need to learn which C++ features require runtime support, which is really not realistic. People would get it wrong all the time, and stuff would definitely break.

  21. henryiii commented on Apr 10, 2024

    @henryiii
    CollaboratorAuthor

    Also, I know one of the issues is that the special arches generally require newer compilers. The reason NumPy couldn't use manylinux_2_24 wan't due to C++ features (they didn't use any), but because the old compiler was buggy on other architectures, they required 6.3+ or something like that. (Maybe it's just the compiler, and the libs are fine, but I'd be a bit worried)

  22. snnn commented on Apr 11, 2024

    @snnn

    Yeah, except then every project would need to learn which C++ features require runtime support, which is really not realistic. People would get it wrong all the time, and stuff would definitely break

    If it does not work, you will get a compile time error instead of a runtime error, so it is easy to handle. macOS already works in this way. The C++17 filesystem limitation is a well-known issue among iOS developers. So it is not unacceptable.

    Also, my proposal has another benefit: better 32-bit support. Nowadays it's often seen a single compiler process takes more than 1GB memory, but a 32-bit system only has 2GB usermode address space. When you see GCC/LD hit the limitation, nothing else you can do.

  23. mattip commented on Apr 11, 2024

    @mattip
    Contributor

    If it does not work, you will get a compile time error instead of a runtime error

    The beginning of the comments in this issue, closed for a year and a half, talked about how there is not a package with a split between the compiler and runtime. Is there now such a package or are you suggesting building a new compiler package? Who would maintain it? Where would it live?

  24. fweimer-rh commented on Apr 11, 2024

    @fweimer-rh

    Yeah, except then every project would need to learn which C++ features require runtime support, which is really not realistic. People would get it wrong all the time, and stuff would definitely break

    If it does not work, you will get a compile time error instead of a runtime error, so it is easy to handle.

    @snnn I'm not sure that's accurate. Why would you get a compiler error if you reference a symbol in the newer libstdc++? If you set things up so that the older libstdc++.so.6 is used for linking, in theory you could get a linker error, but I don't think building Python extensions is compatible with -Wl,-z,defs. So you wouldn't get an error, either.

  25. snnn commented on Apr 12, 2024

    @snnn

    Is there now such a package or are you suggesting building a new compiler package?

    Here are the details:
    First, get a manylinux docker image, run it in interactive mode with bash,

    docker run --rm -it quay.io/pypa/manylinux_2_28_x86_64 /bin/bash
    

    Then in the container do:

    dnf install gcc gcc-c++
    rm -rf /opt/rh
    

    Then, outside of the docker container, get a bash and run

    docker ps
    

    to get the docker instance id, then use docker export command to export the instance's rootfs as a tar file, then extract the tar file to somewhere.
    Then you can shutdown the docker container.
    Then install LLVM/Clang from somewhere. You can use the latest version.
    Then create a hello world C/C++ problem, and use clang to compile it.

    clang -o t test.c  --sysroot=/crossrootfs  -v -fuse-ld=lld

    "/crossrootfs" is where the tarball was extracted to.
    Then you can use objdump to check which glibc version the built binary requires, and transfer the binary to a target system to test.

    If you think this idea is acceptable, then you can rebuild the manylinux_2_28_x86_64 with a different base image that you want to target to, for example, Ubuntu 16.04 or UBI8. If you use a standard UBI8 image from RH and do not manually install additional RPM packages there, you are guaranteed that it will receive good vulnerability management service from RH for free for many many years. The warranty is for the image you downloaded from RH, not RPM packages.

    Is there now such a package or are you suggesting building a new compiler package?

    We do not need to build a custom compiler. We can get everything we need from the operating system's package manager, or from other vendors. For example, you may use Ubuntu 22.04 as the host OS, then get the latest LLVM from https://releases.llvm.org/. The compiler should have very decent support for every new hardware.

  26. mayeut commented on Jun 1, 2024

    @mayeut
    Member

    I think that there are 3/4 different problems that are currently addressed by devtoolset/gcc-toolset:

    1. cross-building is not not well supported in the python ecosystem (see the draft https://peps.python.org/pep-0720/). devtoolset/gcc-toolset are native toolchains that only require minimal setup to work.
    2. getting better support for new hardware: devtoolset is outdated for sure (gcc-10). The current gcc-toolset in AlmaLinux could be updated to gcc-13 (currently gcc-12) - update risk to be evaluated. Using the latest LLVM with cross-compilation definitely works in this case.
    3. newer C++ standard: there are 2 cases (3/4) here as already mentioned earlier in the thread: the part that does not need runtime support & the part that does. The gcc-toolset provides for both through a custom linker script for gcc_s /libstdc++/.... It's most likely possible to use LLVM with gcc-toolset so as not to restrict the runtime features available to the bare feature set of the base distro.

    @snnn, do you have a PoC somewhere (rather than the few commands posted here) ? I'd like to dig into this a bit more.

  27. mayeut commented on Nov 11, 2025

    @mayeut
    Member

    see #1871

    Use the latest clang (statically built as a mitigation to real cross-builds), on top of the existing gcc-toolset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions