You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
SOABI on Linux does not distinguish between GNU libc and musl libc #87278
bpo-46390: Multiple test failures on Alpine 3.15 / musl-1.2.2-r7
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=Nonecreated_at=<Date2021-02-03.09:26:13.443>labels= ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11']
title='SOABI on Linux does not distinguish between GNU libc and musl libc'updated_at=<Date2022-01-29.00:29:09.330>user='https://github.com/ncopa'
The SOABI does not make any difference between GNU libc and musl libc.
Using official docker images:
# debian build with GNU libc
$ docker run --rm python:slim python -c 'import sysconfig;print(sysconfig.get_config_var("SOABI"))'
cpython-39-x86_64-linux-gnu
# alpine build with musl libc
$ docker run --rm python:alpine python -c 'import sysconfig;print(sysconfig.get_config_var("SOABI"))'
cpython-39-x86_64-linux-gnu
Both ends with -gnu, while it would be expected that with musl it would end with -musl
This affects the extension suffix:
$ docker run --rm python:slim python-config --extension-suffix
.cpython-39-x86_64-linux-gnu.so
$ docker run --rm python:alpine python-config --extension-suffix
.cpython-39-x86_64-linux-gnu.so
Which again affects the pre-compiled binary wheels, and binary modules built with musl libc gets mixed up with the GNU libc modules due to the -gnu.so suffix.
The source of the problem is that the configure.ac file assumes that all defined(linux) is -gnu when detecting the PLATFORM_TRIPLET.
...
#if defined(__ANDROID__)
# Android is not a multiarch system.
#elif defined(__linux__)
# if defined(__x86_64__) && defined(__LP64__)
x86_64-linux-gnu
# elif defined(__x86_64__) && defined(__ILP32__)
x86_64-linux-gnux32
# elif defined(__i386__)
...
So when building python with musl libc the PLATFORM_TRIPLET always sets *-linux-gnu.
output from configure run on musl system:
...
checking for a sed that does not truncate output... /bin/sed
checking for --with-cxx-main=<compiler>... no
checking for the platform triplet based on compiler characteristics... x86_64-linux-gnu
...
A first step in fixing this would be to make sure that we only set -gnu when __GLIBC__ is defined:
diff --git a/configure.ac b/configure.ac
index 1f5a008388..1b4690c90f 100644
--- a/configure.ac+++ b/configure.ac@@ -726,7 +726,7 @@ cat >> conftest.c <<EOF
#undef unix
#if defined(__ANDROID__)
# Android is not a multiarch system.
-#elif defined(__linux__)+#elif defined(__linux__) && defined (__GLIBC__)
# if defined(__x86_64__) && defined(__LP64__)
x86_64-linux-gnu
But that would make build with musl fail with "unknown platform triplet".
Not sure what the proper fix would be, but one way to extract the suffix from $CPP -dumpmachine
The triplet defines the calling convention. For example x86_64-linux-gnu means x86_64 / AMD64 CPU architecture, Linux, with standard GNU / GCC calling convention. Other calling conventions are "x86_64-linux-gnux32" for X32 on AMD64 and "arm-linux-gnueabihf" for 32bit ARM with extended ABI and hardware float support.
SOABI basically contains the CPU architecture and Kernel ABI. The libc ABI is yet another dimension that is not encoded in the shared library ABI.
The libc ABI is more complex than just glibc or musl. You need to include the ABI version of all core components. For example manylinux2014 defines the ABI for glibc as GLIBC_2.17, CXXABI_1.3.7, CXXABI_TM_1, GLIBCXX_3.4.19, GCC_4.8.0.
As a rule of thumb, a SOABI like ".cpython-39-x86_64-linux-gnu.so" only works the current host. You cannot safely move the file to another host or bump the SO version of any library, unless you ensure that the ABIs of all libraries are compatible.
Do you have glibc and musl installed side by side?
ncopa commented on Feb 3, 2021
ncopamannequin
MannequinAuthor
More actions
Do you have glibc and musl installed side by side?
No. But there is nothing preventing me to have the libc runtimes installed in parallel with glibc.
/lib/libc.so.6
/lib/libc.musl-x86_64.so.1
And it is not common that people copy libc.so.6 (with friends) to their alpine docker images to run both in same container. If that is a good idea is other discussion.
I do understand that full ABI compatibility also may involve libc ABI version, but I think that is a slightly different problem. Newer versions of glibc and musl libc are backwards compatible. You can expect a binary built with old libc version to run with new libc. But you cannot expect a binary built with musl libc to run with gnu libc.
The standard autotools' config.guess1 also recognizes -musl as different platform.
$ ./config.guess
x86_64-pc-linux-musl
So I think it makes sense to treat *-linux-musl as a different platform than *-linux-gnu.
If you still insist that this is only about calling convention and not platform, then I think you should at least clarify that in the configure.ac script to avoid confusion:
sed -i -e 's/PLATFORM_TRIPLET/CALLING_CONVENTION_TRIPLET/g' -e 's/platform triplet/calling convention triplet/g' configure.ac
I stand corrected. The last element in the platform triplet does seem to indicate libc.
Is there any formal definition of the platform triplet or is it defined by GCC's reference implementation? A quick search didn't reveal any decisive results.
The next steps here would be:
document the platform triplet in regards of musl libc (and potentially other libcs like uclibc and embedded newlibc)
In order to work around this limitation, packaging is forced to call ld to work out which libc the current python is linked to. If changing the SOABI is too much, maybe it would be possible to change either sys.implementation._multiarch or sysconfig.get_config_var('MULTIARCH') to reflect the libc?
I don’t think that multiarch is the right variable to indicate libc, but another field could be added to sys.implementation, or a variable to sysconfig, to support this. Maybe the request needs a broadest audience using discuss.python.org
Activity
ncopa commentedon Feb 3, 2021
The SOABI does not make any difference between GNU libc and musl libc.
Using official docker images:
# debian build with GNU libc
$ docker run --rm python:slim python -c 'import sysconfig;print(sysconfig.get_config_var("SOABI"))'
cpython-39-x86_64-linux-gnu
# alpine build with musl libc
$ docker run --rm python:alpine python -c 'import sysconfig;print(sysconfig.get_config_var("SOABI"))'
cpython-39-x86_64-linux-gnu
Both ends with
-gnu, while it would be expected that with musl it would end with-muslThis affects the extension suffix:
Which again affects the pre-compiled binary wheels, and binary modules built with musl libc gets mixed up with the GNU libc modules due to the -gnu.so suffix.
The source of the problem is that the
configure.acfile assumes that all defined(linux) is -gnu when detecting the PLATFORM_TRIPLET.So when building python with musl libc the PLATFORM_TRIPLET always sets
*-linux-gnu.output from configure run on musl system:
A first step in fixing this would be to make sure that we only set -gnu when __GLIBC__ is defined:
But that would make build with musl fail with "unknown platform triplet".
Not sure what the proper fix would be, but one way to extract the suffix from
$CPP -dumpmachinetiran commentedon Feb 3, 2021
The suffix "-gnu" does not stand for "glibc".
The triplet defines the calling convention. For example x86_64-linux-gnu means x86_64 / AMD64 CPU architecture, Linux, with standard GNU / GCC calling convention. Other calling conventions are "x86_64-linux-gnux32" for X32 on AMD64 and "arm-linux-gnueabihf" for 32bit ARM with extended ABI and hardware float support.
The triplets are standardized beyond Python. Debian's multiarch page lists and explains a large amount of triplets, https://wiki.debian.org/Multiarch/Tuples
ncopa commentedon Feb 3, 2021
Does this mean that the SOABI should be the same for python built with musl libc and GNU libc?
They are not really ABI compatible.
tiran commentedon Feb 3, 2021
SOABI basically contains the CPU architecture and Kernel ABI. The libc ABI is yet another dimension that is not encoded in the shared library ABI.
The libc ABI is more complex than just glibc or musl. You need to include the ABI version of all core components. For example manylinux2014 defines the ABI for glibc as GLIBC_2.17, CXXABI_1.3.7, CXXABI_TM_1, GLIBCXX_3.4.19, GCC_4.8.0.
As a rule of thumb, a SOABI like ".cpython-39-x86_64-linux-gnu.so" only works the current host. You cannot safely move the file to another host or bump the SO version of any library, unless you ensure that the ABIs of all libraries are compatible.
ncopa commentedon Feb 3, 2021
The referenced https://wiki.debian.org/Multiarch/Tuples doc says:
Since GNU libc and musl libc are not ABI compatible they can not share same unique identifier. I think replacing -gnu with -musl makes sense.
tiran commentedon Feb 3, 2021
Do you have glibc and musl installed side by side?
ncopa commentedon Feb 3, 2021
No. But there is nothing preventing me to have the libc runtimes installed in parallel with glibc.
/lib/libc.so.6
/lib/libc.musl-x86_64.so.1
And it is not common that people copy libc.so.6 (with friends) to their alpine docker images to run both in same container. If that is a good idea is other discussion.
I do understand that full ABI compatibility also may involve libc ABI version, but I think that is a slightly different problem. Newer versions of glibc and musl libc are backwards compatible. You can expect a binary built with old libc version to run with new libc. But you cannot expect a binary built with musl libc to run with gnu libc.
gcc recognizes -linux-musl as a valid platform tuple different that differs from -linux-gnu:
https://github.com/gcc-mirror/gcc/blob/master/gcc/config/t-musl
The standard autotools' config.guess1 also recognizes -musl as different platform.
So I think it makes sense to treat *-linux-musl as a different platform than *-linux-gnu.
If you still insist that this is only about calling convention and not platform, then I think you should at least clarify that in the configure.ac script to avoid confusion:
sed -i -e 's/PLATFORM_TRIPLET/CALLING_CONVENTION_TRIPLET/g' -e 's/platform triplet/calling convention triplet/g' configure.ac
tiran commentedon Feb 9, 2021
I stand corrected. The last element in the platform triplet does seem to indicate libc.
Is there any formal definition of the platform triplet or is it defined by GCC's reference implementation? A quick search didn't reveal any decisive results.
The next steps here would be:
Let's continue this on https://discuss.python.org/t/wheels-for-musl-alpine/7084
ncopa commentedon Feb 9, 2021
This mentions some examples for musl triplets/tuples:
https://wiki.musl-libc.org/getting-started.html
It points to https://github.com/richfelker/musl-cross-make/blob/master/README.md#supported-targets which I think is the best documentation. (Rich Felker is the author and lead developer or musl libc).
I'm not sure its worth spend time on uClibc which does not seem to have any commits at all since 2015. https://git.uclibc.org/uClibc/log/
We will work on our side on buildbot with Alpine linux.
Thank you for being understanding.
Arfrever commentedon Feb 9, 2021
While original uClibc is not maintained, its fork called uClibc-ng is supposedly binary-compatible with uClibc and is still somehow maintained:
https://uclibc-ng.org/
https://repo.or.cz/w/uclibc-ng.git
https://www.uclibc.org/news.html also says:
"While uClibc releases are on hold, you may use uClibc-ng"
libc part of platform tuple on uClibc-ng systems is expected to be "uclibc".
ncopa commentedon Feb 10, 2021
I created a PR for this #24502
merwok commentedon Nov 1, 2021
If this is deemed a bugfix, the PR should be backported.
6 remaining items
mattip commentedon Jan 19, 2023
In order to work around this limitation, packaging is forced to call
ldto work out which libc the current python is linked to. If changing the SOABI is too much, maybe it would be possible to change eithersys.implementation._multiarchorsysconfig.get_config_var('MULTIARCH')to reflect the libc?merwok commentedon Jan 19, 2023
I don’t think that multiarch is the right variable to indicate libc, but another field could be added to
sys.implementation, or a variable to sysconfig, to support this. Maybe the request needs a broadest audience using discuss.python.orgmattip commentedon Jan 19, 2023
Maybe the request needs a broadest audience using discuss.python.org
Done
vstinner commentedon Mar 19, 2026
The initial issue has been fixed in Python 3.11 by commit 1f036ed:
SOABInow ends with-muslon Alpine Linux which uses the musl C library.I close the issue.