You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is there a plan to migrate PEP-513 to a new minimum subset of libraries+versions eventually?
Please direct me to an existing discussion or a more appropriate forum if one exists.
Reacted by Olivier Grisel, Gregory P. Smith, Cody Scott, René Leonhardt, Wu Haotian, Daniel Hershcovich, jbarlow, pjxiao, Alex Rothberg, jakirkham and 6 more
There is a plan in the sense that everyone agrees that this is a thing that should happen, but not in the sense of anyone having actually volunteered to do the work :-).
@lukeyeager would you like to volunteer to work on a draft of a new PEP for manylinux2 + CI infrastructure to actually generate manylinux2 compatible binaries (reusing but upgrading the existing scripts that we use to maintain the manylinux1 docker images)?
Well that's a delightfully positive response! Glad to hear there is interest in pushing out an update. Unfortunately I don't think this falls under my purview at work, nor do I currently have the time to do it after hours.
A psycopg user reported that the manylinux psycopg2 2.7 package is failing connection because Centos 5.11 ships with libssl 0.9.8e (psycopg/psycopg2#518), while the missing features are available e.g. in 1.0.2k
It's up to you really, but the"official" manylinux1 solution would be to
build your own openssl on centos5 and then ship that, yeah.
Technically this is pretty easy to do -- the manylinux build scripts
actually do it because the centos 5 openssl is so useless that pip wouldn't
work otherwise. It's just a curl / tar / configure / make. The hard part is
that then you're responsible for releasing a new wheel whenever there's a
new openssl security bug. (Which is why we don't make our openssl build
available -- we want there to be a speedbump so people stop and think about
this.) But IIRC you're already shipping openssl in all your wheels, so for
you this responsibility isn't new...
Yes, needing to build a new package if a new bug in openssl is found is something I had already figured out, no problem with that.
The problem with psycopg is that both libssl and libpq - the postgres client library - must be built with all the dependencies needed to allow different form of authentications (pam, kerberos, etc.). But yesterday I already whipped together a package with several features in this script. I'll try building wheels with that, check if all the linked libraries are present (using system libpq vs. the one I build myself), rinse, repeat, ship.
We were having issues getting manylinux to build on centos 5 because for kivy we need gl es2 and centos 5 doesn't seem to have it available. I started going down the rabit hole to compile mesa on centos 5, but there are just too many dependencies missing.
So, we were wondering, would we be terrible people if we build on centos 7 (even centos 6 is now in maintenance updates only and reaches eol at end of 2020) and distribute them as manylinux? We are aware it may only run on newer system, but it is probably ok for us.
So, we were wondering, would we be terrible people if we build on centos 7 (even centos 6 is now in maintenance updates only and reaches eol at end of 2020) and distribute them as manylinux? We are aware it may only run on newer system, but it is probably ok for us.
I would encourage you not to do this. It would be unfortunate, in my opinion, to distribute wheels on PyPI using the "manylinux1" tag which do not conform to PEP 513.
I mention this issue here because I think it is relevant to the manylinux2 roadmap; I tried to build wheels and run auditwheel repair on centos6, and got bitten by this bug of patchelf: NixOS/patchelf#128
@antocuni: not quite sure what you were hoping to accomplish there, but I can't think of any wheel-related situation where running patchelf on libm is a good idea :-).
If the problem also occurs for other centos6 binaries then that is more concerning...
However, it seems not to be related to pypy at all, as I can reproduce the problem even with cpython. I don't know why auditwheel repair chooses to patch libm: I assumed it was a normal behavior, but from what you write it seems it's not :):
$ docker run -it antocuni/pypy-wheels:3b01df8e /bin/bash
[root@e5eae337a5da /]# /opt/python/cp36-cp36m/bin/pip wheel numpy --no-binary :all:
Collecting numpy
Downloading numpy-1.13.1.zip (5.0MB)
100% |████████████████████████████████| 5.0MB 222kB/s
Building wheels for collected packages: numpy
Running setup.py bdist_wheel for numpy ... done
Stored in directory: /
Successfully built numpy
[root@e5eae337a5da /]# auditwheel repair --plat linux_x86_64 numpy-1.13.1-cp36-cp36m-linux_x86_64.whl
Repairing numpy-1.13.1-cp36-cp36m-linux_x86_64.whl
Grafting: /lib64/libm-2.12.so -> numpy/.libs/libm-2-e51d9a45.12.so
Grafting: /lib64/libpthread-2.12.so -> numpy/.libs/libpthread-2-fdb92d01.12.so
Grafting: /lib64/libc-2.12.so -> numpy/.libs/libc-2-8b5594ec.12.so
Setting RPATH: numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Grafting: /usr/lib64/atlas/libptf77blas.so.3.0 -> numpy/.libs/libptf77blas-842266ad.so.3.0
Grafting: /usr/lib64/libgfortran.so.3.0.0 -> numpy/.libs/libgfortran-6ca0b838.so.3.0.0
Grafting: /usr/lib64/atlas/libptcblas.so.3.0 -> numpy/.libs/libptcblas-f3eb6402.so.3.0
Grafting: /usr/lib64/atlas/libatlas.so.3.0 -> numpy/.libs/libatlas-b6fb69b0.so.3.0
Setting RPATH: numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/umath.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Grafting: /usr/lib64/atlas/libcblas.so.3.0 -> numpy/.libs/libcblas-70dd3697.so.3.0
Grafting: /usr/lib64/atlas/liblapack.so.3.0 -> numpy/.libs/liblapack-7aef6c42.so.3.0
Grafting: /usr/lib64/atlas/libf77blas.so.3.0 -> numpy/.libs/libf77blas-75a5c7e0.so.3.0
Grafting: /lib64/libgcc_s-4.4.7-20120601.so.1 -> numpy/.libs/libgcc_s-4-09117363.4.7-20120601.so.1
Setting RPATH: numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so to "$ORIGIN/../.libs"
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main_repair.py", line 81, in execute
update_tags=args.UPDATE_TAGS)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/repair.py", line 91, in repair_wheel
needed = elf_read_dt_needed(path)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/elfutils.py", line 16, in elf_read_dt_needed
raise ValueError('Could not find soname in %s' % fn)
ValueError: Could not find soname in numpy/.libs/libm-2-e51d9a45.12.so
Activity
njsmith commentedon Feb 14, 2017
There is a plan in the sense that everyone agrees that this is a thing that should happen, but not in the sense of anyone having actually volunteered to do the work :-).
ogrisel commentedon Feb 14, 2017
@lukeyeager would you like to volunteer to work on a draft of a new PEP for manylinux2 + CI infrastructure to actually generate manylinux2 compatible binaries (reusing but upgrading the existing scripts that we use to maintain the manylinux1 docker images)?
lukeyeager commentedon Feb 14, 2017
Well that's a delightfully positive response! Glad to hear there is interest in pushing out an update. Unfortunately I don't think this falls under my purview at work, nor do I currently have the time to do it after hours.
ogrisel commentedon Feb 15, 2017
Defining an updated whitelist for manylinux2 would also be a simple way to tackle the curses 5 -> 6 issue: #94.
dvarrazzo commentedon Mar 2, 2017
A psycopg user reported that the manylinux psycopg2 2.7 package is failing connection because Centos 5.11 ships with libssl 0.9.8e (psycopg/psycopg2#518), while the missing features are available e.g. in 1.0.2k
How would you suggest to proceed?
njsmith commentedon Mar 2, 2017
dvarrazzo commentedon Mar 2, 2017
Yes, needing to build a new package if a new bug in openssl is found is something I had already figured out, no problem with that.
The problem with psycopg is that both libssl and libpq - the postgres client library - must be built with all the dependencies needed to allow different form of authentications (pam, kerberos, etc.). But yesterday I already whipped together a package with several features in this script. I'll try building wheels with that, check if all the linked libraries are present (using system libpq vs. the one I build myself), rinse, repeat, ship.
lukeyeager commentedon Apr 3, 2017
Update: EOL happened
matham commentedon Jul 7, 2017
We were having issues getting manylinux to build on centos 5 because for kivy we need gl es2 and centos 5 doesn't seem to have it available. I started going down the rabit hole to compile mesa on centos 5, but there are just too many dependencies missing.
So, we were wondering, would we be terrible people if we build on centos 7 (even centos 6 is now in maintenance updates only and reaches eol at end of 2020) and distribute them as manylinux? We are aware it may only run on newer system, but it is probably ok for us.
rmcgibbo commentedon Jul 7, 2017
I would encourage you not to do this. It would be unfortunate, in my opinion, to distribute wheels on PyPI using the "manylinux1" tag which do not conform to PEP 513.
njsmith commentedon Jul 7, 2017
Note though that we're actively seeking volunteers to help make manylinux2 happen officially.
rmcgibbo commentedon Jul 8, 2017
Would this help? https://mail.python.org/pipermail/wheel-builders/2017-July/000282.html
antocuni commentedon Jul 25, 2017
I mention this issue here because I think it is relevant to the manylinux2 roadmap; I tried to build wheels and run
auditwheel repairon centos6, and got bitten by this bug of patchelf:NixOS/patchelf#128
njsmith commentedon Jul 25, 2017
@antocuni: not quite sure what you were hoping to accomplish there, but I can't think of any wheel-related situation where running
patchelfonlibmis a good idea :-).If the problem also occurs for other centos6 binaries then that is more concerning...
antocuni commentedon Jul 25, 2017
@njsmith I originally encountered the problem by developing this: https://github.com/antocuni/pypy-wheels
I have to use centos6 because pypy3 apparently does not build on centos5, and because the portable pypy binaries are built on centos6 as well: note that the
antocuni/pypy-wheelsimage is basically the same as manylinux1, only based on centos6 instead of centos5..However, it seems not to be related to pypy at all, as I can reproduce the problem even with cpython. I don't know why
auditwheel repairchooses to patchlibm: I assumed it was a normal behavior, but from what you write it seems it's not :):adrientetar commentedon Oct 30, 2017
wxPython had similar problems with centos 5 having only old gtk2 libs and no gtk3, see: https://github.com/RobinD42/Phoenix/tree/a8d9b0f144e8c12dcd0159e6999804bbff9b726d/vagrant/manylinux
Mizux commentedon Sep 19, 2018
What's about PEP571 aka manylinux2010 ?
Bonus: It solve the issue with kernel
vsyscall=emulatecontrary to aCentos:6based Dockerfile.cf #141
armudgal commentedon Apr 16, 2019
Hi,
should we close this issue since the discussion has entirely moved to #179
trishankatdatadog commentedon Apr 16, 2019
We should, I'll let @mayeut make the call
mayeut commentedon Apr 16, 2019
closing per previous comments. See manylinux2010 in #179