Provide manylinux1 wheels on PyPI

Context: what is manylinux1 tag.

Currently, Tensorflow wheels on PyPI suck. As many know, we cannot upload specific Linux wheels on PyPI because the cheese shop prohibits different flavors (though Windows and macOS are OK). However, one can upload universal Linux wheels aka manylinux1 (see the link from above). There is a tool which helps making manylinux1 wheels called auditwheel.

auditwheel show tensorflow-0.11.0rc0-cp35-cp35m-linux_x86_64.whl 

tensorflow-0.11.0rc0-cp35-cp35m-linux_x86_64.whl is consistent with
the following platform tag: "manylinux1_x86_64".

The wheel references no external versioned symbols from system-
provided shared libraries.

The wheel requires no external shared libraries! :)

This basically means that we can execute auditwheel repair and obtain the manylinux1 wheel.

Thus I propose to upload CPU-only wheels for Linux for Python 2/3 to PyPI with manylinux1 tag.

@martinwicke

@martinwicke has a better understanding of PyPi and wheels than I do, perhaps he can comment or redirect to a more appropriate person.

@yifeif

I did not know about this tool. @yifeif can you look into this one and see if this is something that would work for us? It may actually make the pypi story simpler.

Will look into this.

So?..

This is now available with 0.12rc0 release!

The Linux 0.12rc0 whl files available on PyPI do not conform to the manylinux1 specification provided in PEP531. Specifically, they contain shared libraries which reference versioned symbols which do not meet the GLIBC <= 2.5 and GLIBCXX <= 3.4.9 requirements specified in the PEP.

$ auditwheel show tensorflow-0.12.0rc0-cp35-cp35m-manylinux1_x86_64.whl 

tensorflow-0.12.0rc0-cp35-cp35m-manylinux1_x86_64.whl is consistent
with the following platform tag: "manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.16',
'GLIBC_2.9', 'GLIBC_2.17', 'GLIBC_2.3.4', 'GLIBC_2.10', 'GLIBC_2.3',
'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.14', 'GLIBC_2.11', 'GLIBC_2.6',
'GLIBC_2.7', 'GLIBC_2.3.2'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.9', 'GLIBCXX_3.4.11', 'CXXABI_1.3', 'GLIBCXX_3.4',
'CXXABI_1.3.5', 'GLIBCXX_3.4.14', 'GLIBCXX_3.4.19', 'GLIBCXX_3.4.18',
'GLIBCXX_3.4.15'}, libdl.so.2 with versions {'GLIBC_2.2.5'},
libpthread.so.0 with versions {'GLIBC_2.3.2', 'GLIBC_2.2.5'},
libm.so.6 with versions {'GLIBC_2.2.5'}, libgcc_s.so.1 with versions
{'GCC_3.0'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.23.so",
    "libdl.so.2": "/lib/x86_64-linux-gnu/libdl-2.23.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.23.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.23.so",
    "libstdc++.so.6": "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21"
}

The pypa/manylinux repository includes Docker images which can be used to build manylinux1 compatible wheels.

@yifeif

@yifeif I guess auditwheel repair should be enough to fix this. Just curious, how do you make wheels currently?

auditwheel repair cannot correct versioned symbol outside of the spec. The command only fixes wheel files which are missing external shared libraries.

Mmm, building TF on CentOS 5 looks like a joke. Besides, nvidia/cuda:8.0 is already used, it is still Ubuntu 14.04 - the perfect ABI balance IMHO. I would explicitly write in the docs that TF's "manylinux1" is actually not manylinux1 in the strict definition and it should be enough.

@jjhelmus

Thanks for bring this up guys. We do run our whl files through auditwheel repair, but as @jjhelmus mentioned, auditwheel does not fix symbols when they are from a too-recent version. @vmarkovtsev I will add a note on pypi for next release.

@yifeif

@yifeif Thanks for update and I think the note sounds like the best solution.

Any chance you could share a bit about how the released whl files are built? I help maintain the conda-forge tensorflow package and we have been running into some issues on Linux. We uses CentOS 6 as our Linux target which uses an older GLIBC than the whl files support. I've tried to build tensorflow from source using our build setup but this has proved challenging. Any hints you could provide would be most appreciated.

@jjhelmus

@jjhelmus you can refer to https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/builds/pip.sh and https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/pip_package/build_pip_package.sh for how we build our whls.

Navigation Menu

Repository navigation

Provide manylinux1 wheels on PyPI #5033

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Provide manylinux1 wheels on PyPI #5033

Description

Activity

asimshankar commented on Oct 18, 2016

martinwicke commented on Oct 29, 2016

yifeif commented on Oct 29, 2016

vmarkovtsev commented on Nov 22, 2016

yifeif commented on Nov 29, 2016

jjhelmus commented on Nov 29, 2016

vmarkovtsev commented on Nov 29, 2016

jjhelmus commented on Nov 29, 2016

vmarkovtsev commented on Nov 29, 2016

yifeif commented on Nov 29, 2016

jjhelmus commented on Nov 29, 2016

yifeif commented on Nov 29, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions