Skip to content

Optionally support wheels #608

Open
Open

Description

@jakirkham

It might be interesting for some subset of packages to be available as wheels. conda-build has support (guessing 3.x) for outputting wheels as noted in this doc. Doing this could help consolidate the wheel and conda-forge build projects a bit more, which would lend itself to a larger interested group working in the same ecosystem.

Activity

  1. isuruf commented on Nov 11, 2017

    @isuruf
    Member

    For linux wheels, we'll have to use the manylinux1 docker container right?

  2. jakirkham commented on Nov 11, 2017

    @jakirkham
    MemberAuthor

    It's certainly something we could do. Though I'd like to avoid using two docker images if possible.

  3. isuruf commented on Nov 11, 2017

    @isuruf
    Member

    It's centos5 based while the conda-forge image is centos6 based.

  4. jakirkham commented on Nov 11, 2017

    @jakirkham
    MemberAuthor

    Yep, I’m aware. There has been discussion about a manylinux2. Presumably it would be CentOS 6 based. We could start talking with people involved with manylinux1’s maintenance to see if we could agree on an image.

  5. jakirkham commented on Nov 11, 2017

    @jakirkham
    MemberAuthor

    Should add that I'm mainly thinking these get published to anaconda.org for now, but they could certainly be reused elsewhere. Possibly could publish them to other places if that is of interest too.

  6. jakirkham commented on Nov 11, 2017

    @jakirkham
    MemberAuthor

    Would be good to hear some feedback from those close to the manylinux effort.

    cc @ncoghlan @njsmith (others?)

  7. ncoghlan commented on Nov 12, 2017

    @ncoghlan

    I'm a fan of evolving the manylinux baseline, but I'd like to switch it to a "target compatibility year" model on the next update.

    That is, instead of it being manylinux2, a RHEL/CentOS 6 based version would instead be manylinux2010, then manylinux2014 would be a RHEL/CentOS 7 base, etc.

    I also think the compatibility detection section in https://www.python.org/dev/peps/pep-0513/#platform-detection-for-installers will require an update, such that we give installers a config file with the compatibility profiles in it. That is, the config file wouldn't say "This platform is manylinuxX compliant", but would instead say "For manylinuxX compliance, check for glibc Y.Z".

    (Part of my inspiration for the latter idea is http://www.vfxplatform.com/ - once a means of checking a compatibility profile has been defined, the main thing new editions need to do is update the exact versions to be checked for)

  8. njsmith commented on Nov 12, 2017

    @njsmith

    I think closer interactions between conda folks and wheel folks can only be good for both sides...

    Regarding CentOS 6 and manylinux, everyone agrees that we need to update the setup and it's pretty straightforward to do, but someone needs to do the work and no-one's volunteered. There's a list of the required tasks here: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html

    @ncoghlan Using calver here seems reasonable to me. I'm not sure I follow the part about the config file -- our current config file (the import _manylinux thing) is an escape hatch, mainly useful for distributors who want to be all grar go away I hate wheels. Otherwise, we currently define the glibc version → manylinux version table in the PEP text, which seems as useful as anything else, since every installer has to be updated when a new manylinux spec gets released anyway (we don't have any way to automagically distribute a config file to them).

  9. ncoghlan commented on Nov 12, 2017

    @ncoghlan

    @njsmith I'm thinking something like certifi or pytz - a manylinux package on PyPI that gives you easy access to the kind of information the PEP describes. Even if installer devs write their own variant, they could use a common implementation in their test suite as a cross-check.

  10. jakirkham commented on Mar 5, 2018

    @jakirkham
    MemberAuthor

    For some use cases, was recently asked about pip installing a package that I help maintain. The package was available as an sdist from PyPI. However the user ran into some glibc issues caused by build location vs. installing location discrepancy. In another use case, a user requested wheels for another project that I help maintain.

    In pursuit of these use cases, generated PR ( conda-forge/numcodecs-feedstock#17 ) and PR ( conda-forge/pyfftw-feedstock#16 ). Both of these build the packages in question on CI and package them in both conda and wheel form thanks to some new features in conda-build. Tested building both of them locally and was able to get them to pip install into a minimal Python 3.6 environment and get them to work.

    These particular cases work well IMHO as they don't really have any runtime dependencies that are other libraries. In the first case, everything is linked together into libraries that would be installed in site-packages regardless. In the second case, we already statically link a library to avoid some known issues encountered in the wild.

    There's probably still some things to iron out like whether this is something others are comfortable pursuing in conda-forge, what other tooling changes need to be made (e.g. the upload script), how do we decide what packages should also build wheels, how do we handle more complex wheels (i.e. required shared libraries at runtime), how do we handle discrepancies in compatibility (e.g. CentOS 5 vs 6, macOS 10.6 vs 10.9), and probably others that I have thought of yet.

    Anyways seems like a good time to get some other thoughts and feedback.

  11. jakirkham commented on Mar 5, 2018

    @jakirkham
    MemberAuthor

    cc @conda-forge/core

  12. jakirkham commented on Mar 6, 2018

    @jakirkham
    MemberAuthor

    Put together another PR ( conda-forge/python-snappy-feedstock#12 ) to push on the dynamic library problem. However not totally sure how to proceed. Right now it assumes users have the dependencies on their system, which may not be reasonable. So would welcome feedback from wheel experts.

    Guessing we need to use auditwheel (Linux), delocate (macOS), and ??? (Windows). Betting auditwheel runs into trouble as we are not using CentOS 5. Guessing delocate works, but we need to package it first. ( conda-forge/staged-recipes#5316 ) Hoping that's not too difficult. Not sure what to do about Windows.

    Also read the pynativelib proposal ( pypa/wheel-builders#2 ), which is an interesting alternative. Though would need pointers on how that would work on different platforms and what sort of tooling needs to be employed.

    cc @msarahan (as we're having a related conversation at conda-build)

  13. msarahan commented on Mar 6, 2018

    @msarahan
    Member

    One thing that seems like an inevitable headache is that Anaconda has chosen to allow the newest language features (C++17 and such), and thus we need to ship the c++ runtime. I think that's not something wheel builders want to do, but I might be wrong. I think they'd rather use things like devtoolset so that they can keep older system libstdc++ and not ship it. I don't know what impact that has on being able to use C++ 14/17/whatever - I know 11 works.

    I need to spend some time with PEP517/518 to understand how conda-build fits in. That seems key here, as it nicely defines the abstraction of a build system. @njsmith what is the status of pynativelib? That's also going to be very important to all of this, since it determines how much hacking/repacking of wheels goes on to fulfill native dependencies.

  14. jakirkham commented on Mar 14, 2018

    @jakirkham
    MemberAuthor

    Guessing most people have seen this, but dropping it here for general reference. PEP 571 is an in progress proposal for manylinux2, which would be a CentOS 6-based build image.

  15. njsmith commented on Apr 8, 2018

    @njsmith

    Status of pynativelib: IIRC the proposal linked above was over-optimistic about how easy this would be (in particular MacOS linkers are far more annoying than I realized) but the blocking technical issues are solved now, so "all" that's needed is someone to pick it up and push things forward.

  16. jakirkham commented on Apr 12, 2018

    @jakirkham
    MemberAuthor

    cc-ing @di, as we had talked about this briefly tonight. Would be interested in hearing your thoughts on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions