Skip to content

Building wheels #70

New issue
New issue
Closed
#224
Closed
Building wheels#70
#224

Description

@jakirkham
jakirkham
opened on Mar 3, 2018

As numcodecs includes source that needs to be compiled, there can be some challenges or technical hurtles that can be encountered by a user that they may not be aware of. While we do solve this in some sense by supplying conda-forge packages with prebuilt binaries, pip remains the defacto way for Python users to get packages. However in the case of pip the user will get the sdist, which needs compilation. While this does work, there is definitely some appeal to providing a pip solution that does not require compilation (i.e. prebuilt wheels). This would help users avoid compatibility problems like this one ( #69 ).

One solution would be to piggy back off of whatever conda-forge ends up doing to also supply wheels ( conda-forge/conda-smithy#608 ). This would work well for Windows. Though on macOS conda-forge uses the 10.9 SDK (instead of the 10.6 SDK that Python tries to support). Also on Linux conda-forge uses CentOS 6 with glibc 2.12 (instead of CentOS 5 with glibc 2.5 used by manylinux1). In practice these two cases that don't sync up will be hard for Python to require much longer and some packages already don't comply (happy to go into details if this is of interest). This certainly would enjoy the nice benefit of the architecture and community conda-forge has in place to solve these issues. It also seems that proponents of the wheel format are interested in collaborating, which definitely should help.

If that doesn't work, the alternative would be to build wheels here. For Linux, there is manylinux1 Docker image, which could be used fairly easily to build these. For macOS, we could try to build them here or reach out to MacPython for help. For Windows, it would probably be best to reuse the conda-forge solution as much as possible as that already fits the requirements well. Probably would require some like issue ( conda/conda-build#2490 ) to solved. Though I don't think anyone that has had time to do that. Alternatively one can build a wheel with conda-build.

Activity

  1. jakirkham commented on Mar 5, 2018

    @jakirkham
    jakirkham
    on Mar 5, 2018
    MemberAuthor

    Have a couple of examples of wheel builds in this comment. For Numcodecs, in particular, please take a look at PR ( conda-forge/numcodecs-feedstock#17 ), which would build both conda and wheel packages from the same build.

  2. alimanfoo commented on Mar 5, 2018

    @alimanfoo
    alimanfoo
    on Mar 5, 2018 via email
    Member

  3. jakirkham commented on Mar 5, 2018

    @jakirkham
    jakirkham
    on Mar 5, 2018
    MemberAuthor

    Thanks @alimanfoo.

    Right now my feeling is reusing our work at conda-forge to do this is the best path. We will have more resources generally (compute, manpower, storage, etc.) to push this forward. Not to mention we can easily store the wheels on anaconda.org right along side our conda packages.

    Expect the wheel folks will be interested in smoothing out the compatibility issues once they see some demonstrated success of wheels in conda-forge. Perhaps by making manylinux2 for instance or broadening the manylinux spec to include arbitrary glibc. There's already some pressure on them to do this as tensorflow and other packages are breaking the manylinux spec anyways ( tensorflow/tensorflow#8802 ). Also CentOS 5 is an unsupported OS as of last year. So they will have to update and they know this ( pypa/manylinux#96 ).

    The technical issues of getting wheels in there are minimal IMHO. Will have to see how others in conda-forge feel about having wheels included.

  4. jakirkham commented on Mar 5, 2018

    @jakirkham
    jakirkham
    on Mar 5, 2018
    MemberAuthor

    cc @rabernat

  5. jeromekelleher commented on Apr 25, 2018

    @jeromekelleher
    jeromekelleher
    on Apr 25, 2018
    Member

    Just wanted to say that binary wheels on PyPI would be useful for me. I like to test my code using both the pip and conda infrastructure. Compiling numcodecs for each CI build takes quite a bit of time and a bdist_wheel would be very handy.

    I'm loving zarr/numcodecs by the way; thanks for all the work!

  6. alimanfoo commented on Apr 26, 2018

    @alimanfoo
    alimanfoo
    on Apr 26, 2018 via email
    Member

  7. jakirkham commented on Apr 27, 2018

    @jakirkham
    jakirkham
    on Apr 27, 2018
    MemberAuthor

    Well we are in the midst of migrating to conda-build 3. So that is pretty much zapping all developer time. Not to mention other work activities outside of conda-forge. IOW it is stalled due to lack of time.

    That said, we are able to build wheels using the conda-forge recipe using PR ( conda-forge/numcodecs-feedstock#17 ). There are of course the compatibility issues we already discussed mainly on Linux, but it looks like PyPA is working on a new manylinux2 spec ( pypa/manylinux#152 ), which would resolve this. The main issue is uploading the wheels actually. The current upload script would need to be reworked for wheels ( conda-forge/conda-forge-ci-setup-feedstock#6 ). Not sure what that entails yet.

  8. alimanfoo commented on Apr 30, 2018

    @alimanfoo
    alimanfoo
    on Apr 30, 2018
    Member

    Thanks @jakirkham. Am I right in understanding that conda-forge/numcodecs-feedstock#17 is now successfully building wheels on all platforms?

  9. jakirkham commented on Apr 30, 2018

    @jakirkham
    jakirkham
    on Apr 30, 2018
    MemberAuthor

    It does appear that way. Hadn't had time to check since that comment. :)

    The main challenge is we don't have a way to upload them from the CIs yet. Though one could certainly build those locally and upload them to PyPI or wherever one wishes.

  10. alimanfoo commented on May 1, 2018

    @alimanfoo
    alimanfoo
    on May 1, 2018 via email
    Member

  11. jakirkham commented on Nov 9, 2019

    @jakirkham
    jakirkham
    on Nov 9, 2019
    MemberAuthor

    So these days there is conda-press, we could try using this to make wheels from the conda-forge packages.

    ref: https://regro.github.io/conda-press-docs/

  12. jakirkham commented on Nov 9, 2019

    @jakirkham
    jakirkham
    on Nov 9, 2019
    MemberAuthor

    So I tried creating some wheels from the conda-forge packages using conda-press and uploaded them to my Anaconda channel. If some people could please try these out and let us know if this works, that would be very help. 🙂

    pip install -i https://pypi.anaconda.org/jakirkham/label/test/simple numcodecs

    ref: https://anaconda.org/jakirkham/numcodecs

  13. jrbourbeau commented on Nov 9, 2019

    @jrbourbeau
    jrbourbeau
    on Nov 9, 2019
    Member

    I'm getting this error:

    (numcodecs-test-2) ➜  ~ pip install -i https://pypi.anaconda.org/jakirkham/label/test/simple numcodecs
Looking in indexes: https://pypi.anaconda.org/jakirkham/label/test/simple
Collecting numcodecs
  Downloading https://pypi.anaconda.org/jakirkham/label/test/simple/numcodecs/0.6.4/numcodecs-0.6.4-0_py37h4a8c4bd-cp37-cp37m-macosx_10_9_x86_64.whl (1.2MB)
     |████████████████████████████████| 1.2MB 724kB/s
ERROR: Could not find a version that satisfies the requirement numpy (from numcodecs) (from versions: none)
ERROR: No matching distribution found for numpy (from numcodecs)

  14. jakirkham commented on Nov 9, 2019

    @jakirkham
    jakirkham
    on Nov 9, 2019
    MemberAuthor

    Yeah I think that is because pip is assuming this is a full mirror. We probably have to tell it to supplement it with PyPI or manually install the missing pieces.

    That said, it appears some content was missing from the wheels. Am following up in issue ( conda-incubator/conda-press#52 ).

  15. jni commented on Nov 9, 2019

    @jni
    jni
    on Nov 9, 2019

    @jakirkham I thought the manylinux problems had been fixed upstream recently? Something about manylinux2010? Anyway, I'm generally ok with requiring Linux users to have a compiler, so even just providing windows and Mac wheels would make me happy...

  16. 18 remaining items

  17. rabernat commented on Sep 4, 2020

    @rabernat
    rabernat
    on Sep 4, 2020
    Contributor

    Furthermore, if we could frame this as a service (rather than a donation), any number of universities or federal labs could potentially support it.

  18. hammer commented on Sep 4, 2020

    @hammer
    hammer
    on Sep 4, 2020

    @rabernat I appreciate your optimism but I am skeptical that obtaining funding for improving the build and release process of an infrastructure open source library is possible from other sources. I'd be happy to share my experiences asking for money for this kind of work in the past!

    I'm personally offering to make this donation once I get the EIN from @FrancescAlted, so luckily in this case it will not be necessary to seek funding from those alternative sources.

  19. rabernat commented on Sep 4, 2020

    @rabernat
    rabernat
    on Sep 4, 2020
    Contributor

    I'm personally offering to make this donation once I get the EIN

    🥇 Well this is of course much easier! Thanks for your amazing generosity!

  20. FrancescAlted commented on Sep 4, 2020

    @FrancescAlted
    FrancescAlted
    on Sep 4, 2020

    @hammer For the record, I have replied to your private message. In the name of the Blosc team, thank you so much!

  21. joshmoore
    closed this as completedin #224on Sep 7, 2020

  22. joshmoore commented on Sep 7, 2020

    @joshmoore
    joshmoore
    on Sep 7, 2020
    Member

    Sorry, closing was automatically triggered via #224. Happy to (have) re-open(-ed), or a new issue can be started for blosc build integration.

  23. FrancescAlted commented on Nov 19, 2020

    @FrancescAlted
    FrancescAlted
    on Nov 19, 2020

    Now that @hammer donation is complete (thanks Jeff!) we are working towards producing wheels for Blosc. Right now our focus is python-blosc, but perhaps you guys are rather more interested on C-Blosc itself. We have two possibilities:

    1. Provide both static and dynamic libraries in the python-blosc wheel. That means that after installing it via pip, the libraries are accessible to other projects that just need C-Blosc libraries (like zarr/numcodecs).

    2. Do another wheel for just the C-Blosc library. AFAIK, this is possible (people like cmake folks or Intel are doing this for pure C/C++ libraries for long time), but we are not sure on how to proceed in this case, so we would be grateful to get some advice here.

    Indeed 1) is easier for us, but we would like to keep the zarr team as happy as possible ;-). Thoughts?

  24. jakirkham commented on Nov 19, 2020

    @jakirkham
    jakirkham
    on Nov 19, 2020
    MemberAuthor

    Thanks Jeff! 😄

    Originally was also thinking 1 would make the most sense. So agree with that as well. There are some parts of Numcodecs that link directly to things like LZ4, Zstd, etc. as included by Blosc. That said, there are also other Python libraries that supply wheels and Conda packages for those. So we could handle them that way. Alternatively we could do the same thing we are doing now just with the sources of those libraries. IOW I think we have enough other options to handle C/C++ without issues.

    If anyone else here has thoughts, feel free to share 🙂

  25. Czaki commented on Nov 19, 2020

    @Czaki
    Czaki
    on Nov 19, 2020
    Contributor

    I also meet the idea of writing some small wrapper that loads c/c++ extension during runtime rather than linking during compilation. This allows using the external library without going through python code (speedup) but without repetition of a binary blob.

    But I do not implement such a thing.

  26. FrancescAlted commented on Nov 20, 2020

    @FrancescAlted
    FrancescAlted
    on Nov 20, 2020

    Thanks for the feedback. For now, we will work towards providing wheels for python-blosc (option 1), as suggested. However, we will still try to pursue option 2) so as to build wheels for the pure C-Blosc library (and perhaps make it a dependency for python-blosc). Again, any info on how to create a wheel for a pure C project is welcome.

    Will keep you informed.

  27. jakirkham commented on Nov 20, 2020

    @jakirkham
    jakirkham
    on Nov 20, 2020
    MemberAuthor

    Thanks Francesc!

  28. FrancescAlted commented on Dec 15, 2020

    @FrancescAlted
    FrancescAlted
    on Dec 15, 2020

    Hi there!

    We have been hard at work generating wheels for python-blosc and we have come with a pretty comprehensive set at: https://pypi.org/manage/project/blosc/release/1.9.3.dev0/ (this release is yanked, so install it via pip install blosc==1.9.3.dev0).

    We are including not only the python-blosc extension inside the wheels, but also the static and dynamic libraries of C-Blosc (I think this is what you need for numcodecs). This is our first attempt at building wheels, but we have been successful on testing the libraries on Mac and Linux already (Windows is still being explored, but the libraries are there already for you to test).

    Could the Zarr team please give this a try and tell us how it goes?

    BTW, the guys behind this effort have mainly been @oscargm98 and @aleixalcacer, and they will be glad to hear and help you.

  29. FrancescAlted commented on Dec 21, 2020

    @FrancescAlted
    FrancescAlted
    on Dec 21, 2020

    Hi. We have manually (and not only via CI) tested the wheels on Windows, Linux and Mac, and they look good to us. We have made some detailed instructions on how to compile some C-Blosc examples by using the python-blosc wheels at: https://github.com/Blosc/c-blosc/blob/master/COMPILING_WITH_WHEELS.rst.

    Our intention is to release soon a new version of C-Blosc (1.21.0) with updated codecs (specially the new zstd 1.4.8), and shortly after an updated version of python-blosc (1.10.0) so as to use latest C-Blosc sources and do a release of it with the (official) wheels. It would be nice if you can provide feedback for Zarr (please note that, until we do the new release, you still need to use pip install blosc==1.9.3.dev0). Thanks!

  30. oscargm98 commented on Dec 23, 2020

    @oscargm98
    oscargm98
    on Dec 23, 2020

    Hi there! We have just released a new version of python-blosc with binary wheels fully implemented: https://github.com/Blosc/python-blosc/releases/tag/v1.10.0

    Note that support for AVX2 run time discovery has been activated. That means that, if you use the libraries in this wheels, you can leverage the full capabilities of Intel CPUs. Please give it a spin and report back any possible flaws. Enjoy!

  31. hammer commented on Dec 23, 2020

    @hammer
    hammer
    on Dec 23, 2020

    Thanks @oscargm98! I've filed #262 to track using these wheels in the numcodecs build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions