Missing package: openPMD-api

Package name: openPMD-api
Issue type: Missing package
Link to PyPI page: https://pypi.org/project/openPMD-api
Link to piwheels page: https://www.piwheels.org/project/openPMD-api
Version: latest (0.8.0a0)
Build dependencies: https://openpmd-api.readthedocs.io/en/latest/dev/dependencies.html
Build options: https://openpmd-api.readthedocs.io/en/latest/dev/buildoptions.html
Python version: all
More information:

I recently created manylinux2010 build scripts within docker in openPMD/openPMD-api#523 which builds all dependencies as statically linked.

Is it possible to add/re-use those for piwheels?

FYI the latest release (0.8.0a0) hasn't been attempted yet because we're not currently building.

0.7.1a failed on both Jessie and Stretch. Jessie due to requiring Python 3.5, Stretch due to requiring cmake 3.11. Here's the output from that build attempt: http://paste.debian.net/1087972/

Debian Stretch has cmake 3.7.2, but Buster has 3.13.4. Stretch-backports this version too. See https://packages.debian.org/buster/cmake

Don't think we should update cmake on the Stretch builders, but we could build it off-system in Stretch with a newer cmake and import the wheel if that works.

I've successfully built it on Buster - does this look ok to you? https://bennuttall.com/files/openPMD_api-0.8.0a0-cp37-cp37m-linux_armv7l.whl

piwheels works by automating building wheels from packages with essentially pip wheel <package>, so apart from adding build dependencies we don't generally do custom build options or manually build from github. Looks like it's ok in Buster, which will soon become the supported distro. Might be able to get it working for Stretch.

Thank you for the information, that's super helpful!

Regarding CMake, we can just use the piwheels cmake versions. I probably have to ~~"search harder" for its install with setup.py since the installed cmake wheel is usually not in the user's path~~ add a pyproject.toml. will address that on my side for the next release :)

regarding dependencies (especially HDF5 and ADIOS1), it looked to me that you seam to prefer shared dependencies that the user has then to install on top of the wheel?
https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/

On PyPI/manylinux2010, I plan to upload HDF5 & ADIOS1 as static deps which can be more recent and avoid additional deps to be installed after the wheel (also, I am afraid to loose some that are unable to google the "shared lib not found" problems).

apart from adding build dependencies

Either way, how can we specify such additional build dependencies? By default, my scripts search for "take everything you can find and continue without if not found" which potentially means it has no HDF5/ADIOS1 support at all. What is the apt package list that is installed when we build the wheels? E.g. the linked buster wheel looks like it has not found HDF5 during build.

Thanks a lot for your help! :)

On PyPI/manylinux2010, I plan to upload HDF5 & ADIOS1 as static deps

How do I build a manylinux2010 wheel?

regarding dependencies (especially HDF5 and ADIOS1), it looked to me that you seam to prefer shared dependencies that the user has then to install on top of the wheel?

One of the new features of piwheels is that we're now able to calculate dependencies and list them on the project page, e.g. https://www.piwheels.org/project/tensorflow/ so once we're building again, your latest version will be built and dependencies will be calculated and shown there.

There's a suggestion (#117) to try to make import errors more helpful.

Either way, how can we specify such additional build dependencies?

If installing extra apt packages on the builders will help, you can request them (in this issue).

On PyPI/manylinux2010, I plan to upload HDF5 & ADIOS1 as static deps

How do I build a manylinux2010 wheel?

manylinux2010 (PEP-571) is a portability subset of allowed symbols and external dependencies. It's not super relevant for piwheels since you build for one distro anyway, but the static build scripts that I use (Dockerfile, instructions) would be the same logic to avoid manually installing apt-packages alongside a wheel. It also helps me to use newer dependency versions than shipped with in a distro.

One of the new features of piwheels is that we're now able to calculate dependencies and list them on the project page, e.g. https://www.piwheels.org/project/tensorflow/ so once we're building again, your latest version will be built and dependencies will be calculated and shown there.

That's super cool, that would totally work for me as well (as long as the buster deps are new enough, which they should be).

If installing extra apt packages on the builders will help, you can request them (in this issue).

Perfect, thanks! I would need the following apt packages for openPMD-api:

libadios-dev
libhdf5-dev

libadios-dev is likely only picked up for Buster and newer (shipped version), yet the installer will just skip it for older distributions.

This is successfully building on Python 3.7 but not 3.5 - is that ok, or is 3.5 fixable?

Can you show me the build log? I am not seeing it at https://www.piwheels.org/project/openpmd-api/

The next release will probably fix some build issues you are seeing, since it will ship this fix: openPMD/openPMD-api#527 So yes, Python 3.5 is fixable :)

Also, where can I add the required sudo apt install libadios-dev libhdf5-dev line for this package as well? :)

Add libadios-dev packages to build plaves for package 'openPMD-api'

@bennuttall

@bennuttall with piwheels/piwheels#190 merged, the latest PyPI release 0.9.0a1 should build on Python 3.5-3.7 targets of piwheels. Can you confirm?

Passed on 3.7. Here's the build log for 3.5: https://paste.ubuntu.com/p/mw4zNkkcWc/

Thanks!
Hm, two interesting things here: first of all it tries to download CMake from pypi instead of piwheels and second it seams to lack a dependency while installing CMake itself, skbuild, which means it decided to download a source package for CMake:

File "/tmp/pip-install-pls3e7ps/cmake/setup.py", line 7, in <module> 
from skbuild import setup
ImportError: No module named 'skbuild'

Here is its upstream repo: https://github.com/scikit-build/cmake-python-distributions

But actually, I do not understand why it's not simply taking the piwheels wheel for that? My pyproject.toml is actually quite simple and fits what is available on piwheels.

@waveform80

ping @waveform80 and @bennuttall just a gentle reminder to update the pip.conf to include piwheels own binaries as dependencies during builds of new packages.

@waveform80

pingpong @waveform80 and @bennuttall :)

@bennuttall

Sorry for the late response - horribly busy at the moment!

I'm a little reticent about this because the command we use to build wheels includes the --no-deps switch (in other words, build with the dependencies you've got and if they're not sufficient just fail). In the case of wheels that have unsatisfied dependencies (stuff outside that which we install in the deployment), we don't want the build-slave modifying its state in such a way that it affects the next build it performs. We can't actually stop people from doing things like calling pip from their setup.py (the only thing we can do about that is periodically reset the builders, which I do on occasion), but it's not something I'm keen to encourage.

Anyway, that's why I'm not convinced that adding piwheels back into /etc/pip.conf is a great idea. Having said that, we should include scikit-build in the set of deployed dependencies as I've noticed it's becoming increasingly important as a build-dep.

@bennuttall any chance you can get that installed on the build-slaves at some point? It'll need to be via pip instead of apt (I don't think it's packaged in any version of raspbian) so it'll need to go next to kervi et al in the deployment script. I think the package is named scikit-build on pypi (rather than skbuild which is the module it installs).

Thanks for the details!

I am not sure I follow the "build with the dependencies you've got" step - the dependencies I need are listed in pyproject.toml (build-time) and requirements.txt (run-time) only. Building CMake currently fails, which is listed in my pyproject.toml.

Yes, adding scikit-build would solve my specific problem as well since it is caused by the missing/failing CMake dependency which needs this, but compiling it again instead of using the readily available wheel from piwheels looks a bit expensive to me.

I do not fully understand why this affects subsequent builds but this seams like an isolation problem in the builders to me. At least with pyproject.toml dependencies are, as far as I can see, gone after install of the main project. Would a new, fresh virtual environment for each build potentially isolate them more? (Could be removed after each build.)

Been a while since we updated this issue but now scikit-build is installed on builders, the package is successfully building on Buster.

Thanks! But it does not seem to pick up libadios-dev yet, do you have a log on what the install step produces? piwheels/piwheels#190

…

On January 19, 2020 6:26:04 AM PST, Ben Nuttall ***@***.***> wrote: Been a while since we updated this issue but now `scikit-build` is installed on builders, the package is successfully building on Buster. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #23 (comment)

Oh, sorry. I just looked and saw that the wheels were there and thought the issue was resolved. Should have read more of the comments! I have installed libadios-dev on all the builders. I'll have a go at building it now and see what happens.

Ah dang, forgot again that adios1 is a static library and therefore won't show up in the apt-get list on piwheels :)

Ok so it's succeeded for cp37m with these instructions:

sudo apt install libsz2 libaec0 libhdf5-103
sudo pip3 install openPMD-api

Do these instructions work, or is there more to do to get it to work?

Yep, that's it. The compile-time dependency libadios-dev is static and should not show up to users anymore (besides transitive dependencies it could potentially carry).
Thanks a lot!

Excellent. Sorry for the delay - glad we got it sorted in the end.

Thanks a lot for your splendid support :)

Navigation Menu

Repository navigation

Missing package: openPMD-api #23

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Missing package: openPMD-api #23

Description

Activity

bennuttall commented on Jun 15, 2019

ax3l commented on Jun 16, 2019

bennuttall commented on Jun 16, 2019

ax3l commented on Jun 16, 2019

bennuttall commented on Jul 18, 2019

ax3l commented on Jul 18, 2019

ax3l commented on Jul 18, 2019

ax3l commented on Jul 26, 2019

bennuttall commented on Aug 7, 2019

ax3l commented on Aug 8, 2019

5 remaining items

ax3l commented on Aug 27, 2019

ax3l commented on Sep 9, 2019

waveform80 commented on Sep 12, 2019

ax3l commented on Sep 12, 2019

bennuttall commented on Jan 19, 2020

ax3l commented on Jan 19, 2020

bennuttall commented on Jan 19, 2020

ax3l commented on Jan 20, 2020

bennuttall commented on Jan 20, 2020

ax3l commented on Jan 21, 2020

bennuttall commented on Jan 21, 2020

ax3l commented on Jan 22, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions