Build 1.4 wheels with CIBW#3381
Conversation
Relevant scripts from files listed below, customized updated and merged into one file https://github.com/rasterio/rasterio-wheels/blob/main/env_vars.sh https://github.com/rasterio/rasterio-wheels/blob/main/config.sh https://github.com/multi-build/multibuild/blob/devel/common_utils.sh https://github.com/multi-build/multibuild/blob/devel/library_builders.sh
win-wheels.yaml from rasterio-wheels updated and extended, to build wheels for 6 platforms: - Windows x64 - Windows ARM64 - Linux x86_64 - Linux aarch64 - macOS x86_64 - macOS arm64
|
Something is off with librpoj on linux. It was 4,5 mb on my previous build. It works and is linked correctly but has more sections. I will check flags and correct. |
|
I have made attempt to make global flags to early and lost strip flags. I will test and fix it. |
|
@w8sl thanks for demonstrating interest in this! Rasterio's wheel building system has been impacted by bit rot of GitHub runners, GitHub actions, and changes in library dependencies (like GDAL). Things are in a bad state at the moment. Help is appreciated. That said, I have some big issues in mind and any new work has to partially solve or at least acknowledge some of them.
What do you think about this context for working on wheel builds? |
|
@sgillies Thank you for this fascinating project ! To have all these libraries installed by just "pip install rasterio" is magic. Working with it is very interesting and has something to do with my daily work, which is more or less: "find what is broken and fix". I am not a developer, not programmer. My workflow is a proof of concept. It will not last very long: macos-13 is closing, manylinux2014 is EOL. (manylinux image may be replaced even with CIBW 2) I have no idea, how much work it needs to make it working with python 3.14 and CIBW 3.1.3. I have indeed replaced many download links for GitHub repositories whenever possible, as downloads from SourceForge.net were unstable. I hope, there are many developers using CIBW, who can help, while Multibuild is used by few projects only. I prefer to have a freedom to help, when I have idea and when I have time. Regarding libraries: my first goal was to have wheels passing tests as 1.43 does. Second goal could be syncing versions with vcpkg which is only partially done. For security reasons, it may be better to keep this quarantine few weeks. Best regards |
|
Few modifications are required to build standard (non-free-threaded) Python 3.10-3.14 wheels with CIBW 3.1.4: Revert to compiling LibTIFF with ./configure is necessary as cmake compilation of this particular library on macOS-13 doesn’t work correctly anymore. For libraries, I needed working download links with switchable versions for testing. I’m sure it can be done more securely for releases. I chose wget for code simplicity. Another option was to keep fetch-unpack for some libraries and use wget for libraries from sourceforge.net. It works very well on the main branch, after recent updates ! Thanks @QuLogic and @snowman2 ! |
|
Nice work @w8sl 👍 |
|
Tried to build with latest updates and it works fine including 3.14t wheels. Computations done by Rasterio are 2 x faster but pure Python code runs 25% slower. Three failures (to many files open), we have seen on GDAL latest are also appearing randomly on Python 3.14t: |
|
The download for GIFLib by script is currently forbidden, making it |
+ minor tweaks
Add macos-15-intel build, Test on Python 3.14t
|
It looks like download-artifact@v5 has crashed once on Windows-11-ARM + Python 3.14. |
|
Thanks @w8sl 👍 |
|
@w8sl @snowman2 does this great work give us a way to enumerate the version of dependencies in the wheels? Previously, there was a https://github.com/rasterio/rasterio-wheels/blob/main/env_vars.sh file that listed the key dependencies, but wasn't complete. For 1.4.4 and 1.5.0 can we say in the release notes which versions are included? |
|
The versions are listed here: Line 1 in 4694e9d |
|
The list has been moved to rasterio/ci/config.sh and is now complete. Before merging, it was updated (for corresponding libraries only) to versions used by vcpkg build of GDAL 3.11.5, which can be seen here: |
|
I have noticed now that on macOS x86_64 build, for some reason, libavif, libaom and libdav1d were also pulled by GDAL itself from Homebrew and included. It is a new format for GDAL, supported by 3.10+ I will check brew uninstall libavif |
|
I am trying to get exactly 27 libraries on macOS and Linux x86_64/arm64 with these changes: |
|
There are Adding: rasterio/.github/workflows/build-wheels.yaml Line 174 in 4694e9d Trying to add re-try logic is meaningless. Builds may fail anyway. |
Build 1.4 wheels with GDAL 3.10.3 and CIBW 23.3 for 6 platforms:
New:
Inspired by pyproj and shapely.
Tried with rasterio-wheels first but it didn't work for macOS 13
Included workaround for 6 test failing with click, as suggested by Copilot