Skip to content

Comments

Build 1.4 wheels with CIBW#3381

Merged
snowman2 merged 28 commits intorasterio:maint-1.4from
w8sl:wheels-1.4
Dec 4, 2025
Merged

Build 1.4 wheels with CIBW#3381
snowman2 merged 28 commits intorasterio:maint-1.4from
w8sl:wheels-1.4

Conversation

@w8sl
Copy link
Contributor

@w8sl w8sl commented Aug 17, 2025

Build 1.4 wheels with GDAL 3.10.3 and CIBW 23.3 for 6 platforms:

Windows x64
Linux x86_64
macOS x86_64
macOS arm64

New:

Windows ARM64
Linux aarch64

Inspired by pyproj and shapely.
Tried with rasterio-wheels first but it didn't work for macOS 13
Included workaround for 6 test failing with click, as suggested by Copilot

@w8sl
Copy link
Contributor Author

w8sl commented Aug 17, 2025

Something is off with librpoj on linux. It was 4,5 mb on my previous build. It works and is linked correctly but has more sections. I will check flags and correct.

@w8sl
Copy link
Contributor Author

w8sl commented Aug 17, 2025

I have made attempt to make global flags to early and lost strip flags. I will test and fix it.

@sgillies
Copy link
Member

@w8sl thanks for demonstrating interest in this! Rasterio's wheel building system has been impacted by bit rot of GitHub runners, GitHub actions, and changes in library dependencies (like GDAL). Things are in a bad state at the moment. Help is appreciated.

That said, I have some big issues in mind and any new work has to partially solve or at least acknowledge some of them.

  1. GitHub is going in an unpredictable direction. It's under Microsoft's "AI" monetization org now. Should we be investing in more GitHub Actions YAML at all? I don't know the answer to this.
  2. Making installable binaries for Rasterio, if we're going to do it, has to be sustainable. I think that means there should be a small team of people who are paid to work on this, instead of volunteers that come and go.
  3. Making installable binaries for Rasterio has to be reasonably secure and safe. https://en.wikipedia.org/wiki/XZ_Utils_backdoor is the context here. At the very least, that means no third party GitHub actions in workflows, no unverified build dependencies, and some transparency about who is working on this stuff.

What do you think about this context for working on wheel builds?

@w8sl
Copy link
Contributor Author

w8sl commented Aug 18, 2025

@sgillies Thank you for this fascinating project ! To have all these libraries installed by just "pip install rasterio" is magic. Working with it is very interesting and has something to do with my daily work, which is more or less: "find what is broken and fix". I am not a developer, not programmer. My workflow is a proof of concept. It will not last very long: macos-13 is closing, manylinux2014 is EOL. (manylinux image may be replaced even with CIBW 2) I have no idea, how much work it needs to make it working with python 3.14 and CIBW 3.1.3. I have indeed replaced many download links for GitHub repositories whenever possible, as downloads from SourceForge.net were unstable. I hope, there are many developers using CIBW, who can help, while Multibuild is used by few projects only. I prefer to have a freedom to help, when I have idea and when I have time.

Regarding libraries: my first goal was to have wheels passing tests as 1.43 does. Second goal could be syncing versions with vcpkg which is only partially done. For security reasons, it may be better to keep this quarantine few weeks.

Best regards
Sylvester

@w8sl
Copy link
Contributor Author

w8sl commented Aug 20, 2025

Few modifications are required to build standard (non-free-threaded) Python 3.10-3.14 wheels with CIBW 3.1.4:

name: Build wheels
 uses: pypa/[email protected]
 env:
   CIBW_SKIP:  ' *cp3??t* '
   CIBW_REPAIR_WHEEL_COMMAND_LINUX: export 'LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:$LD_LIBRARY_PATH' && auditwheel repair -w {dest_dir} {wheel}

Revert to compiling LibTIFF with ./configure is necessary as cmake compilation of this particular library on macOS-13 doesn’t work correctly anymore.

For libraries, I needed working download links with switchable versions for testing. I’m sure it can be done more securely for releases. I chose wget for code simplicity. Another option was to keep fetch-unpack for some libraries and use wget for libraries from sourceforge.net.

It works very well on the main branch, after recent updates ! Thanks @QuLogic and @snowman2 !

@snowman2 snowman2 added ci dependencies Pull requests that update a dependency file enhancement packaging github_actions Pull requests that update GitHub Actions code labels Sep 18, 2025
@snowman2
Copy link
Member

Nice work @w8sl 👍

@w8sl w8sl changed the title Build 1.4 wheels with CIBW 2.23.3 Build 1.4 wheels with CIBW Sep 19, 2025
@w8sl
Copy link
Contributor Author

w8sl commented Nov 8, 2025

Tried to build with latest updates and it works fine including 3.14t wheels. Computations done by Rasterio are 2 x faster but pure Python code runs 25% slower.
Interestingly, with 3.14t wheels, fsspec failures are back on macOS 15.7.1, when tested locally (fixed on non free-threaded Python)
FAILED tests/test_pyopener.py::test_opener_fsspec_http_fs - rasterio.errors.RasterioIOError: /vsiriopener_27e6fdfd759941a1b71f43dcad56bde0/https://raw.githubusercontent.com/rasterio/rasterio/main/tests/data/float32.tif: Unknown error: 316 FAILED tests/test_pyopener.py::test_fsspec_http_msk_sidecar - rasterio.errors.RasterioIOError: /vsiriopener_7e9059cf7eef44d0b7a62bf50219fa98/https://raw.githubusercontent.com/rasterio/rasterio/main/tests/data/RGB2.byte.tif: Unknown error: 316

Three failures (to many files open), we have seen on GDAL latest are also appearing randomly on Python 3.14t:

FAILED tests/test_warpedvrt.py::test_warped_vrt_add_alpha - AssertionError: assert 2 == 1
 +  where 2 = len(['  1 N GTiff       -1 10x10x1 /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/pytest-of-runner/pytest-0/test_rasterize_like_raster_src0/pixelated_image.tif', '  2 N GTiff       -1 791x718x3 /Users/runner/work/rasterio/rasterio/tests/data/RGB.byte.tif'])
FAILED tests/test_warpedvrt.py::test_open_datasets - AssertionError: assert '1 N GTiff' not in 'Open GDAL Datasets:\n  1 N GTiff       -1 10x10x1 /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/pytest-of-runner/pytest-0/test_rasterize_like_raster_src0/pixelated_image.tif\n  1 N GTiff       -1 791x718x3 /Users/runner/work/rasterio/rasterio/tests/data/RGB.byte.tif\n'
  
  '1 N GTiff' is contained here:
    Open GDAL Datasets:
      1 N GTiff       -1 10x10x1 /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/pytest-of-runner/pytest-0/test_rasterize_like_raster_src0/pixelated_image.tif
  ?   +++++++++
      1 N GTiff       -1 791x718x3 /Users/runner/work/rasterio/rasterio/tests/data/RGB.byte.tif
FAILED tests/test_warpedvrt.py::test_warp_warp - AssertionError: assert 3 == 1
 +  where 3 = len(['  1 N GTiff       -1 10x10x1 /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/pytest-of-runner/pytest-0/test_rasterize_like_raster_src0/pixelated_image.tif', '  1 N GTiff       -1 791x718x3 /Users/runner/work/rasterio/rasterio/tests/data/RGB.byte.tif', '  1 N GTiff       -1 791x718x3 /Users/runner/work/rasterio/rasterio/tests/data/RGB.byte.tif'])

@snowman2 snowman2 mentioned this pull request Nov 9, 2025
@w8sl
Copy link
Contributor Author

w8sl commented Nov 9, 2025

The download for GIFLib by script is currently forbidden, making it
impossible to update the pull request at the moment.

@w8sl w8sl marked this pull request as draft November 11, 2025 13:10
@w8sl
Copy link
Contributor Author

w8sl commented Nov 17, 2025

It looks like download-artifact@v5 has crashed once on Windows-11-ARM + Python 3.14.
Some tests are failing randomly with GDAL 3.11.5 / failing consistently with GDAL 3.12

@snowman2 snowman2 added this to the 1.4.4 milestone Dec 4, 2025
@snowman2 snowman2 marked this pull request as ready for review December 4, 2025 21:44
@snowman2 snowman2 merged commit b5b14bd into rasterio:maint-1.4 Dec 4, 2025
41 of 44 checks passed
@snowman2
Copy link
Member

snowman2 commented Dec 4, 2025

Thanks @w8sl 👍

snowman2 pushed a commit that referenced this pull request Dec 5, 2025
@w8sl w8sl deleted the wheels-1.4 branch December 10, 2025 11:54
@sgillies
Copy link
Member

@w8sl @snowman2 does this great work give us a way to enumerate the version of dependencies in the wheels? Previously, there was a https://github.com/rasterio/rasterio-wheels/blob/main/env_vars.sh file that listed the key dependencies, but wasn't complete. For 1.4.4 and 1.5.0 can we say in the release notes which versions are included?

@snowman2
Copy link
Member

The versions are listed here:

PROJ_VERSION=9.7.1
. I think referencing them in the release notes is a good idea.

@w8sl
Copy link
Contributor Author

w8sl commented Dec 12, 2025

The list has been moved to rasterio/ci/config.sh and is now complete. Before merging, it was updated (for corresponding libraries only) to versions used by vcpkg build of GDAL 3.11.5, which can be seen here:
https://github.com/rasterio/rasterio/actions/runs/19277879572/job/55592645958#step:6:70

@w8sl
Copy link
Contributor Author

w8sl commented Dec 12, 2025

I have noticed now that on macOS x86_64 build, for some reason, libavif, libaom and libdav1d were also pulled by GDAL itself from Homebrew and included. It is a new format for GDAL, supported by 3.10+ I will check brew uninstall libavif

@w8sl
Copy link
Contributor Author

w8sl commented Dec 12, 2025

I am trying to get exactly 27 libraries on macOS and Linux x86_64/arm64 with these changes:
w8sl#9

@w8sl
Copy link
Contributor Author

w8sl commented Dec 12, 2025

There are uninvited unintentionally included libraries on macOS x86_64 build: libavif with dependencies and libzip.

Adding: libavif libzip at the end of this line solves the problem:

brew remove -f --ignore-dependencies liblerc webp zstd libtiff libxcb libxdcmp lcms2 xmlto ghostscript lz4 openjpeg xz giflib rtmpdump brotli snappy

Trying to add re-try logic is meaningless. Builds may fail anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci dependencies Pull requests that update a dependency file enhancement github_actions Pull requests that update GitHub Actions code packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants