Notes on building wheels

Adding new packages

  1. Create a new fork of the project you want to build under https://gitlab.com/riseproject/python.

  2. Add the new project to the “Allow CI job tokens from the following projects to access this project” list under CI/CD settings of the manylinux and wheel_builder project. If this is not done your new project won’t be able to download the manylinux image and won’t be able to upload its wheels when built.

  3. Provide a custom configuration CI file for the project in the “CI/CD configuration file” setting under CI/CD - General pipelines. When adding a new project I usually create a new branch on wheel_builder for the new CI file and point the project’s configuration file setting at the branch, e.g.,

wheel_builder/pandas/gitlab-ci.yml@riseproject/python/wheel_builder:dev/build-pandas

I push and amend commits on this branch until the build is working.

  1. Try a build. You may need to turn on AutoDevOps before gitlab will allow you to trigger a manual build. Once the build starts you can disable AutoDevOps.

  2. Once the build is working correctly, the branch can be merged into main and the “CI/CD configuration file” setting of the project can be updated to point to main rather than to the branch.

Project Mirrors

We maintain a fork of the upstream project for each package that we build. These forks are really mirrors as we never modify them, apart from pulling in the latest chanegs from upstream. They should be commit by commit identical to the upstream projects. This is the reason the CI files live in the wheel_builder project rather than in the repos of the projects to which they pertain. This way we can avoid modifying the mirrors of the upstream projects.

There’s one exception here, manylinux. For manylinux we create new branches in the manylinux fork with our patches.

Builder

The RISE python project has its own runner. Specify

tags: ['python-wheels-xl']

in your jobs to use it. It’s faster than the gitlab runners.

Building new versions of existing projects

Building newer version of the packages supported by wheel builder is not a fully automated process, even for projects that we don’t need to patch. There are a few problems here:

  1. Upstream projects typically specify some configuration options for cibuildwheel. These options can be in the project.toml file or in the CI scripts of the CI used to build the wheels, e.g., yaml files under .github. CIBuildwheel options specified in the project.toml file will be automatically picked up by our build system. Those specified in github action files will not and so need to be manually copied into our gitlab-ci.yml files.

  2. We typically specify a value for CIBW_ENVIRONMENT in order to pass the URL of our package registry to the pip that runs inside the build wheel. This is needed so that pip can use our registry for downloading packages when building wheels. When building scipy, cibuildwheel needs to download numpy, and we want it to download our numpy rather than building the pypi source distribution. Overriding CIBW_ENVIRONMENT can cause an issue if the upstream project uses CIBW_ENVIRONMENT for other reasons. We must make sure that we add to CIBW_ENVIRONMENT rather than replacing it.

So for the simple projects that do not require patching, before updating the appropriate repo and building a new version, perform the following steps.

  1. Check that there are no changes to the CIBW settings in the CI files for that project, between the old and new releases. If there are, our .gitlab CI files will need to be updated.

  2. Check to see whether there are any changes to the way the project uses CIBW_ENVIRONMENT and make sure that they are updated correctly in our gitlab files.

There’s another issue. We have one gitlab-ci.yml per project. However, that file has to be able to build every version of that project. So if changes are needed, the changes cannot be simply applied directly to the gitlab-ci.yml file as doing so would prevent us from building the previous versions. gitlab variables will need to be used and set conditionally on the release tag so that the correct values are used for each release.

Once you’re happy nothing’s changed, or you’ve already made the changes and committed them to wheel_builder, just update the relevant project’s fork, pushing all the relevant tags, and the new versions should build automatically.

If the project is patched, you need to check whether the patches are still required with the newer version of the project. If not (and they might not be if they’ve been merged upstream) they can be dropped. Otherwise they must be rebased and placed into the appropriate folder under wheel_builder/<PROJECT>/patches/<TAG>. Currently, the gitlab-ci.yml file of projects we patch only enable the build for specific tags, that we’ved patched, so the workflow rules for the project will need to be updated to include the new release.