Skip to content

CI: Setup compilation cache and use it for building the wheel - #31474

Closed
luhenry wants to merge 1 commit into
numpy:mainfrom
luhenry:ccache
Closed

luhenry wants to merge 1 commit into
numpy:mainfrom
luhenry:ccache

Conversation

@luhenry

@luhenry luhenry commented May 20, 2026

Copy link
Copy Markdown
Contributor

PR summary

On slower platforms like linux-riscv64, it's very beneficial to use ccache to speed up the build. It also benefits other platforms as well.

cc @rgommers done as follow up to #30995 (comment)

AI Disclosure

AI tools used for research and understanding of the problem. Artisanal coding by (human) hands

@luhenry luhenry changed the title Setup compilation cache and use it for building the wheel CI: Setup compilation cache and use it for building the wheel May 20, 2026
@luhenry

luhenry commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

Some gains observed at https://github.com/luhenry/numpy/actions/runs/26189048070:

wheel run 1 (cold cache) run 2 (hot cache)
Build wheel cp312-manylinux_x86_64- 182s 20s
Build wheel cp312-musllinux_x86_64- 271s 26s
Build wheel cp312-manylinux_aarch64- 93s 14s
Build wheel cp312-musllinux_aarch64- 124s 16s

The cost of installing ccache and restoring+saving the cache also doesn't exceed the savings we get on compilation time. So it's a net win overall.

@luhenry
luhenry marked this pull request as ready for review May 20, 2026 21:25
On slower platforms like linux-riscv64, it's very benefitial to use ccache
to speed up the build. It also benefits other platforms as well.
@andyfaff

Copy link
Copy Markdown
Member

I'm -0.5 on this PR. The reason being is that I'd like to keep this wheels.yml as close as possible to the setup we have for numpy/numpy-release.. One of the design principles on that repo is that it does clean builds from scratch, with no caching (#29178). Therefore, I'd like to keep caching out of this workflow.

@andyfaff

Copy link
Copy Markdown
Member

Appropriate use of caching in the other workflows is fine.

@luhenry

luhenry commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

I'm -0.5 on this PR. The reason being is that I'd like to keep this wheels.yml as close as possible to the setup we have for numpy/numpy-release.. One of the design principles on that repo is that it does clean builds from scratch, with no caching (#29178). Therefore, I'd like to keep caching out of this workflow.

Would you be open to have it enabled for select architectures like linux-riscv64 for example?

@andyfaff

Copy link
Copy Markdown
Member

Would you be open to have it enabled for select architectures like linux-riscv64 for example?

I think the best roadmap for this is to have a separate linux_riscv64.yml workflow for CI for this platform, rather than it being in the wheels workflow. That workflow can make use of caching.
Another of the numpy/numpy-release features is to not have any self-hosted runners. So if we are to eventually release wheels for this platform, then presumably it'll be done via QEMU. I'd want to be doing the same in numpy/numpy:wheels.yml.

@luhenry

luhenry commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

The ccache on riscv64 brings the Building wheel step from 971s to 155s

@andyfaff

Copy link
Copy Markdown
Member

The ccache on riscv64 brings the Building wheel step from

That's great for a linux_riscv64 job, but any caching and the native build won't be being used for wheel build and distribution.

@rgommers

Copy link
Copy Markdown
Member

I think the best roadmap for this is to have a separate linux_riscv64.yml workflow for CI for this platform

That's exactly what I suggested as well in #30995 (comment) (and including running it on main to upload the ccache cache).

The ccache on riscv64 brings the Building wheel step from 971s to 155s

That's great, that will bring the job runtime to a good level.

@luhenry

luhenry commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Following 1, I'll close this PR and open a new one with linux_riscv64.yml using this ccache-based workflow.

@luhenry luhenry closed this May 22, 2026
@luhenry
luhenry deleted the ccache branch May 22, 2026 08:51
@luhenry
luhenry restored the ccache branch May 22, 2026 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants