Skip to content

Comments

Build once per platform & use ccache#7756

Draft
stefanv wants to merge 33 commits intoscikit-image:mainfrom
stefanv:cache-meson-build-dir
Draft

Build once per platform & use ccache#7756
stefanv wants to merge 33 commits intoscikit-image:mainfrom
stefanv:cache-meson-build-dir

Conversation

@stefanv
Copy link
Member

@stefanv stefanv commented Mar 15, 2025

See if we can speed up builds by caching meson build dir

@stefanv stefanv added the 🤖 type: Infrastructure CI, packaging, tools and automation label Mar 15, 2025
@stefanv stefanv force-pushed the cache-meson-build-dir branch from 43c6aa3 to f6c16d0 Compare March 15, 2025 01:00
@mdhaber
Copy link
Contributor

mdhaber commented Mar 15, 2025

It looks like it's successful at restoring the cache, but it's not faster to build. It still counts from 1 to 165. Is it using the cache?

In retrospect, sorry for suggesting caching the build folder! I was just interested in the idea of caching the build, but knew I didn't know how to do it.

@stefanv stefanv marked this pull request as draft March 15, 2025 03:26
@stefanv
Copy link
Member Author

stefanv commented Mar 15, 2025

Timestamps may be a problem. Not sure meson cache is meant to be restored like this.

@lagru
Copy link
Member

lagru commented Mar 16, 2025

Could ccache be another option (in case you haven't thought of it already)?

@stefanv
Copy link
Member Author

stefanv commented Mar 16, 2025

Yes, that's what I realized we'd have to do! You can't cache meson this way because of timestamps.

@stefanv stefanv changed the title Cache meson build dir Cache builds using ccache Mar 17, 2025
@stefanv stefanv force-pushed the cache-meson-build-dir branch from 1192698 to 4886285 Compare March 17, 2025 05:14
@stefanv
Copy link
Member Author

stefanv commented Mar 17, 2025

We should not be building N times for a given Python version on a given platform. Build once, test several configurations?

@mdhaber
Copy link
Contributor

mdhaber commented Mar 17, 2025

What configurations do you have other than different platforms/Python versions? Different versions of other dependencies?

(That said, this will already be a huge improvement, so besides getting the cache working, optimizations could be a follow-up.)

@stefanv
Copy link
Member Author

stefanv commented Mar 17, 2025

For any given platform, we build with a known set of dependencies. But, you're right, that can be a secondary optimization.

@stefanv
Copy link
Member Author

stefanv commented Mar 18, 2025

It doesn't look like ccache helped here 🤔

@stefanv stefanv force-pushed the cache-meson-build-dir branch from 0f5a1f0 to 2665540 Compare March 19, 2025 08:01
@stefanv
Copy link
Member Author

stefanv commented Mar 19, 2025

OK, the caching is roughly working. Takes a build down from ~4.5mins to ~1min. Will need to adjust the CI a bit, because looks like the caching is too granular. E.g., the pre-build still rebuilds from source, despite having access to a cache (hits: <2%).

@stefanv stefanv force-pushed the cache-meson-build-dir branch from 5e6d141 to 5617def Compare March 19, 2025 18:42
@stefanv stefanv force-pushed the cache-meson-build-dir branch from 6cbd4d8 to 93a9725 Compare March 19, 2025 19:07
@lagru lagru self-requested a review March 20, 2025 18:24
@stefanv stefanv changed the title Cache builds using ccache Build once per platform & use ccache Apr 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🤖 type: Infrastructure CI, packaging, tools and automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants