You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Config: add cache key to save state between builds #9183
I'm not sure if this is a too-requested feature, but I wanted to open this issue to start the conversation at least and collect 👍🏼's via GitHub reactions in case people want this feature to be implemented.
Currently, we are allowing users to use build.jobs to run custom commands in the middle of the build process. There are some projects that are already running some CPU/bandwidth intense actions there and they may have some benefit by caching some directories.
Initial implementation
I'm thinking the basic implementation for this would be something like:
You raise are good questions. However, I'm still not sold on caching being important for our platform, and it's mostly this:
what's the minimal time where using a cached directory makes it faster than running the command itself?
This was the big blocker for the initial version of this. The speed increase was negligible for builds when we tested this. Very likely, the user is only going to notice a speed increase when doing something computationally expensive, and even then there isn't a guarantee it will be noticeably faster.
We do differ from CI services in this regard. The vast majority of projects do not need or benefit from the caching we can provide, as most projects don't have computationally expensive operations. They do instead use bandwidth, but this is what we noticed doesn't translate to speed improvements when caching.
Computationally expensive tasks are mostly limited to doc generation, and mostly for large projects. This we don't want to cache either way though. We do know that build tool compilation can be expensive, but that doesn't need a configurable cache even.
@humitos in your testing weren't prebuilt binaries marginally faster than compiling binaries?
This does seem like a place to apply some effort, but we should look at what tools users are installing and really see if anyone is compiling uncached build tools before we assume users need more complexity here.
Though I was excited initially about the prospect of caching, unfortunately the performance really didn't back up the feature at all. We learned this lesson already, so not sure this is going to be valuable unless we're trying something new.
I would say to make progress here, we should find projects that are using uncached build tools and compare timing of full compilation vs downloading a compiled tarball. If users are really struggling with caching, they do have ways out, like using build.tools to download pre-built binaries.
@humitos in your testing weren't prebuilt binaries marginally faster than compiling binaries?
When I originally implemented the feature, we were interested in caching a large git repository, not binaries. So, for that particular case, we didn't notice a big difference and we decided to remove the complexity because of that.
Now, with the addition of build.tools, there is a big noticeable difference between having a pre-compiled Python version in the cache, downloading and untar it compared with downloading its source code and compiling it on-the-fly. I'm mentioning Python in particular here because it does not happens with all the build.tools we support. For example, when installing nodejs, asdf just downloads a pre-compiled version of it and it's mainly the same time required.
I opened this issue to discuss this between us but mainly to start collecting user feedback as 👍🏼's and also use cases where they can explain to us how they would use it if we implement this.
Yeah users might have some good use cases here that we're not aware of. Collecting feedback is wise.
But it's worth noting the limitations that we found and that our caching mechanism only might help computationally heavy tasks that are not doc generation tasks. The questions above to lean into implementation, but I feel the question we need to answer first is much higher level.
On paper, caching looks great. But in application, and with it's limitations, we learned it's not super effective for most workloads.
When I originally implemented the feature, we were interested in caching a large git repository
We were also caching python packages installed, correct? Both operations are network operations primarily, which is why we didn't notice an increase in speed with caching -- S3 wasn't faster than ingress traffic.
Now, with the addition of build.tools, there is a big noticeable difference between having a pre-compiled Python version in the cache,
So yeah, this is an area where we could do something. But user-configurable cache isn't required for this, and is a rather maximal option. The minimum viable option here is to expand what build.tools we prebuild. Again, we should see what users are actually doing though. I suspect we're over-optimizing here for an unknown user profile.
We were also caching python packages installed, correct?
I think so, yes. We were caching the whole folder for the particular version. So, that also included the virtualenv already created with all the packages installed.
At this point, I don't think we need user-configurable cache. Let's see what feedback the community can share with us and if that feedback matches with an implementation and the benefits we can provide there.
Activity
agjohnson commentedon Jun 1, 2022
You raise are good questions. However, I'm still not sold on caching being important for our platform, and it's mostly this:
This was the big blocker for the initial version of this. The speed increase was negligible for builds when we tested this. Very likely, the user is only going to notice a speed increase when doing something computationally expensive, and even then there isn't a guarantee it will be noticeably faster.
We do differ from CI services in this regard. The vast majority of projects do not need or benefit from the caching we can provide, as most projects don't have computationally expensive operations. They do instead use bandwidth, but this is what we noticed doesn't translate to speed improvements when caching.
Computationally expensive tasks are mostly limited to doc generation, and mostly for large projects. This we don't want to cache either way though. We do know that build tool compilation can be expensive, but that doesn't need a configurable cache even.
@humitos in your testing weren't prebuilt binaries marginally faster than compiling binaries?
This does seem like a place to apply some effort, but we should look at what tools users are installing and really see if anyone is compiling uncached build tools before we assume users need more complexity here.
Though I was excited initially about the prospect of caching, unfortunately the performance really didn't back up the feature at all. We learned this lesson already, so not sure this is going to be valuable unless we're trying something new.
I would say to make progress here, we should find projects that are using uncached build tools and compare timing of full compilation vs downloading a compiled tarball. If users are really struggling with caching, they do have ways out, like using build.tools to download pre-built binaries.
humitos commentedon Jun 2, 2022
@agjohnson
When I originally implemented the feature, we were interested in caching a large git repository, not binaries. So, for that particular case, we didn't notice a big difference and we decided to remove the complexity because of that.
Now, with the addition of
build.tools, there is a big noticeable difference between having a pre-compiled Python version in the cache, downloading and untar it compared with downloading its source code and compiling it on-the-fly. I'm mentioning Python in particular here because it does not happens with all thebuild.toolswe support. For example, when installingnodejs,asdfjust downloads a pre-compiled version of it and it's mainly the same time required.I opened this issue to discuss this between us but mainly to start collecting user feedback as 👍🏼's and also use cases where they can explain to us how they would use it if we implement this.
agjohnson commentedon Jun 2, 2022
Yeah users might have some good use cases here that we're not aware of. Collecting feedback is wise.
But it's worth noting the limitations that we found and that our caching mechanism only might help computationally heavy tasks that are not doc generation tasks. The questions above to lean into implementation, but I feel the question we need to answer first is much higher level.
On paper, caching looks great. But in application, and with it's limitations, we learned it's not super effective for most workloads.
We were also caching python packages installed, correct? Both operations are network operations primarily, which is why we didn't notice an increase in speed with caching -- S3 wasn't faster than ingress traffic.
So yeah, this is an area where we could do something. But user-configurable cache isn't required for this, and is a rather maximal option. The minimum viable option here is to expand what build.tools we prebuild. Again, we should see what users are actually doing though. I suspect we're over-optimizing here for an unknown user profile.
humitos commentedon Jun 3, 2022
@agjohnson
I think so, yes. We were caching the whole folder for the particular version. So, that also included the virtualenv already created with all the packages installed.
At this point, I don't think we need user-configurable cache. Let's see what feedback the community can share with us and if that feedback matches with an implementation and the benefits we can provide there.