Skip to content

Config: add cache key to save state between builds #9183

Open

Description

@humitos

I'm not sure if this is a too-requested feature, but I wanted to open this issue to start the conversation at least and collect 👍🏼's via GitHub reactions in case people want this feature to be implemented.

Currently, we are allowing users to use build.jobs to run custom commands in the middle of the build process. There are some projects that are already running some CPU/bandwidth intense actions there and they may have some benefit by caching some directories.

Initial implementation

I'm thinking the basic implementation for this would be something like:

build:
  pre_build:
    - bash cpu_intense_script.sh >  /path/to/my/cached/directory/file.out
  cache:
    - "/path/to/my/cached/directory"
    - "/home/docs/.pip/cache"
    - ...

The implementation can be done with the code we used to have: CachedEnvironmentMixin

  • compress into .tar.gz and upload to S3 the paths defined by the user after the build finishes
  • at the beginning of the build will download from S3 cached paths if there are any

Questions

  • what are the most common use cases for cache?
  • what kind of integrations do we want to implement for this feature?
  • how do we know that a cache path has to be purged?
  • should we cache and retrieve each path independently?
  • should PR and normal builds allow to use the cache? should this be configurable at project level?
  • what's the minimal time where using a cached directory makes it faster than running the command itself?

Activity

  1. agjohnson commented on Jun 1, 2022

    @agjohnson
    Contributor

    You raise are good questions. However, I'm still not sold on caching being important for our platform, and it's mostly this:

    what's the minimal time where using a cached directory makes it faster than running the command itself?

    This was the big blocker for the initial version of this. The speed increase was negligible for builds when we tested this. Very likely, the user is only going to notice a speed increase when doing something computationally expensive, and even then there isn't a guarantee it will be noticeably faster.

    We do differ from CI services in this regard. The vast majority of projects do not need or benefit from the caching we can provide, as most projects don't have computationally expensive operations. They do instead use bandwidth, but this is what we noticed doesn't translate to speed improvements when caching.

    Computationally expensive tasks are mostly limited to doc generation, and mostly for large projects. This we don't want to cache either way though. We do know that build tool compilation can be expensive, but that doesn't need a configurable cache even.

    @humitos in your testing weren't prebuilt binaries marginally faster than compiling binaries?

    This does seem like a place to apply some effort, but we should look at what tools users are installing and really see if anyone is compiling uncached build tools before we assume users need more complexity here.

    Though I was excited initially about the prospect of caching, unfortunately the performance really didn't back up the feature at all. We learned this lesson already, so not sure this is going to be valuable unless we're trying something new.

    I would say to make progress here, we should find projects that are using uncached build tools and compare timing of full compilation vs downloading a compiled tarball. If users are really struggling with caching, they do have ways out, like using build.tools to download pre-built binaries.

  2. humitos commented on Jun 2, 2022

    @humitos
    MemberAuthor

    @agjohnson

    @humitos in your testing weren't prebuilt binaries marginally faster than compiling binaries?

    When I originally implemented the feature, we were interested in caching a large git repository, not binaries. So, for that particular case, we didn't notice a big difference and we decided to remove the complexity because of that.

    Now, with the addition of build.tools, there is a big noticeable difference between having a pre-compiled Python version in the cache, downloading and untar it compared with downloading its source code and compiling it on-the-fly. I'm mentioning Python in particular here because it does not happens with all the build.tools we support. For example, when installing nodejs, asdf just downloads a pre-compiled version of it and it's mainly the same time required.

    I opened this issue to discuss this between us but mainly to start collecting user feedback as 👍🏼's and also use cases where they can explain to us how they would use it if we implement this.

  3. agjohnson commented on Jun 2, 2022

    @agjohnson
    Contributor

    Yeah users might have some good use cases here that we're not aware of. Collecting feedback is wise.

    But it's worth noting the limitations that we found and that our caching mechanism only might help computationally heavy tasks that are not doc generation tasks. The questions above to lean into implementation, but I feel the question we need to answer first is much higher level.

    On paper, caching looks great. But in application, and with it's limitations, we learned it's not super effective for most workloads.

    When I originally implemented the feature, we were interested in caching a large git repository

    We were also caching python packages installed, correct? Both operations are network operations primarily, which is why we didn't notice an increase in speed with caching -- S3 wasn't faster than ingress traffic.

    Now, with the addition of build.tools, there is a big noticeable difference between having a pre-compiled Python version in the cache,

    So yeah, this is an area where we could do something. But user-configurable cache isn't required for this, and is a rather maximal option. The minimum viable option here is to expand what build.tools we prebuild. Again, we should see what users are actually doing though. I suspect we're over-optimizing here for an unknown user profile.

  4. humitos commented on Jun 3, 2022

    @humitos
    MemberAuthor

    @agjohnson

    We were also caching python packages installed, correct?

    I think so, yes. We were caching the whole folder for the particular version. So, that also included the virtualenv already created with all the packages installed.

    At this point, I don't think we need user-configurable cache. Let's see what feedback the community can share with us and if that feedback matches with an implementation and the benefits we can provide there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions