Hi everyone! Is it possible, or has it been considered, to allow multiple levels of cache? For instance, using local disk as an L1 cache, and s3/azure/gcs as L2.

I envision it working something like the following:

Check disk cache according to current rules
if (found)
    return item

check s3/azure/gcs according to current rules
if (found)
    return item

compile item
populate disk cache
populate s3/azure/gcs cache

return item

This lets us use nvme as a hot cache, while still being able to hit the network cache if needed. This (theoretically) avoids a lot of network traffic and associated costs, and reduces build times on average. It also gives us a transparent way to seed new build hosts when they spin up, which makes automatic scaling of the build fleet more effective. Since each host has it's own hot cache, the load on s3/azure/gcs should scale better than linearly with the number of build hosts (on average; linear being the worst case).

Generically, it could be useful to mix/match all the cache types. For example, local disk as L1, on-prem memcached/redis as L2, and s3/azure/gcs as L3. Such a setup could be used to share build cache with all the devs in an office without hitting the internet as much, while still being able to share that build cache with other offices, etc.

Thanks!