We have a large build (200k+ actions) and have noticed that clean BWOB builds with low cache hit rates (<70%) actually perform worse than no-cache builds, by a significant magnitude (2x). When we change the remote_download_outputs to all, then the no-cache build is the same performance wise as the cache-enabled one.

We are populating our remote cache, which is a simple nginx server backed by AWS S3, from a CI build on every commit to our default branch. These populate builds are incremental and are using remote_download_minimal as well. Should also note, it seems we see a higher cache hit % for read-only builds when the populate builds are fully clean.

Is this expected behavior and what are the recommendations around using remote_download_minimal vs remote_download_all? I've looked through the source code and forums and have not found any answers. Thanks.

Bazel Version: 7.0.0