Swarming caches are ignored in builds created by sub-builds in buildbucket. This became a larger problem after the migration to real builds. After some research we found that we can have deterministic checkouts with less than 1m 30s by using a combination of GCS and CAS.
This also has the additional benefit of allowing the use of secure caches across buildbucket buckets.