We've seen a lot of reported issues that, one way or another, reflect a lack of flexibility in the way we apply spawn strategies.

We believe it's time to rethink how spawn strategies are implemented and what the right API to choose among them should look like. In particular, it would be nice to:

• Reimplement the disk/remote cache lookup as a strategy (instead of as part of the spawn runner)
• Generalize dynamic execution to an arbitrary number of strategies (or combinations of strategies) to be tried in parallel
• Generalize local fallback to an arbitrary number of strategies (or combinations of strategies) to be tried in series
• Have a way to distinguish "permanent" from "transient" failures (permanent are not worth retrying with a different strategy, transient might be)
• Make it possible to toggle strategies along dimensions other than mnemonic (e.g. based on the execution platform, selected toolchain, or estimated resource consumption)

This would make it possible to provide solutions to the following issues:

• remote: remove local fallback for remote execution #7202
• Dynamic spawn scheduler prevents remote cache from being populated #7328
• Bazel remote cache is not a clear win #7664
• Unify execution strategies and platforms #11432
• Make --remote_local_fallback honor --spawn_strategy #15519
• --remote_local_fallback lead to undeclared inclusion #16132
• Combined disk & remote cache unconditionally uploads all output blobs on cache miss #18140
• Expose workspace provenance for strategy selection #18244
• Remote caching should be a strategy #18245
• Remote exec is not resilient to remote build farm worker deaths #18319
• No local fallback after cache timeout #20123
• Remote falls back to local, not sandboxed #21725

In particular, #11432 references https://docs.google.com/document/d/1U9HzdDmtRnm244CaRM6JV-q2408mbNODAMewcGjnnbM/edit#heading=h.5mcn15i0e1ch which contains many interesting ideas we should revisit in light of this set of requirements.