In the current implementation of remote execution, outputs can only be downloaded when an action executes. We should
separate downloads from action execution so that we can provide more download options to users. For example:

1. Provide a download API so that other code can download outputs of previously executed actions when necessary. Use cases:
   • Currently, we download the outputs after executed an action remotely and wait for the downloads to complete before starting execution of other actions. We can change to download the outputs of current action and start executing next action in the same time. This will improve the performance of full-downloads mode a lot.
   • Some files downloaded by the minimal/toplevel mode are symlink pointing to files we didn't download. There are some issues root-caused by this behavior. We can provide options to download the actual files when reading the symlink. This will fix most of bugs related to symlink produced by remote execution.
   • Dynamic execution can download outputs of previously remotely executed action in the local branch.
2. Add a post-build step for downloads so that we can download files which are only known after the build complete. Use cases:
   • We can combine this with aspects and BEP to download files required by IDE after the build.
   • Tests require some of the outputs to run. There are some issues related to running tests with top-level mode which should be fixed by this change.
3. Add a command line flag to only download arbitrary output files. Use cases:
   • Allow to download anything of the output tree without going through the build steps

We need a way to specify which files to download after the build (option 2) or with the download flag (option 3). For example:

• Reuse the flags --remote_download_toplevel to only download output files of top level targets and --remote_download_minimal to only download necessary files.
• Add a new flag which accept a list of targets whose output files should be downloaded.
• I am not sure how to specify download files at file level. Is that useful (and can be implemented) to support regex search over the output tree? (like rg over a directory)

Options for downloading the content of a symlink file:

• Provide a flag to control whether the download should follow the symlinks, this applies to all 3 download options above
• When the code is reading the content of a symlink file (e.g. in the RemoteActionFS), we can follow the symlink using option 1
• When other tools are reading the content of a symlink (e.g. IDE), the tool can use option 3 to download the file.
• When users want to read the content of a symlink, they can use option 3 to download the file manually.

Related Issues

• Build artifacts are missing with "--remote_download_toplevel" cache option #13625
• remote_download_minimal: downloaded inputs shouldn't be deleted after the build #12855
• BwtB deletes needed runfiles from output base #12635
• No coverage files when running a build with --remote_download_minimal or --remote_download_toplevel #12592
• Bazel sometimes execs file downloaded from remote cache before executable bits have been set #12137
• Build event file creation fails with symlinks when using --remote_download_minimal #11942
• UX: make --remote_download_output=minimal download the necessary files when invoking bazel run #11920
• remote_download_toplevel doesn't download binary after testing  #11834
• tools/test/test-setup.sh fails to collect TEST_UNDECLARED_OUTPUTS_DIR in a minimal remote execution environment #11558
• Allow remote-caching for Symlink and SolibSymlink actions #11119
• --remote_download_minimal does not work with --build_event_publish_all_actions #10971
• Unexpected IO Error (Not a directory) on macOS #10618
• Async download of build artifacts #10525
• Provide more helpful information if a cache key is missing with builds-without-the-bits #10025
• remote_download_minimal: Error running tests with a directory dependency #9386
• Revisit --enable_runfiles #9248
• --experimental_remote_download_outputs breaks symlink outputs #8581