File output inside directory output fails to download from remote cache #15328
Description
Description of the bug:
Bazel allows an action to generate an output file and an output directory, where the file is a child of the directory. This is useful for actions where the output structure is only partially understood/deterministic.
When using such an output structure with remote execution, Bazel fails to download outputs from the remote CAS. It appears that the downloads list contains duplicate entries for nested files, which breaks the atomic file renaming in RemoteExecutionService.moveOutputsToFinalLocation().
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
# BUILD
load("//:rules.bzl", "nested_outputs")
nested_outputs(name = "nested_outputs")# rules.bzl
_COMMAND = """
echo 'Hello, world!' > $1/hello.txt
echo 'additional output' > $1/extra.txt
"""
def _nested_outputs(ctx):
out_dir = ctx.actions.declare_directory(ctx.attr.name)
out_file = ctx.actions.declare_file(ctx.attr.name + "/hello.txt")
ctx.actions.run_shell(
inputs = [],
outputs = [out_dir, out_file],
command = _COMMAND,
arguments = [out_dir.path],
)
return DefaultInfo(files = depset([out_dir, out_file]))
nested_outputs = rule(implementation = _nested_outputs)$ bazel-5.1.0 build --config=remote --verbose_failures //:nested_outputs
INFO: Invocation ID: 379fd243-d030-4a7f-8d3e-0c4173e42877
INFO: Analyzed target //:nested_outputs (1 packages loaded, 1 target configured).
INFO: Found 1 target...
ERROR: /Users/john/src/rexec_file_and_dir/BUILD.bazel:2:15: Action nested_outputs failed: (Exit 34): /private/var/tmp/_bazel_john/ad23c0fba73bd9655514a30e2d1746b4/execroot/__main__/bazel-out/darwin-fastbuild/bin/nested_outputs/hello.txt.tmp (No such file or directory)
java.io.FileNotFoundException: /private/var/tmp/_bazel_john/ad23c0fba73bd9655514a30e2d1746b4/execroot/__main__/bazel-out/darwin-fastbuild/bin/nested_outputs/hello.txt.tmp (No such file or directory)
at com.google.devtools.build.lib.unix.NativePosixFiles.lstat(Native Method)
at com.google.devtools.build.lib.unix.UnixFileSystem.statInternal(UnixFileSystem.java:185)
at com.google.devtools.build.lib.unix.UnixFileSystem.stat(UnixFileSystem.java:174)
at com.google.devtools.build.lib.vfs.Path.stat(Path.java:319)
at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:456)
at com.google.devtools.build.lib.remote.RemoteExecutionService.moveOutputsToFinalLocation(RemoteExecutionService.java:741)
at com.google.devtools.build.lib.remote.RemoteExecutionService.downloadOutputs(RemoteExecutionService.java:1096)
[...]
Which operating system are you running Bazel on?
I can reproduce the error on macOS 12.2.1 and Ubuntu 20.04
What is the output of bazel info release?
release 5.1.0
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?
No response
Have you found anything relevant by searching the web?
The discussion in bazelbuild/remote-apis#63 documents that output files are allowed to be the child of an output directory.
Any other information, logs, or outputs that you want to share?
No response