Skip to content

"failed to delete output files before executing action" with tree artifacts and --remote_download_minimal #12678

@scele

Description

@scele

Description of the problem / feature request:

Getting this error when doing an incremental build with --remote_download_minimal.

Mnemonic target [for host] failed: failed to delete output files before executing action

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ cat BUILD 
load("rule.bzl", "foo")

foo(
    name = "foo",
    srcs = ["temp.txt"],
)
$ cat rule.bzl 
def _foo_impl(ctx):
    d = ctx.actions.declare_directory(ctx.attr.name)
    ctx.actions.run_shell(
        command = "mkdir {d}/subdir && echo foo > {d}/subdir/foo.txt".format(d = d.path),
        inputs = ctx.files.srcs,
        outputs = [d],
    )
    return [DefaultInfo(files = depset([d]), runfiles = ctx.runfiles([d]))]

foo = rule(
    implementation = _foo_impl,
    attrs = {"srcs": attr.label_list(allow_files = True)},
)
$ cat temp.txt 
foo
$ cat repro.sh 
#!/bin/bash
BAZEL=${1:-bazel}
bazel version
rm -rf $PWD/disk-cache
$BAZEL clean
$BAZEL build :foo
echo foo >> temp.txt
$BAZEL build :foo --disk_cache=$PWD/disk-cache --remote_download_minimal

$ USE_BAZEL_VERSION=3.7.1 ./repro.sh
Bazelisk version: development
Starting local Bazel server and connecting to it... 
Build label: 3.7.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Nov 24 17:38:30 2020 (1606239510)
Build timestamp: 1606239510
Build timestamp as int: 1606239510
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed target //:foo (4 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
  bazel-bin/foo
INFO: Elapsed time: 0.819s, Critical Path: 0.06s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Invocation ID: 1829a11b-6583-4511-9a27-13eb559eceb6
INFO: Build options --build_runfile_links, --experimental_inmemory_dotd_files, and --experimental_inmemory_jdeps_files have changed, discarding analysis cache.
INFO: Analyzed target //:foo (0 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
  bazel-bin/foo
INFO: Elapsed time: 0.243s, Critical Path: 0.10s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

$ USE_BAZEL_VERSION=4.0.0rc4 ./repro.sh 
Bazelisk version: development
Starting local Bazel server and connecting to it...
Build label: 4.0.0rc4
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Dec 7 18:32:14 2020 (1607365934)
Build timestamp: 1607365934
Build timestamp as int: 1607365934
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed target //:foo (4 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
  bazel-bin/foo
INFO: Elapsed time: 0.764s, Critical Path: 0.07s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Invocation ID: 661bf76e-7634-4dec-86f4-814701634b05
INFO: Build options --build_runfile_links, --experimental_inmemory_dotd_files, and --experimental_inmemory_jdeps_files have changed, discarding analysis cache.
INFO: Analyzed target //:foo (0 packages loaded, 7 targets configured).
INFO: Found 1 target...
ERROR: /home/lpeltonen/tmp/bazel-repro2/BUILD:3:4: Action foo failed: failed to delete output files before executing action
Target //:foo failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.133s, Critical Path: 0.00s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

What operating system are you running Bazel on?

Ubuntu 18.04.3 LTS

What's the output of bazel info release?

release 4.0.0rc4

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

Looks like the exception is thrown from this branch in src/main/java/com/google/devtools/build/lib/actions/AbstractAction.java:deleteOutput

      Root outputRoot = root.getRoot();
      if (!outputRoot.contains(path)) {
        throw new IOException(e);
      }

Reverting 4009b17 seems to fix this.

Metadata

Metadata

Assignees

Labels

P1I'll work on this now. (Assignee required)team-Remote-ExecIssues and PRs for the Execution (Remote) teamtype: bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions