Description of the bug:
Setup:
- Ten targets with one output file each (“prerequisite targets”).
- One target that takes each of the ten prerequisite targets as input (“final target”).
- Remote cache and remote execution.
Reproduction:
- Build the prerequisite targets in output base A.
- Delete all entries from the remote cache (AC and CAS).
- Build the final target in output base B.
Expected behavior:
Bazel rebuilds everything successfully. Or Bazel exits with 39 because expected cache entries cannot be found remotely during evaluation.
Actual behavior:
Logs from step 3 indicate exit code 34:
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 671b6d5b-516d-4685-8489-5b6e2e51c394
INFO: Analyzed target //:all_files (5 packages loaded, 27 targets configured).
[1 / 1] checking cached actions
[1 / 12] 8 actions, 0 running
[Sched] Executing genrule //:target_2
[Sched] Executing genrule //:target_3
[Sched] Executing genrule //:target_6
[Sched] Executing genrule //:target_0
[Sched] Executing genrule //:target_7
[Sched] Executing genrule //:target_5
[Sched] Executing genrule //:target_9
[Sched] Executing genrule //:target_1
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_6 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_9 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_1 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_5 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_3 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_0 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
ERROR: /pay/src/cache-eviction-hang/BUILD:1:9: Executing genrule //:target_2 failed: (Exit 34): FAILED_PRECONDITION: Failed to obtain action: Object not found
Target //:all_files failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.747s, Critical Path: 0.24s
INFO: 11 processes: 11 internal.
ERROR: Build did NOT complete successfully
Bazel exited with 34
Investigation:
We have not tried going deep for this bug, this was found by accident while trying to reproduce #21777.
Which category does this issue belong to?
Remote Execution
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Enable remote executor locally on port 8980. In our reproduction, we used https://github.com/buildbarn/bb-deployments/.
Check out https://github.com/hoj-stripe/cache-eviction-hang, run ./repro_34_exit.sh.
Which operating system are you running Bazel on?
Ubuntu 20.04.6 LTS
What is the output of bazel info release?
release 7.1.0
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse HEAD ?
No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
cc @clintharrison @sushain97 @coeuvre
Description of the bug:
Setup:
Reproduction:
Expected behavior:
Bazel rebuilds everything successfully. Or Bazel exits with 39 because expected cache entries cannot be found remotely during evaluation.
Actual behavior:
Logs from step 3 indicate exit code 34:
Investigation:
We have not tried going deep for this bug, this was found by accident while trying to reproduce #21777.
Which category does this issue belong to?
Remote Execution
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Enable remote executor locally on port 8980. In our reproduction, we used https://github.com/buildbarn/bb-deployments/.
Check out https://github.com/hoj-stripe/cache-eviction-hang, run ./repro_34_exit.sh.
Which operating system are you running Bazel on?
Ubuntu 20.04.6 LTS
What is the output of
bazel info release?release 7.1.0
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse HEAD?No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
cc @clintharrison @sushain97 @coeuvre