Hey @fweikert
rules_nodejs has had almost all our RBE builds red for the last few days. On PRs I'm pressing the Retry button multiple times to get them green, and our default branch has been red for a day.
https://buildkite.com/bazel/rules-nodejs-nodejs/builds/9591#bdced136-a515-4099-a52b-5984d9575a61 is an example
failure:
(03:24:59) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/bazel_tools/tools/jdk/BUILD:346:14: Action external/bazel_tools/tools/jdk/platformclasspath.jar [for host] failed: (Exit 34): com.google.devtools.build.lib.remote.BulkTransferException
at com.google.devtools.build.lib.remote.RemoteCache.waitForBulkTransfer(RemoteCache.java:227)
[...]
Suppressed: java.io.IOException: Output download failed: Expected digest '6f5ff115e713ede319bc832024f78d018ae0c5da7a810c8af68b2b5368d00a0d/85582084' does not match received digest 'e294ea66b89ce1dee25c6c6f354ddb8ebffd303f6ca9255c5fbe7d0d6f31d374/85782084'.
at com.google.devtools.build.lib.remote.util.Utils.verifyBlobContents(Utils.java:201)
at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onCompleted(GrpcCacheClient.java:372)
It's always this same entry causing the problem. Seems like something has gotten into the remote cache that shouldn't be there. Is it easy to just blow away the storage for that cache instance (no idea if it's shared-tenant with other rulesets or other reasons to be careful there)
Thanks!!
Hey @fweikert
rules_nodejs has had almost all our RBE builds red for the last few days. On PRs I'm pressing the Retry button multiple times to get them green, and our default branch has been red for a day.
https://buildkite.com/bazel/rules-nodejs-nodejs/builds/9591#bdced136-a515-4099-a52b-5984d9575a61 is an example
failure:
It's always this same entry causing the problem. Seems like something has gotten into the remote cache that shouldn't be there. Is it easy to just blow away the storage for that cache instance (no idea if it's shared-tenant with other rulesets or other reasons to be careful there)
Thanks!!