Description of the problem / feature request:
Bazel remote cache grpc backend seems to optimize away uploading the zero-sized CAS blob (sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). This means that all remote cache implementations must add special logic to handle that digest, for example: buchgr/bazel-remote#233
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
I'm running buildbarn/bb-storage@fb0be44 as a remote cache at grpc://localhost:9092.
$ cat BUILD
cc_binary(
name = "test",
srcs = ["test.cc"],
)
$ bazel clean && bazel build //:test --remote_cache=grpc://localhost:9092 --experimental_remote_grpc_log=$PWD/grpc.log
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Invocation ID: 8e4eba6d-d513-4afb-8260-63a2b182eb21
INFO: Analyzed target //:test (13 packages loaded, 47 targets configured).
INFO: Found 1 target...
Target //:test up-to-date:
bazel-bin/test
INFO: Elapsed time: 0.524s, Critical Path: 0.25s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed successfully, 6 total actions
Here is the decoded remote grpc log:
Click to expand
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "capabilities"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.Capabilities/GetCapabilities"
details {
get_capabilities {
request {
instance_name: "local"
}
response {
cache_capabilities {
digest_function: MD5
digest_function: SHA1
digest_function: SHA256
digest_function: UNKNOWN_ENUM_VALUE_DigestFunction_5
digest_function: UNKNOWN_ENUM_VALUE_DigestFunction_6
action_cache_update_capabilities {
update_enabled: true
}
symlink_absolute_path_strategy: ALLOWED
}
low_api_version {
major: 2
}
high_api_version {
major: 2
}
}
}
}
start_time {
seconds: 1585913715
nanos: 962000000
}
end_time {
seconds: 1585913715
nanos: 966000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
code: 5
message: "https://our-build-cache/ac/b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
}
method_name: "build.bazel.remote.execution.v2.ActionCache/GetActionResult"
details {
get_action_result {
request {
instance_name: "local"
action_digest {
hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
size_bytes: 142
}
}
}
}
start_time {
seconds: 1585913716
nanos: 222000000
}
end_time {
seconds: 1585913717
nanos: 53000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ContentAddressableStorage/FindMissingBlobs"
details {
find_missing_blobs {
request {
instance_name: "local"
blob_digests {
hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
size_bytes: 1224
}
blob_digests {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
blob_digests {
hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
size_bytes: 88
}
blob_digests {
hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
size_bytes: 142
}
blob_digests {
hash: "d70d1a8abd6791629f51490151fd7d2ee34b89e8147ba081ff2a4e1fa79a21ed"
size_bytes: 989
}
}
response {
missing_blob_digests {
hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
size_bytes: 142
}
missing_blob_digests {
hash: "d70d1a8abd6791629f51490151fd7d2ee34b89e8147ba081ff2a4e1fa79a21ed"
size_bytes: 989
}
missing_blob_digests {
hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
size_bytes: 88
}
missing_blob_digests {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
missing_blob_digests {
hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
size_bytes: 1224
}
}
}
}
start_time {
seconds: 1585913717
nanos: 143000000
}
end_time {
seconds: 1585913719
nanos: 726000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/b41d91ce-f43e-4d20-bf4d-740faae6f32f/blobs/f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259/1224"
num_writes: 1
bytes_sent: 1224
response {
committed_size: 1224
}
}
}
start_time {
seconds: 1585913719
nanos: 729000000
}
end_time {
seconds: 1585913720
nanos: 463000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/40f630ae-528e-41a5-bb77-d924e7dc01b2/blobs/decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9/88"
num_writes: 1
bytes_sent: 88
response {
committed_size: 88
}
}
}
start_time {
seconds: 1585913719
nanos: 728000000
}
end_time {
seconds: 1585913720
nanos: 479000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/2f6c53ee-4bc1-4000-a476-ffc67bd074a0/blobs/d70d1a8abd6791629f51490151fd7d2ee34b89e8147ba081ff2a4e1fa79a21ed/989"
num_writes: 1
bytes_sent: 989
response {
committed_size: 989
}
}
}
start_time {
seconds: 1585913719
nanos: 727000000
}
end_time {
seconds: 1585913720
nanos: 485000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/1ac78d17-a236-42b4-9215-522abf656ad5/blobs/b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1/142"
num_writes: 1
bytes_sent: 142
response {
committed_size: 142
}
}
}
start_time {
seconds: 1585913719
nanos: 727000000
}
end_time {
seconds: 1585913720
nanos: 496000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ActionCache/UpdateActionResult"
details {
update_action_result {
request {
instance_name: "local"
action_digest {
hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
size_bytes: 142
}
action_result {
output_files {
path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.o"
digest {
hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
size_bytes: 1224
}
}
output_files {
path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.d"
digest {
hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
size_bytes: 88
}
}
stdout_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
stderr_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
}
}
response {
output_files {
path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.o"
digest {
hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
size_bytes: 1224
}
}
output_files {
path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.d"
digest {
hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
size_bytes: 88
}
}
stdout_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
stderr_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
}
}
}
start_time {
seconds: 1585913720
nanos: 497000000
}
end_time {
seconds: 1585913721
nanos: 419000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
code: 5
message: "https://our-build-cache/ac/3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
}
method_name: "build.bazel.remote.execution.v2.ActionCache/GetActionResult"
details {
get_action_result {
request {
instance_name: "local"
action_digest {
hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
size_bytes: 142
}
}
}
}
start_time {
seconds: 1585913721
nanos: 427000000
}
end_time {
seconds: 1585913722
nanos: 309000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ContentAddressableStorage/FindMissingBlobs"
details {
find_missing_blobs {
request {
instance_name: "local"
blob_digests {
hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
size_bytes: 7856
}
blob_digests {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
blob_digests {
hash: "5d42134e4bf9156e629091745de7d258decaec422022b1b1399df3c81c4291bd"
size_bytes: 358
}
blob_digests {
hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
size_bytes: 142
}
}
response {
missing_blob_digests {
hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
size_bytes: 142
}
missing_blob_digests {
hash: "5d42134e4bf9156e629091745de7d258decaec422022b1b1399df3c81c4291bd"
size_bytes: 358
}
missing_blob_digests {
hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
size_bytes: 7856
}
missing_blob_digests {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
}
}
}
start_time {
seconds: 1585913722
nanos: 437000000
}
end_time {
seconds: 1585913724
nanos: 692000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/403d946f-38b2-45f0-8afa-f9ae704160ac/blobs/cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c/7856"
num_writes: 1
bytes_sent: 7856
response {
committed_size: 7856
}
}
}
start_time {
seconds: 1585913724
nanos: 695000000
}
end_time {
seconds: 1585913725
nanos: 430000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/c39d05f9-349e-4318-b6eb-0c97a9f34cea/blobs/3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0/142"
num_writes: 1
bytes_sent: 142
response {
committed_size: 142
}
}
}
start_time {
seconds: 1585913724
nanos: 693000000
}
end_time {
seconds: 1585913725
nanos: 430000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
write {
resource_names: "local/uploads/59868396-c124-49ed-ba9e-e50a2b1f2de1/blobs/5d42134e4bf9156e629091745de7d258decaec422022b1b1399df3c81c4291bd/358"
num_writes: 1
bytes_sent: 358
response {
committed_size: 358
}
}
}
start_time {
seconds: 1585913724
nanos: 694000000
}
end_time {
seconds: 1585913725
nanos: 443000000
}
---------------------------------------------------------
metadata {
tool_details {
tool_name: "bazel"
tool_version: "2.2.0"
}
action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ActionCache/UpdateActionResult"
details {
update_action_result {
request {
instance_name: "local"
action_digest {
hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
size_bytes: 142
}
action_result {
output_files {
path: "bazel-out/k8-fastbuild/bin/test"
digest {
hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
size_bytes: 7856
}
is_executable: true
}
stdout_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
stderr_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
}
}
response {
output_files {
path: "bazel-out/k8-fastbuild/bin/test"
digest {
hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
size_bytes: 7856
}
is_executable: true
}
stdout_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
stderr_digest {
hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
}
}
}
start_time {
seconds: 1585913725
nanos: 444000000
}
end_time {
seconds: 1585913725
nanos: 658000000
}
---------------------------------------------------------
The log shows that even though the remote cache reports e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 as missing, bazel does not attempt to upload it. Yet, it is referencing it from the ActionResult messages that it uploads.
In normal use this would not be noticed, since on a remote cache hit, I believe bazel grpc cache backend will notice that the referenced CAS blob has zero size and will not attempt to download it. However, if I try to consume the "corrupt" AC entry using the http protocol, bazel will try to download the e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 CAS blob, gets back 404, and will fall back to local execution.
What operating system are you running Bazel on?
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic
What's the output of bazel info release?
release 2.2.0
Have you found anything relevant by searching the web?
Seems that some remote cache implementations have added special handling for this, see e.g. buchgr/bazel-remote#233
Description of the problem / feature request:
Bazel remote cache grpc backend seems to optimize away uploading the zero-sized CAS blob (sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). This means that all remote cache implementations must add special logic to handle that digest, for example: buchgr/bazel-remote#233
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
I'm running buildbarn/bb-storage@fb0be44 as a remote cache at grpc://localhost:9092.
Here is the decoded remote grpc log:
Click to expand
The log shows that even though the remote cache reports e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 as missing, bazel does not attempt to upload it. Yet, it is referencing it from the ActionResult messages that it uploads.
In normal use this would not be noticed, since on a remote cache hit, I believe bazel grpc cache backend will notice that the referenced CAS blob has zero size and will not attempt to download it. However, if I try to consume the "corrupt" AC entry using the http protocol, bazel will try to download the e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 CAS blob, gets back 404, and will fall back to local execution.
What operating system are you running Bazel on?
What's the output of
bazel info release?release 2.2.0
Have you found anything relevant by searching the web?
Seems that some remote cache implementations have added special handling for this, see e.g. buchgr/bazel-remote#233