If grpc remote cache is inaccessible, fall back to building without the cache #11989
Description
Description of the problem / feature request:
It seems bazel is falling back to building without remote cache only when using a http remote cache. If a grpc cache is specified, the build fails with the following error:
INFO: Invocation ID: 6c2d307e-e24d-46b3-bce6-6fb116de093e
ERROR: java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Unable to resolve host fakeurl.net
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.get(RemoteServerCapabilities.java:82)
at com.google.devtools.build.lib.remote.RemoteModule.verifyServerCapabilities(RemoteModule.java:148)
at com.google.devtools.build.lib.remote.RemoteModule.beforeCommand(RemoteModule.java:349)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:390)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:236)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:654)
at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$2(GrpcServerImpl.java:719)
at io.grpc.Context$1.run(Context.java:595)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Unable to resolve host fakeurl.net
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
at build.bazel.remote.execution.v2.CapabilitiesGrpc$CapabilitiesBlockingStub.getCapabilities(CapabilitiesGrpc.java:221)
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.lambda$get$0(RemoteServerCapabilities.java:77)
at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:237)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:115)
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.get(RemoteServerCapabilities.java:77)
... 10 more
Caused by: java.lang.RuntimeException: java.net.UnknownHostException: fakeurl.net
at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:414)
at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:252)
at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:214)
... 3 more
Caused by: java.net.UnknownHostException: fakeurl.net
at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)
at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:634)
at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:382)
... 5 more
Feature requests: what underlying problem are you trying to solve with this feature?
Allow developers to build offline or when our grpc remote cache service goes down. Maybe I'm missing a special flag to allow the build to continue or have a custom timeout if the grpc service is unreachable?
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Pass --remote_cache=grpc://any-grpc-service:8980 to any build and if the service is not reachable, the build will fail.
What operating system are you running Bazel on?
macOS 10.15
What's the output of bazel info release?
release 3.4.1
Have you found anything relevant by searching the web?
Similar issue here, but it seems like it was related to the HTTP cache instead: #2964