Skip to content

4.1.0rc1 regression: java.lang.RuntimeException: Unrecoverable error while evaluating node #13239

@brentleyjones

Description

@brentleyjones

Description of the problem / feature request:

bazel server crashes using Buildbarn remote-exec and getting a build failure very early in the build (though not always):

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=ConfiguredTargetKey{label=@com_github_protocolbuffers_protobuf//:protobuf, config=BuildConfigurationValue.Key[db75135daf4c6c2bd5a60ccb4b97f58d6679376413cb33fc5e2220ea72e2f366]}, actionIndex=16}' (requested by nodes 'ArtifactNestedSetKey{rawChildren=[File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/any.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/any.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/api.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/importer.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/parser.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/descriptor.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/descriptor.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/descriptor_database.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/duration.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/dynamic_message.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/empty.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/extension_set_heavy.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/field_mask.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/generated_message_reflection.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/generated_message_table_driven.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/gzip_stream.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/printer.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/tokenizer.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/map_field.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/message.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/reflection_ops.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/service.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/source_context.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/struct.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/substitute.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/text_format.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/timestamp.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/type.pb.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/unknown_field_set.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/delimited_message_util.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/field_comparator.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/field_mask_util.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/datapiece.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/default_value_objectwriter.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/error_listener.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/field_mask_utility.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/json_escaping.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/json_objectwriter.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/json_stream_parser.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/object_writer.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/proto_writer.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/protostream_objectsource.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/protostream_objectwriter.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/type_info.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/type_info_test_helper.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/utility.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/json_util.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/message_differencer.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/time_util.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/type_resolver_util.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/wire_format.o, File:[[<execution_root>]bazel-out/host/bin]external/com_github_protocolbuffers_protobuf/_objs/protobuf/wrappers.pb.o]}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:563)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
        at io.reactivex.rxjava3.internal.util.ExceptionHelper.wrapOrThrow(ExceptionHelper.java:46)
        at io.reactivex.rxjava3.internal.observers.BlockingMultiObserver.blockingGet(BlockingMultiObserver.java:94)
        at io.reactivex.rxjava3.core.Single.blockingGet(Single.java:3645)
        at com.google.devtools.build.lib.remote.ReferenceCountedChannel.newCall(ReferenceCountedChannel.java:100)
        at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.call(ByteStreamUploader.java:509)
        at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.callAndQueryOnFailure(ByteStreamUploader.java:434)
        at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.lambda$start$0(ByteStreamUploader.java:400)
        at com.google.devtools.build.lib.remote.Retrier.executeAsync(Retrier.java:277)
        at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.lambda$start$1(ByteStreamUploader.java:397)
        at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticatedAsync(Utils.java:433)
        at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.start(ByteStreamUploader.java:395)
        at com.google.devtools.build.lib.remote.ByteStreamUploader.startAsyncUpload(ByteStreamUploader.java:329)
        at com.google.devtools.build.lib.remote.ByteStreamUploader.uploadBlobAsync(ByteStreamUploader.java:250)
        at com.google.devtools.build.lib.remote.GrpcCacheClient.uploadBlob(GrpcCacheClient.java:409)
        at com.google.devtools.build.lib.remote.RemoteExecutionCache.uploadBlob(RemoteExecutionCache.java:82)
        at com.google.devtools.build.lib.remote.RemoteExecutionCache.ensureInputsPresent(RemoteExecutionCache.java:69)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:335)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:244)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:125)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:114)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:323)
        at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:240)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:140)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:102)
        at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)
        at com.google.devtools.build.lib.rules.cpp.CppCompileAction.beginExecution(CppCompileAction.java:1452)
        at com.google.devtools.build.lib.actions.Action.execute(Action.java:127)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:855)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1016)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
        ... 4 more
Caused by: java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(Unknown Source)
        at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(Unknown Source)
        at com.google.devtools.build.lib.remote.grpc.SharedConnectionFactory.createUnderlyingConnectionIfNot(SharedConnectionFactory.java:94)
        at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleDoOnError.subscribeActual(SingleDoOnError.java:35)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleDoOnDispose.subscribeActual(SingleDoOnDispose.java:38)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:35)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:85)
        at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:68)
        at com.google.devtools.build.lib.remote.grpc.TokenBucket$1.onNext(TokenBucket.java:79)
        at io.reactivex.rxjava3.internal.util.NotificationLite.accept(NotificationLite.java:246)
        at io.reactivex.rxjava3.subjects.BehaviorSubject$BehaviorDisposable.test(BehaviorSubject.java:507)
        at io.reactivex.rxjava3.subjects.BehaviorSubject$BehaviorDisposable.emitFirst(BehaviorSubject.java:468)
        at io.reactivex.rxjava3.subjects.BehaviorSubject.subscribeActual(BehaviorSubject.java:224)
        at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13102)
        at com.google.devtools.build.lib.remote.grpc.TokenBucket.lambda$acquireToken$0(TokenBucket.java:64)
        at io.reactivex.rxjava3.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:40)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleDefer.subscribeActual(SingleDefer.java:43)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.core.Single.blockingGet(Single.java:3644)
        ... 38 more

This does not happen with 4.0.0. It does happen with 4.1.0rc1 and last_green.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I'm not exactly sure, but it seems when a build failure occurs early enough in the build, while using remote execution, the above error happens.

What operating system are you running Bazel on?

macOS 11.2.1

What's the output of bazel info release?

release 4.1.0rc1

Metadata

Metadata

Assignees

Labels

P1I'll work on this now. (Assignee required)team-Remote-ExecIssues and PRs for the Execution (Remote) teamtype: bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions