"Basic Tests ObjC iOS" is failing often #23066
Description
Dashboard for this test: https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod:grpc%2Fcore%2Fpull_request%2Fmacos%2Fgrpc_basictests_objc_ios
Note that this is failing very often recently.
Some of the runs passed, mostly only because tests are skipped. Example of a passed run: https://source.cloud.google.com/results/invocations/b3f336f6-4374-4abd-85b3-cbf153bd533c/targets
Here are some of the errors (may not be root cause):
Wed May 27 21:27:00 PDT 2020 - 2020-05-27 21:25:40.261171-0700 xctest[87908:180719] []
nw_socket_handle_socket_event [C69:1] Socket SO_ERROR [61: Connection refused]
And looks like most of the failures happen to
ios-test-cronettests
while
ios-test-cfstream-tests
ios-test-interoptests
ios-test-plugintest
ios-test-unittests
are usually passing just fine.
This is the test config: https://github.com/grpc/grpc/blob/master/tools/internal_ci/macos/pull_request/grpc_basictests_objc_ios.cfg
This is the kokoro macos worker config: cs://google3/devtools/kokoro/config/prod/grpc/core/pull_request/macos/common.cfg
Previously also logged #20020 #20094
On closer look at the log files, there are errors like this:
Testing failed:
'grpc/impl/codegen/byte_buffer.h' file not found
** TEST FAILED **
The following build commands failed:
CompileC /Users/kbuilder/Library/Developer/Xcode/DerivedData/CFStreamTests-eysagtowlpfcwyfevvouytnaykfx/Build/Intermediates.noindex/CFStreamTests.build/Debug-iphonesimulator/CFStreamTests.build/Objects-normal/x86_64/CFStreamEndpointTests.o CFStreamEndpointTests.mm normal x86_64 objective-c++ com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
/Volumes/BuildData/tmpfs/src/github/grpc/workspace_objc_macos_opt_native/test/core/iomgr/ios/CFStreamTests/Pods/Headers/Public/gRPC-Core/grpc/byte_buffer.h:24:10: fatal error: 'grpc/impl/codegen/byte_buffer.h' file not found
#include <grpc/impl/codegen/byte_buffer.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
2020-05-27 21:41:03,267 FAILED: ios-test-cfstream-tests [ret=65, pid=38459, time=270.9sec
-[CoreCronetEnd2EndTests testServerFinishesRequest]
** TEST FAILED **