-
Notifications
You must be signed in to change notification settings - Fork 11.1k
linux/grpc_interop_toprod failure in dart #26652
Description
2021-07-09 13:18:10,209 FAILED: cloud_to_prod:dart:default:large_unary:tls [ret=255, pid=14541, time=5.5sec]
2021-07-09 13:18:10,209 START: cloud_to_prod:dart:default:timeout_on_sleeping_server:tls
2021-07-09 13:18:10,216 WAITING: ETA 207.4 sec; 458 queued, 8 jobs running, 28 complete, 1 failed (load 8.00) next: cloud_to_prod:python:default:large_unary:tls @ 1.00 cpu
2021-07-09 13:18:10,342 Unhandled exception:
gRPC Error (code: 14, codeName: UNAVAILABLE, message: Error connecting: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354)), details: null, rawResponse: null)
2021-07-09 13:18:10,342 FAILED: cloud_to_prod:dart:default:client_streaming:tls [ret=255, pid=14617, time=5.5sec]
2021-07-09 13:18:10,343 START: cloud_to_prod:python:default:large_unary:tls
2021-07-09 13:18:10,350 WAITING: ETA 201.9 sec; 457 queued, 8 jobs running, 29 complete, 2 failed (load 8.00) next: cloud_to_prod:python:default:empty_unary:tls @ 1.00 cpu
2021-07-09 13:18:10,473 Unhandled exception:
gRPC Error (code: 14, codeName: UNAVAILABLE, message: Error connecting: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354)), details: null, rawResponse: null)
2021-07-09 13:18:10,473 FAILED: cloud_to_prod:dart:default:server_streaming:tls [ret=255, pid=14655, time=5.6sec]
2021-07-09 13:18:10,474 START: cloud_to_prod:python:default:empty_unary:tls
2021-07-09 13:18:10,484 WAITING: ETA 196.8 sec; 456 queued, 8 jobs running, 30 complete, 3 failed (load 8.00) next: cloud_to_prod:python:default:ping_pong:tls @ 1.00 cpu
2021-07-09 13:18:10,553 Unhandled exception:
gRPC Error (code: 14, codeName: UNAVAILABLE, message: Error connecting: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354)), details: null, rawResponse: null)
2021-07-09 13:18:10,553 FAILED: cloud_to_prod:dart:default:empty_unary:tls [ret=255, pid=14564, time=5.8sec]
2021-07-09 13:18:10,554 START: cloud_to_prod:python:default:ping_pong:tls
2021-07-09 13:18:10,560 WAITING: ETA 191.2 sec; 455 queued, 8 jobs running, 31 complete, 4 failed (load 8.00) next: cloud_to_prod:python:default:empty_stream:tls @ 1.00 cpu
2021-07-09 13:18:10,836 Unhandled exception:
gRPC Error (code: 14, codeName: UNAVAILABLE, message: Error connecting: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354)), details: null, rawResponse: null)
2021-07-09 13:18:10,836 FAILED: cloud_to_prod:dart:default:ping_pong:tls [ret=255, pid=14571, time=6.1sec]
2021-07-09 13:18:10,836 START: cloud_to_prod:python:default:empty_stream:tls
2021-07-09 13:18:10,840 WAITING: ETA 188.9 sec; 454 queued, 8 jobs running, 32 complete, 5 failed (load 8.00) next: cloud_to_prod:python:default:client_streaming:tls @ 1.00 cpu
2021-07-09 13:18:10,893 Unhandled exception:
gRPC Error (code: 14, codeName: UNAVAILABLE, message: Error connecting: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354)), details: null, rawResponse: null)
2021-07-09 13:18:10,893 FAILED: cloud_to_prod:dart:default:empty_stream:tls [ret=255, pid=14609, time=6.1sec]
2021-07-09 13:18:10,894 START: cloud_to_prod:python:default:client_streaming:tls
2021-07-09 13:18:10,898 WAITING: ETA 183.6 sec; 453 queued, 8 jobs running, 33 complete, 6 failed (load 8.00) next: cloud_to_prod:python:default:server_streaming:tls @ 1.00 cpu
2021-07-09 13:18:11,052 PASSED: cloud_to_prod:dart:default:cancel_after_begin:tls [time=5.5sec, retries=0:0]
2021-07-09 13:18:11,053 START: cloud_to_prod:python:default:server_streaming:tls
2021-07-09 13:18:11,058 WAITING: ETA 180.0 sec; 452 queued, 8 jobs running, 34 complete, 6 failed (load 8.00) next: cloud_to_prod:python:default:cancel_after_begin:tls @ 1.00 cpu
2021-07-09 13:18:11,428 Unhandled exception:
Unexpected status code 14: Error connecting: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:354)).
2021-07-09 13:18:11,428 FAILED: cloud_to_prod:dart:default:cancel_after_first_response:tls [ret=255, pid=15194, time=5.3sec]
2021-07-09 13:18:11,429 START: cloud_to_prod:python:default:cancel_after_begin:tls
2021-07-09 13:18:11,434 WAITING: ETA 179.4 sec; 451 queued, 8 jobs running, 35 complete, 7 failed (load 8.00) next: cloud_to_prod:python:default:cancel_after_first_response:tls @ 1.00 cpu
2021-07-09 13:18:12,165 PASSED: cloud_to_prod:python:default:empty_unary:tls [time=1.7sec, retries=0:0]
2021-07-09 13:18:12,165 START: cloud_to_prod:python:default:cancel_after_first_response:tls
2021-07-09 13:18:12,169 WAITING: ETA 183.4 sec; 450 queued, 8 jobs running, 36 complete, 7 failed (load 8.00) next: cloud_to_prod:python:default:timeout_on_sleeping_server:tls @ 1.00 cpu
2021-07-09 13:18:12,237 PASSED: cloud_to_prod:python:default:ping_pong:tls [time=1.7sec, retries=0:0]
2021-07-09 13:18:12,238 START: cloud_to_prod:python:default:timeout_on_sleeping_server:tls
2021-07-09 13:18:12,243 WAITING: ETA 178.9 sec; 449 queued, 8 jobs running, 37 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:large_unary:tls @ 1.00 cpu
2021-07-09 13:18:12,300 PASSED: cloud_to_prod:python:default:large_unary:tls [time=2.0sec, retries=0:0]
2021-07-09 13:18:12,300 START: cloud_to_prod:go:default:large_unary:tls
2021-07-09 13:18:12,305 WAITING: ETA 174.6 sec; 448 queued, 8 jobs running, 38 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:large_unary:google_default_credentials @ 1.00 cpu
2021-07-09 13:18:12,451 PASSED: cloud_to_prod:python:default:empty_stream:tls [time=1.6sec, retries=0:0]
2021-07-09 13:18:12,451 START: cloud_to_prod:go:default:large_unary:google_default_credentials
2021-07-09 13:18:12,456 WAITING: ETA 171.5 sec; 447 queued, 8 jobs running, 39 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:large_unary:compute_engine_channel_creds @ 1.00 cpu
2021-07-09 13:18:12,501 PASSED: cloud_to_prod:python:default:client_streaming:tls [time=1.6sec, retries=0:0]
2021-07-09 13:18:12,502 START: cloud_to_prod:go:default:large_unary:compute_engine_channel_creds
2021-07-09 13:18:12,507 WAITING: ETA 167.4 sec; 446 queued, 8 jobs running, 40 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:empty_unary:tls @ 1.00 cpu
2021-07-09 13:18:12,678 PASSED: cloud_to_prod:python:default:server_streaming:tls [time=1.6sec, retries=0:0]
2021-07-09 13:18:12,679 START: cloud_to_prod:go:default:empty_unary:tls
2021-07-09 13:18:12,683 WAITING: ETA 164.9 sec; 445 queued, 8 jobs running, 41 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:empty_unary:google_default_credentials @ 1.00 cpu
2021-07-09 13:18:12,928 PASSED: cloud_to_prod:python:default:cancel_after_begin:tls [time=1.5sec, retries=0:0]
2021-07-09 13:18:12,928 START: cloud_to_prod:go:default:empty_unary:google_default_credentials
2021-07-09 13:18:12,933 WAITING: ETA 163.3 sec; 444 queued, 8 jobs running, 42 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:empty_unary:compute_engine_channel_creds @ 1.00 cpu
2021-07-09 13:18:13,803 PASSED: cloud_to_prod:python:default:timeout_on_sleeping_server:tls [time=1.6sec, retries=0:0]
2021-07-09 13:18:13,804 START: cloud_to_prod:go:default:empty_unary:compute_engine_channel_creds
2021-07-09 13:18:13,810 WAITING: ETA 168.4 sec; 443 queued, 8 jobs running, 43 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:ping_pong:tls @ 1.00 cpu
2021-07-09 13:18:13,823 PASSED: cloud_to_prod:python:default:cancel_after_first_response:tls [time=1.7sec, retries=0:0]
2021-07-09 13:18:13,824 START: cloud_to_prod:go:default:ping_pong:tls
2021-07-09 13:18:13,827 WAITING: ETA 164.4 sec; 442 queued, 8 jobs running, 44 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:ping_pong:google_default_credentials @ 1.00 cpu
2021-07-09 13:18:14,391 PASSED: cloud_to_prod:dart:default:timeout_on_sleeping_server:tls [time=4.2sec, retries=0:0]
2021-07-09 13:18:14,392 START: cloud_to_prod:go:default:ping_pong:google_default_credentials
2021-07-09 13:18:14,398 WAITING: ETA 166.0 sec; 441 queued, 8 jobs running, 45 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:ping_pong:compute_engine_channel_creds @ 1.00 cpu
2021-07-09 13:18:16,697 PASSED: cloud_to_prod:go:default:empty_unary:tls [time=4.0sec, retries=0:0]
2021-07-09 13:18:16,698 START: cloud_to_prod:go:default:ping_pong:compute_engine_channel_creds
2021-07-09 13:18:16,702 WAITING: ETA 184.5 sec; 440 queued, 8 jobs running, 46 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:empty_stream:tls @ 1.00 cpu
2021-07-09 13:18:16,706 PASSED: cloud_to_prod:go:default:empty_unary:google_default_credentials [time=3.8sec, retries=0:0]
2021-07-09 13:18:16,706 START: cloud_to_prod:go:default:empty_stream:tls
2021-07-09 13:18:16,712 PASSED: cloud_to_prod:go:default:empty_unary:compute_engine_channel_creds [time=2.9sec, retries=0:0]
2021-07-09 13:18:16,712 START: cloud_to_prod:go:default:empty_stream:google_default_credentials
2021-07-09 13:18:16,716 WAITING: ETA 176.2 sec; 438 queued, 8 jobs running, 48 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:empty_stream:compute_engine_channel_creds @ 1.00 cpu
2021-07-09 13:18:16,922 PASSED: cloud_to_prod:go:default:large_unary:compute_engine_channel_creds [time=4.4sec, retries=0:0]
2021-07-09 13:18:16,924 START: cloud_to_prod:go:default:empty_stream:compute_engine_channel_creds
2021-07-09 13:18:16,930 PASSED: cloud_to_prod:go:default:large_unary:tls [time=4.6sec, retries=0:0]
2021-07-09 13:18:16,931 START: cloud_to_prod:go:default:client_streaming:tls
2021-07-09 13:18:16,934 PASSED: cloud_to_prod:go:default:ping_pong:tls [time=3.1sec, retries=0:0]
2021-07-09 13:18:16,935 START: cloud_to_prod:go:default:client_streaming:google_default_credentials
2021-07-09 13:18:16,939 PASSED: cloud_to_prod:go:default:large_unary:google_default_credentials [time=4.5sec, retries=0:0]
2021-07-09 13:18:16,940 START: cloud_to_prod:go:default:client_streaming:compute_engine_channel_creds
2021-07-09 13:18:16,945 WAITING: ETA 163.1 sec; 434 queued, 8 jobs running, 52 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:server_streaming:tls @ 1.00 cpu
2021-07-09 13:18:17,421 PASSED: cloud_to_prod:go:default:ping_pong:google_default_credentials [time=3.0sec, retries=0:0]
2021-07-09 13:18:17,421 START: cloud_to_prod:go:default:server_streaming:tls
2021-07-09 13:18:17,426 WAITING: ETA 163.7 sec; 433 queued, 8 jobs running, 53 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:server_streaming:google_default_credentials @ 1.00 cpu
2021-07-09 13:18:19,781 PASSED: cloud_to_prod:go:default:ping_pong:compute_engine_channel_creds [time=3.1sec, retries=0:0]
2021-07-09 13:18:19,781 START: cloud_to_prod:go:default:server_streaming:google_default_credentials
2021-07-09 13:18:19,786 WAITING: ETA 179.5 sec; 432 queued, 8 jobs running, 54 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:server_streaming:compute_engine_channel_creds @ 1.00 cpu
2021-07-09 13:18:19,904 PASSED: cloud_to_prod:go:default:empty_stream:tls [time=3.2sec, retries=0:0]
2021-07-09 13:18:19,905 START: cloud_to_prod:go:default:server_streaming:compute_engine_channel_creds
2021-07-09 13:18:19,910 PASSED: cloud_to_prod:go:default:empty_stream:google_default_credentials [time=3.2sec, retries=0:0]
2021-07-09 13:18:19,910 START: cloud_to_prod:go:default:cancel_after_begin:tls
2021-07-09 13:18:19,914 WAITING: ETA 173.3 sec; 430 queued, 8 jobs running, 56 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:cancel_after_begin:google_default_credentials @ 1.00 cpu
2021-07-09 13:18:20,088 PASSED: cloud_to_prod:go:default:client_streaming:compute_engine_channel_creds [time=3.1sec, retries=0:0]
2021-07-09 13:18:20,088 START: cloud_to_prod:go:default:cancel_after_begin:google_default_credentials
2021-07-09 13:18:20,093 PASSED: cloud_to_prod:go:default:client_streaming:tls [time=3.2sec, retries=0:0]
2021-07-09 13:18:20,093 START: cloud_to_prod:go:default:cancel_after_begin:compute_engine_channel_creds
2021-07-09 13:18:20,096 PASSED: cloud_to_prod:go:default:client_streaming:google_default_credentials [time=3.2sec, retries=0:0]
2021-07-09 13:18:20,097 START: cloud_to_prod:go:default:cancel_after_first_response:tls
2021-07-09 13:18:20,102 WAITING: ETA 164.8 sec; 427 queued, 8 jobs running, 59 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:cancel_after_first_response:google_default_credentials @ 1.00 cpu
2021-07-09 13:18:20,147 PASSED: cloud_to_prod:go:default:empty_stream:compute_engine_channel_creds [time=3.2sec, retries=0:0]
2021-07-09 13:18:20,147 START: cloud_to_prod:go:default:cancel_after_first_response:google_default_credentials
2021-07-09 13:18:20,152 WAITING: ETA 162.0 sec; 426 queued, 8 jobs running, 60 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:cancel_after_first_response:compute_engine_channel_creds @ 1.00 cpu
2021-07-09 13:18:20,399 PASSED: cloud_to_prod:go:default:server_streaming:tls [time=3.0sec, retries=0:0]
2021-07-09 13:18:20,401 START: cloud_to_prod:go:default:cancel_after_first_response:compute_engine_channel_creds
2021-07-09 13:18:20,407 WAITING: ETA 160.8 sec; 425 queued, 8 jobs running, 61 complete, 7 failed (load 8.00) next: cloud_to_prod:go:default:timeout_on_sleeping_server:tls @ 1.00 cpu
2021-07-09 13:18:22,822 PASSED: cloud_to_prod:go:default:cancel_after_begin:tls [time=2.9sec, retries=0:0]
2021-07-09 13:18:22,823 START: cloud_to_prod:go:default:timeout_on_sleeping_server:tls
2021-07-09 13:18:22,827 PASSED: cloud_to_prod:go:default:server_streaming:google_default_credentials [time=3.0sec, retries=0:0]
2021-07-09 13:18:22,828 START: cloud_to_prod:go:default:timeout_on_sleeping_server:google_default_credentials