Skip to content

502 errors in cloud to prod interop tests for aspnetcore, php, and dart clients #19723

@apolcyn

Description

@apolcyn

Saw a bunch of 502 errors from aspentcore, php, and dart clients when targetting the default cloud to prod interop server.

E.g.:

2019-07-22 21:01:52,284 START: cloud_to_prod:php7:default:ping_pong:tls
2019-07-22 21:01:52,289 WAITING: ETA 791.9 sec; 470 queued, 12 jobs running, 18 complete, 12 failed (load 12.00) next: cloud_to_prod:php7:default:empty_stream:tls @ 1.00 cpu
2019-07-22 21:02:02,289 WAITING: ETA 1059.7 sec; 470 queued, 12 jobs running, 18 complete, 12 failed (load 12.00) next: cloud_to_prod:php7:default:empty_stream:tls @ 1.00 cpu
2019-07-22 21:02:12,290 WAITING: ETA 1327.5 sec; 470 queued, 12 jobs running, 18 complete, 12 failed (load 12.00) next: cloud_to_prod:php7:default:empty_stream:tls @ 1.00 cpu
2019-07-22 21:02:12,964 Unhandled exception:
gRPC Error (14, 502:Bad Gateway)
#0      Tester.emptyUnary (package:interop/src/client.dart:177:22)
<asynchronous suspension>
#1      Tester.runTestCase (package:interop/src/client.dart:115:16)
<asynchronous suspension>
#2      Tester.runTest (package:interop/src/client.dart:108:11)
<asynchronous suspension>
#3      main (file:///var/local/git/grpc-dart/interop/bin/client.dart:107:20)
<asynchronous suspension>
#4      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#5      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)
2019-07-22 21:02:12,964 FAILED: cloud_to_prod:dart:default:empty_unary:tls [ret=255, pid=10952, time=28.4sec]
2019-07-22 21:02:12,964 START: cloud_to_prod:php7:default:empty_stream:tls
2019-07-22 21:02:12,969 WAITING: ETA 1272.2 sec; 469 queued, 12 jobs running, 19 complete, 13 failed (load 12.00) next: cloud_to_prod:php7:default:client_streaming:tls @ 1.00 cpu
2019-07-22 21:02:13,354 Running test case ping_pong
Server returned too few responses
2019-07-22 21:02:13,355 FAILED: cloud_to_prod:php7:default:ping_pong:tls [ret=1, pid=12085, time=21.1sec]
2019-07-22 21:02:13,355 START: cloud_to_prod:php7:default:client_streaming:tls
2019-07-22 21:02:13,359 WAITING: ETA 1215.4 sec; 468 queued, 12 jobs running, 20 complete, 14 failed (load 12.00) next: cloud_to_prod:php7:default:server_streaming:tls @ 1.00 cpu
2019-07-22 21:02:13,398 Unhandled exception:
gRPC Error (14, 502:Bad Gateway)
#0      Tester.pingPong (package:interop/src/client.dart:585:5)
<asynchronous suspension>
#1      Tester.runTestCase (package:interop/src/client.dart:133:16)
<asynchronous suspension>
#2      Tester.runTest (package:interop/src/client.dart:108:11)
<asynchronous suspension>
#3      main (file:///var/local/git/grpc-dart/interop/bin/client.dart:107:20)
<asynchronous suspension>
#4      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#5      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)
2019-07-22 21:02:13,398 FAILED: cloud_to_prod:dart:default:ping_pong:tls [ret=255, pid=10974, time=28.8sec]
2019-07-22 21:02:13,398 START: cloud_to_prod:php7:default:server_streaming:tls
2019-07-22 21:02:13,403 WAITING: ETA 1156.1 sec; 467 queued, 12 jobs running, 21 complete, 15 failed (load 12.00) next: cloud_to_prod:php7:default:cancel_after_begin:tls @ 1.00 cpu
2019-07-22 21:02:13,809 Unhandled exception:
gRPC Error (14, 502:Bad Gateway)
#0      Tester.serverStreaming (package:interop/src/client.dart:466:23)
<asynchronous suspension>
#1      Tester.runTestCase (package:interop/src/client.dart:129:16)
<asynchronous suspension>
#2      Tester.runTest (package:interop/src/client.dart:108:11)
<asynchronous suspension>
#3      main (file:///var/local/git/grpc-dart/interop/bin/client.dart:107:20)
<asynchronous suspension>
#4      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#5      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)
2019-07-22 21:02:13,810 FAILED: cloud_to_prod:dart:default:server_streaming:tls [ret=255, pid=11011, time=29.2sec]
2019-07-22 21:02:13,810 START: cloud_to_prod:php7:default:cancel_after_begin:tls
2019-07-22 21:02:13,815 WAITING: ETA 1110.2 sec; 466 queued, 12 jobs running, 22 complete, 16 failed (load 12.00) next: cloud_to_prod:php7:default:cancel_after_first_response:tls @ 1.00 cpu
2019-07-22 21:02:13,833 Running test case empty_unary
Call did not complete successfully. Status object:
object(stdClass)#389 (3) {
  ["metadata"]=>
  array(3) {
    ["content-length"]=>
    array(1) {
      [0]=>
      string(1) "0"
    }
    ["date"]=>
    array(1) {
      [0]=>
      string(29) "Tue, 23 Jul 2019 04:02:14 GMT"
    }
    ["expect-ct"]=>
    array(1) {
      [0]=>
      string(7) "preload"
    }
  }
  ["code"]=>
  int(14)
  ["details"]=>
  string(15) "502:Bad Gateway"
}
2019-07-22 21:02:13,834 FAILED: cloud_to_prod:php7:default:empty_unary:tls [ret=1, pid=11956, time=22.0sec]
2019-07-22 21:02:13,834 START: cloud_to_prod:php7:default:cancel_after_first_response:tls
2019-07-22 21:02:13,839 WAITING: ETA 1060.2 sec; 465 queued, 12 jobs running, 23 complete, 17 failed (load 12.00) next: cloud_to_prod:php7:default:timeout_on_sleeping_server:tls @ 1.00 cpu
2019-07-22 21:02:13,841 Running test case large_unary
Call did not complete successfully. Status object:
object(stdClass)#390 (3) {

Full log in https://source.cloud.google.com/results/invocations/149b68de-e762-457b-9be3-1babe2915130/targets/grpc%2Fcore%2Fpull_request%2Flinux%2Fgrpc_interop_toprod/log

Seen in PR: #19693

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions