Upgrade c-ares to 1.13.0 by jtattermusch · Pull Request #12305 · grpc/grpc

jtattermusch · 2017-08-28T17:52:06Z

-- starting from this version, c-ares has its own support for cmake, switch to using c-ares cmake
-- update ares_build.h and ares_config.h that's used for bazel build & Makefile build

Using c-ares' cmake is useful because it supports installing c-ares, which is in turn needed for properly supporting installing gRPC as a cmake package (I have some in-progress work to improve gRPC's cmake build).

jtattermusch · 2017-08-28T17:52:43Z

@y-zeng Where can I find a freebsd machine to regenerate the ares_config.h?

y-zeng · 2017-08-28T18:37:45Z

@jtattermusch Thanks for doing this upgrade!
@mehrdada Do you know where we can find a freebsd machine to regenerate the ares_config.h?

grpc-kokoro · 2017-08-28T19:15:54Z

Corrupt JSON data (indicates timeout or crash): 
    bm_fullstack_trickle.BM_PumpStreamServerToClient_Trickle_8_4k.opt.new: 1


[trickle] No significant performance differences

grpc-kokoro · 2017-08-28T19:43:26Z

[microbenchmarks] Performance differences noted:
Benchmark                                                                               cpu_time    real_time
--------------------------------------------------------------------------------------  ----------  -----------
BM_PumpStreamServerToClient<InProcess>/256k                                             -6%         -5%
BM_StreamingPingPong<InProcess, NoOpMutator, NoOpMutator>/32k/2                         +4%         +4%
BM_StreamingPingPong<MinInProcess, NoOpMutator, NoOpMutator>/256k/2                     +4%         +4%
BM_StreamingPingPongMsgs<InProcess, NoOpMutator, NoOpMutator>/32k                       +5%         +5%
BM_StreamingPingPongMsgs<MinInProcess, NoOpMutator, NoOpMutator>/256k                   +6%         +6%
BM_StreamingPingPongMsgs<MinInProcess, NoOpMutator, NoOpMutator>/32k                    +11%        +11%
BM_StreamingPingPongWithCoalescingApi<InProcess, NoOpMutator, NoOpMutator>/256k/1/1     +9%         +9%
BM_StreamingPingPongWithCoalescingApi<InProcess, NoOpMutator, NoOpMutator>/32k/1/1      +8%         +8%
BM_StreamingPingPongWithCoalescingApi<InProcess, NoOpMutator, NoOpMutator>/32k/2/1      +11%        +11%
BM_StreamingPingPongWithCoalescingApi<MinInProcess, NoOpMutator, NoOpMutator>/256k/1/0  +9%         +10%
BM_StreamingPingPongWithCoalescingApi<MinInProcess, NoOpMutator, NoOpMutator>/32k/1/1   +5%         +5%
BM_StreamingPingPongWithCoalescingApi<MinInProcess, NoOpMutator, NoOpMutator>/32k/2/0   +5%         +5%
BM_UnaryPingPong<InProcess, NoOpMutator, NoOpMutator>/0/32k                             +5%         +6%
BM_UnaryPingPong<InProcess, NoOpMutator, NoOpMutator>/32k/0                             +4%         +4%
BM_UnaryPingPong<MinInProcess, NoOpMutator, NoOpMutator>/32k/0                          +10%        +11%
BM_UnaryPingPong<MinInProcessCHTTP2, NoOpMutator, NoOpMutator>/32k/32k                  +4%

apolcyn · 2017-08-28T19:39:47Z

third_party/cares/config_linux/ares_config.h

- * Note: setting HAVE_CLOCK_GETTIME_MONOTONIC causes use of the clock_gettime
- * function from glibc, don't set it to support glibc < 2.17 */
-#ifndef GPR_BACKWARDS_COMPATIBILITY_MODE
-  #define HAVE_CLOCK_GETTIME_MONOTONIC 1


I think we need to preserve this manual edit. HAVE_CLOCK_GETTIME_MONOTONIC being defined forces a dependency on glibc 1.17+ - it should break a couple of ruby distrib tests.

apolcyn · 2017-08-28T19:43:57Z

third_party/cares/config_linux/ares_config.h

-#endif
-
-#ifdef GPR_BACKWARDS_COMPATIBILITY_MODE
-  /* Redefine the fd_set macros for GLIBC < 2.15 support.


Same here, I think we also need to preserve this manual edit. Building against latest FD_SET macros forces the binary dependency on glibc1.5+ (a couple of ruby distrib tests should also fail on this, #11346 for context)

jtattermusch · 2017-08-29T16:04:32Z

@apolcyn, how was the ares_config.h for macos generated? I generated one one my gMac, but I'm getting a build error:

mkdir -p `dirname /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares__timeval.o`
cc -Ithird_party/protobuf/src -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -Ithird_party/boringssl/include -Ithird_party/cares -Ithird_party/cares/cares -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -O0 -fPIC -I. -Iinclude -I/jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/gens -I/opt/local/include -I/usr/local/include -D_DEBUG -DDEBUG -DINSTALL_PREFIX=\"/usr/local\" -DGRPC_TEST_SLOWDOWN_MACHINE_FACTOR=1.000000    -Ithird_party/cares -Ithird_party/cares/cares  -Ithird_party/cares/config_darwin -fvisibility=hidden -D_GNU_SOURCE -DWIN32_LEAN_AND_MEAN -D_HAS_EXCEPTIONS=0 -DNOMINMAX -DHAVE_CONFIG_H -std=c99 -Wsign-conversion -Wconversion -Wshadow -Wextra-semi   -Wno-sign-conversion -Wno-shorten-64-to-32 -Wno-invalid-source-encoding -MMD -MF /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares__read_line.dep -c -o /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares__read_line.o third_party/cares/cares/ares__read_line.c
cc -Ithird_party/protobuf/src -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -Ithird_party/boringssl/include -Ithird_party/cares -Ithird_party/cares/cares -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -O0 -fPIC -I. -Iinclude -I/jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/gens -I/opt/local/include -I/usr/local/include -D_DEBUG -DDEBUG -DINSTALL_PREFIX=\"/usr/local\" -DGRPC_TEST_SLOWDOWN_MACHINE_FACTOR=1.000000    -Ithird_party/cares -Ithird_party/cares/cares  -Ithird_party/cares/config_darwin -fvisibility=hidden -D_GNU_SOURCE -DWIN32_LEAN_AND_MEAN -D_HAS_EXCEPTIONS=0 -DNOMINMAX -DHAVE_CONFIG_H -std=c99 -Wsign-conversion -Wconversion -Wshadow -Wextra-semi   -Wno-sign-conversion -Wno-shorten-64-to-32 -Wno-invalid-source-encoding -MMD -MF /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares__timeval.dep -c -o /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares__timeval.o third_party/cares/cares/ares__timeval.c
third_party/cares/cares/ares__timeval.c:48:11: error: implicit declaration of function 'clock_gettime' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  if(0 == clock_gettime(CLOCK_MONOTONIC, &tsnow)) {
          ^
[C]       Compiling third_party/cares/cares/ares_cancel.c
third_party/cares/cares/ares__timeval.c:48:25: error: use of undeclared identifier 'CLOCK_MONOTONIC'
  if(0 == clock_gettime(CLOCK_MONOTONIC, &tsnow)) {
                        ^
2 errors generated.
mkdir -p `dirname /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares_cancel.o`
make: *** [/jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares__timeval.o] Error 1
make: *** Waiting for unfinished jobs....
cc -Ithird_party/protobuf/src -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -Ithird_party/boringssl/include -Ithird_party/cares -Ithird_party/cares/cares -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -O0 -fPIC -I. -Iinclude -I/jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/gens -I/opt/local/include -I/usr/local/include -D_DEBUG -DDEBUG -DINSTALL_PREFIX=\"/usr/local\" -DGRPC_TEST_SLOWDOWN_MACHINE_FACTOR=1.000000    -Ithird_party/cares -Ithird_party/cares/cares  -Ithird_party/cares/config_darwin -fvisibility=hidden -D_GNU_SOURCE -DWIN32_LEAN_AND_MEAN -D_HAS_EXCEPTIONS=0 -DNOMINMAX -DHAVE_CONFIG_H -std=c99 -Wsign-conversion -Wconversion -Wshadow -Wextra-semi   -Wno-sign-conversion -Wno-shorten-64-to-32 -Wno-invalid-source-encoding -MMD -MF /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares_cancel.dep -c -o /jenkins/workspace/gRPC_pull_requests_macos/workspace_c_macos_dbg_native/objs/dbg/third_party/cares/cares/ares_cancel.o third_party/cares/cares/ares_cancel.c

2017-08-28 11:47:33,531 FAILED: make [ret=2, pid=11412]
2017-08-28 11:47:33,531 FAILED: Some tests failed

Also, any idea how to regenerate on freebsd?

apolcyn · 2017-08-29T16:31:17Z

I hadn't generated these header files before and am unsure off the top of my head. But I believe the current macos config_ares does also undefine HAVE_CLOCK_GETTIME_MONOTONIC https://github.com/grpc/grpc/blob/master/third_party/cares/config_darwin/ares_config.h#L73

jtattermusch · 2017-08-29T16:42:07Z

@mehrdada it looks like config_freebsd/ares_config.h was added by you in #10724. How do I regenerate it?

mehrdada · 2017-08-29T16:59:20Z

@jtattermusch If I remember correctly, I tried running ./configure for c-ares on a FreeBSD machine and extracted the generated ares_config.h.

mehrdada · 2017-08-29T17:00:34Z

@jtattermusch If you like, I can try generating a new one and push to your branch from a FreeBSD image?

jtattermusch · 2017-08-29T17:15:06Z

That would be great, thanks. Either push there or just send me the config file.

…

On Aug 29, 2017 19:00, "Mehrdad Afshari" ***@***.***> wrote: @jtattermusch <https://github.com/jtattermusch> If you like, I can try generating a new one and push to your branch from a FreeBSD image? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12305 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJeq5Mpd7bzmkgjsVGPUxX3qOB8TjhoOks5sdEPHgaJpZM4PE2Gn> .

grpc-kokoro · 2017-08-29T17:19:32Z

[trickle] No significant performance differences

mehrdada · 2017-08-29T17:26:40Z

Done. Please merge this PR: jtattermusch#10

grpc-kokoro · 2017-08-29T17:53:07Z

[microbenchmarks] No significant performance differences

mehrdada · 2017-08-29T18:32:03Z

@y-zeng sorry I missed your comment above. I already regenerated it and @jtattermusch has merged. I have a FreeBSD machine at home, but just in case for future reference, you can create a GCE machine with a FreeBSD image with:

gcloud compute instances create "freebsd11-machine-name" \
--machine-type "n1-standard-8" \
--maintenance-policy "MIGRATE" \
--image "freebsd-11-0-release-p1-amd64" --image-project=freebsd-org-cloud-dev \
--boot-disk-size "40" --boot-disk-type "pd-ssd"

grpc-kokoro · 2017-08-29T18:38:58Z

[trickle] No significant performance differences

grpc-kokoro · 2017-08-29T19:15:18Z

[microbenchmarks] No significant performance differences

y-zeng · 2017-08-29T21:29:11Z

Thanks @mehrdada!
@jtattermusch Please run the Jenkins tests with GRPC_DNS_RESOLVER=ares to make sure the upgrade will not break the resolver.

jtattermusch · 2017-08-30T06:19:11Z

Python 32 bit windows artifact compiled with mingw still broken:

In file included from third_party/cares/cares/ares_init.c:58:0:
third_party/cares/cares/ares_library_init.h:31:30: error: expected declaration specifiers or '...' before '*' token
 typedef NETIO_STATUS (WINAPI *fpGetBestRoute2_t) ( NET_LUID *, NET_IFINDEX, const SOCKADDR_INET *, const SOCKADDR_INET *, ULONG, PMIB_IPFORWARD_ROW2, SOCKADDR_INET * );
                              ^
In file included from third_party/cares/cares/ares_init.c:58:0:
third_party/cares/cares/ares_library_init.h:38:8: error: unknown type name 'fpGetBestRoute2_t'
 extern fpGetBestRoute2_t ares_fpGetBestRoute2;
        ^~~~~~~~~~~~~~~~~
third_party/cares/cares/ares_init.c: In function 'getBestRouteMetric':
third_party/cares/cares/ares_init.c:1022:3: error: unknown type name 'MIB_IPFORWARD_ROW2'
   MIB_IPFORWARD_ROW2 row;
   ^~~~~~~~~~~~~~~~~~
third_party/cares/cares/ares_init.c:1025:6: error: called object 'ares_fpGetBestRoute2' is not a function or function pointer
      ares_fpGetBestRoute2(/* The interface to use.  The index is ignored since we are
      ^~~~~~~~~~~~~~~~~~~~
In file included from third_party/cares/cares/ares_init.c:58:0:
third_party/cares/cares/ares_library_init.h:38:26: note: declared here
 extern fpGetBestRoute2_t ares_fpGetBestRoute2;
                          ^~~~~~~~~~~~~~~~~~~~
third_party/cares/cares/ares_init.c:1042:12: error: request for member 'Metric' in something not a structure or union
      || row.Metric == (ULONG)-1
            ^
third_party/cares/cares/ares_init.c:1043:12: error: request for member 'Metric' in something not a structure or union
      || row.Metric > ((ULONG)-1) - interfaceMetric) {
            ^
third_party/cares/cares/ares_init.c:1054:13: error: request for member 'Metric' in something not a structure or union
   return row.Metric + interfaceMetric;
             ^
third_party/cares/cares/ares_init.c: In function 'get_SuffixList_Windows':
third_party/cares/cares/ares_init.c:1430:32: warning: passing argument 1 of 'next_suffix' from incompatible pointer type [-Wincompatible-pointer-types]
       while (len = next_suffix(&pp, len))
                                ^
third_party/cares/cares/ares_init.c:1347:15: note: expected 'const char **' but argument is of type 'char **'
 static size_t next_suffix(const char** list, const size_t advance)
               ^~~~~~~~~~~
third_party/cares/cares/ares_init.c:1430:7: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
       while (len = next_suffix(&pp, len))
       ^~~~~
third_party/cares/cares/ares_init.c: In function 'getBestRouteMetric':
third_party/cares/cares/ares_init.c:1055:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
C:\tools\msys64\mingw32\bin\gcc.exe -mdll -O -Wall -IC:\Python27_32bits\include -IC:\Python27_32bits\PC -c t:\tmp\tmpcumiuy\a.c -o t:\tmp\tmpcumiuy\a.o

jtattermusch · 2017-08-30T06:23:04Z

@y-zeng what exactly do you mean by "GRPC_DNS_RESOLVER=ares to make sure the upgrade will not break the resolver."? We are currently not running those tests by default? That seems like a problem.

I can run tests with GRPC_DNS_RESOLVER=ares locally, but it sounds like a bad idea to only do it as a one-off instead of running them continuously.

jtattermusch · 2017-08-30T09:26:26Z

@y-zeng ok, I see, the GRPC_DNS_RESOLVER=ares tests are being run as part of linux portability suite (and only runs as build_only on PR)

grpc/tools/run_tests/run_tests_matrix.py

Line 245 in f769bf1

test_jobs += _generate_jobs(languages=['c', 'c++'],

.

jtattermusch · 2017-08-30T17:13:43Z

@y-zeng I ran the GRPC_DNS_RESOLVER=ares tests for C and C++ locally (with --use_docker)
and I only got this one failure which seems unrelated:

2017-08-30 15:58:42,392 FAILED: bins/dbg/json_run_localhost --scenarios_json '{"scenarios": [{"name": "cpp_protobuf_async_client_unary_1channel_64wide_128Breq_8MBresp_insecure", "warmup_seconds": 0, "benchmark_seconds": 1, "num_servers": 1, "server_config": {"async_server_threads": 0, "channel_args": [{"str_value": "latency", "name": "grpc.optimization_target"}, {"int_value": 1, "name": "grpc.minimal_stack"}], "security_params": null, "threads_per_cq": 0, "server_type": "ASYNC_SERVER"}, "num_clients": 1, "client_config": {"security_params": null, "channel_args": [{"str_value": "latency", "name": "grpc.optimization_target"}, {"int_value": 1, "name": "grpc.minimal_stack"}], "async_client_threads": 1, "outstanding_rpcs_per_channel": 1, "rpc_type": "UNARY", "payload_config": {"simple_params": {"resp_size": 8388608, "req_size": 128}}, "client_channels": 1, "threads_per_cq": 0, "load_params": {"closed_loop": {}}, "client_type": "ASYNC_CLIENT", "histogram_params": {"max_possible": 60000000000.0, "resolution": 0.01}}}]}' GRPC_POLL_STRATEGY=epoll1

I still need to fix the Python 32-bit artifact build on Windows, but after that, this PR should be good to go.

apolcyn · 2017-08-30T17:31:50Z

I ran this locally with not-checked-in-tests from PR #12210 and can confirm this doesn't break those

jtattermusch · 2017-09-01T09:45:01Z

https://grpc-testing.appspot.com/view/Artifacts/job/gRPC_build_artifacts/776/

grpc-kokoro · 2017-09-09T01:58:46Z

[trickle] No significant performance differences

grpc-kokoro · 2017-09-09T02:27:22Z

[microbenchmarks] Performance differences noted:
Benchmark                                                                                     atm_cas_per_iteration    cpu_time    real_time
--------------------------------------------------------------------------------------------  -----------------------  ----------  -----------
BM_PumpStreamServerToClient<InProcess>/256k                                                   +5%
BM_PumpStreamServerToClient<InProcess>/32k                                                                             -15%        -14%
BM_StreamingPingPong<InProcess, NoOpMutator, NoOpMutator>/32k/2                                                        +12%        +13%
BM_StreamingPingPongMsgs<InProcess, NoOpMutator, NoOpMutator>/256k                                                     +7%         +7%
BM_StreamingPingPongMsgs<InProcess, NoOpMutator, NoOpMutator>/32k                                                      +11%        +10%
BM_StreamingPingPongMsgs<MinInProcess, NoOpMutator, NoOpMutator>/32k                                                   +5%         +5%
BM_StreamingPingPongWithCoalescingApi<InProcess, NoOpMutator, NoOpMutator>/32k/2/0                                     +13%        +13%
BM_StreamingPingPongWithCoalescingApi<MinInProcess, NoOpMutator, NoOpMutator>/32k/1/1                                  +8%         +8%
BM_StreamingPingPongWithCoalescingApi<MinInProcess, NoOpMutator, NoOpMutator>/32k/2/1                                  +7%         +8%
BM_StreamingPingPongWithCoalescingApi<MinInProcessCHTTP2, NoOpMutator, NoOpMutator>/256k/1/1                           +5%         +5%
BM_StreamingPingPongWithCoalescingApi<MinInProcessCHTTP2, NoOpMutator, NoOpMutator>/32k/1/0                            +4%         +4%

grpc-kokoro · 2017-09-09T14:59:37Z

[trickle] No significant performance differences

grpc-kokoro · 2017-09-09T15:35:06Z

[microbenchmarks] No significant performance differences

jtattermusch · 2017-09-09T20:52:49Z

Known failures:

jtattermusch assigned y-zeng Aug 28, 2017

jtattermusch requested a review from y-zeng August 28, 2017 17:52

googlebot added the cla: yes label Aug 28, 2017

apolcyn requested changes Aug 28, 2017

View reviewed changes

jtattermusch force-pushed the cares_bump_1_13 branch from 01d6bb6 to 825ad4d Compare August 29, 2017 16:51

jtattermusch mentioned this pull request Sep 6, 2017

cmake: choose the right protoc executable with gRPC_PROTOBUF_PROVIDER=package #12411

Merged

jtattermusch force-pushed the cares_bump_1_13 branch from 6df9005 to 571c546 Compare September 6, 2017 16:34

jtattermusch and others added 17 commits September 9, 2017 16:32

bump c-ares submodule to 1.13

b0f180a

fix ares 1.13 build on linux

ade414a

cherrypick fixed version of ares_build.h

138eea8

regenerate darwin ares_config.h

7a81686

build using c-ares own cmake

ba2801f

fix macos build

1d93e22

add back manual edits to config_linux/ares_config.h

f1a3e0c

ares using monotonic clock on darwin breaks the build

f0208ac

reintroduce ares_build.h modification from grpc#10719

bb650ca

Update config_freebsd/ares_config.h to FreeBSD 11 ares 1.13

e8e9541

disable c-ares in python on windows x64 and x86

dc57fbd

update mingw gcc on docker file

74a4d5a

avoid __int64 for mingw portability

6dcbc1b

experiment

5fbc577

disable some compiler warnings

d0033d9

remove unnecessary libssl-dev installation

c45d672

tentatively set embed_cares in mingw build as is done in mac/linux build

3a00f79

jtattermusch force-pushed the cares_bump_1_13 branch from c34a344 to 3a00f79 Compare September 9, 2017 14:33

jtattermusch merged commit c7520d0 into grpc:master Sep 10, 2017

BusyJay mentioned this pull request Sep 12, 2017

cmake: specify c-ares source explicitly #11565

Closed

y-zeng mentioned this pull request Sep 22, 2017

Upgrade c-ares to 1.13.0 #11912

Closed

apolcyn mentioned this pull request Oct 11, 2017

Support Ruby build for OpenBSD/FreeBSD #12923

Merged

This was referenced Nov 16, 2017

make buildtests fails on MacOS #13414

Closed

Fix cares MacOS build #13415

Closed

lock bot locked as resolved and limited conversation to collaborators Jan 22, 2019

Conversation

jtattermusch commented Aug 28, 2017

Uh oh!

jtattermusch commented Aug 28, 2017

Uh oh!

y-zeng commented Aug 28, 2017

Uh oh!

grpc-kokoro commented Aug 28, 2017

Uh oh!

grpc-kokoro commented Aug 28, 2017

Uh oh!

apolcyn Aug 28, 2017

Choose a reason for hiding this comment

Uh oh!

jtattermusch Aug 29, 2017

Choose a reason for hiding this comment

Uh oh!

apolcyn Aug 28, 2017

Choose a reason for hiding this comment

Uh oh!

jtattermusch Aug 29, 2017

Choose a reason for hiding this comment

Uh oh!

jtattermusch commented Aug 29, 2017

Uh oh!

apolcyn commented Aug 29, 2017

Uh oh!

jtattermusch commented Aug 29, 2017

Uh oh!

mehrdada commented Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mehrdada commented Aug 29, 2017

Uh oh!

jtattermusch commented Aug 29, 2017 via email

Uh oh!

grpc-kokoro commented Aug 29, 2017

Uh oh!

mehrdada commented Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grpc-kokoro commented Aug 29, 2017

Uh oh!

mehrdada commented Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grpc-kokoro commented Aug 29, 2017

Uh oh!

grpc-kokoro commented Aug 29, 2017

Uh oh!

y-zeng commented Aug 29, 2017

Uh oh!

jtattermusch commented Aug 30, 2017

Uh oh!

jtattermusch commented Aug 30, 2017

Uh oh!

jtattermusch commented Aug 30, 2017

Uh oh!

jtattermusch commented Aug 30, 2017

Uh oh!

apolcyn commented Aug 30, 2017

Uh oh!

jtattermusch commented Sep 1, 2017

Uh oh!

grpc-kokoro commented Sep 9, 2017

Uh oh!

grpc-kokoro commented Sep 9, 2017

Uh oh!

grpc-kokoro commented Sep 9, 2017

Uh oh!

grpc-kokoro commented Sep 9, 2017

Uh oh!

jtattermusch commented Sep 9, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

mehrdada commented Aug 29, 2017 •

edited

Loading

mehrdada commented Aug 29, 2017 •

edited

Loading

mehrdada commented Aug 29, 2017 •

edited

Loading