Add a grpclb-in-DNS interop test suite by apolcyn · Pull Request #16727 · grpc/grpc

apolcyn · 2018-09-27T21:55:12Z

This PR adds a test suite that can bring up fake and local grpclb servers, backends, fallback servers, and a controlled DNS server. It currently runs grpc interop clients against the servers. The test driver takes a list of scenarios that tell it the number of "backends", "fallback" servers, and "grpclb" servers to bring up, what credentials to listen on, and certain behaviors to provide. For each scenario, it brings up all of the fake servers in docker containers, wires up the DNS records of the DNS server to point to the IP addresses of each docker container, and then runs the clients. Note also this currently only tests the native C++, Java, and Go stacks, but it could be extended.

General things that this PR does:

Create tools/run_tests/run_grpclb_interop_tests.py, which is the main test driver that basically follows the description above.
Add tools/run_tests/lb_interop_tests/gen_build.yaml, which generates a list of "test scenario" JSON configs into tools/run_tests/generated/lb_interop_test_scenarios.json
Add a new docker file under tools/run_tests/dockerfile/lb_interop_fake_servers/ which has the dependencies needed to build the fake backend, fallback, and balancer server, and to run the local python DNS server (the balancer is from Provide a fake grpclb balancer server that can be used for testing grpc-go#2338)
Make a couple of tweaks to existing interop client builds to add logging and compile faster

See: https://source.cloud.google.com/results/invocations/f93c0e2c-975d-48af-8464-1d86488ad505/targets for a passing run

These new tests exposed a few bugs in a few scenarios:

.... and so those scenarios are currently being skipped.

Remaining TODO is to use a non-experimental kokoro config file, otherwise I think this is ready for review

cc @jtattermusch can you please review for general sanity on adding this new suite?

cc @zhangkun83 can you please review, in particular, for sanity of the devised scenarios here? Scenarios are in tools/run_tests/lb_interop_tests/gen_build_yaml.py . . . happy to also talk offline here.

grpc-testing · 2018-10-10T23:36:23Z

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,965,918      Total (=)      1,965,918

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
10,891,799      Total (<)     10,891,800

 No significant differences in binary sizes

grpc-testing · 2018-10-10T23:43:29Z

[trickle] No significant performance differences

grpc-testing · 2018-10-10T23:49:37Z

[microbenchmarks] No significant performance differences

grpc-testing · 2018-10-10T23:59:20Z

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-10-11T00:05:29Z

[trickle] No significant performance differences

grpc-testing · 2018-10-11T00:13:18Z

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,965,918      Total (=)      1,965,918

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
10,891,799      Total (>)     10,891,788

 No significant differences in binary sizes

grpc-testing · 2018-10-11T00:26:16Z

[microbenchmarks] No significant performance differences

grpc-testing · 2018-10-11T00:38:07Z

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,965,918      Total (=)      1,965,918

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
10,891,796      Total (<)     10,891,797

 No significant differences in binary sizes

grpc-testing · 2018-10-11T00:51:08Z

[microbenchmarks] No significant performance differences

grpc-testing · 2018-10-11T01:10:02Z

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-10-11T01:16:28Z

[trickle] No significant performance differences

grpc-testing · 2018-10-11T01:49:27Z

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,965,918      Total (=)      1,965,918

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
10,891,798      Total (>)     10,891,795

 No significant differences in binary sizes

grpc-testing · 2018-10-11T02:01:20Z

[microbenchmarks] No significant performance differences

grpc-testing · 2018-10-12T19:21:19Z

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-10-12T19:28:28Z

[trickle] No significant performance differences

grpc-testing · 2018-10-12T20:01:17Z

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,965,922      Total (=)      1,965,922

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
10,909,005      Total (>)     10,909,003

 No significant differences in binary sizes

grpc-testing · 2018-10-12T20:12:53Z

[microbenchmarks] No significant performance differences

grpc-testing · 2018-10-23T07:05:32Z

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-10-23T07:13:22Z

[trickle] No significant performance differences

grpc-testing · 2018-10-23T07:44:25Z

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,997,294      Total (=)      1,997,294

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
11,021,760      Total (>)     11,021,756

 No significant differences in binary sizes

grpc-testing · 2018-10-23T07:57:29Z

Corrupt JSON data (indicates timeout or crash): 
    bm_call_create.BM_IsolatedFilter_ClientChannelFilter_NoOp_.counters.new: 10
    bm_call_create.BM_IsolatedFilter_ClientChannelFilter_NoOp_.counters.old: 10


[microbenchmarks] No significant performance differences

apolcyn · 2018-10-23T18:31:17Z

python test failures: #15308

apolcyn · 2018-10-23T20:06:17Z

merging, FTR this test will not be running per-PR, at least for now. It will be running continuously on the master branch but will need to be manually monitored for now (if possibly, this can later be promoted to run per-PR, possibly).

apolcyn requested review from jtattermusch, mehrdada and nicolasnoble as code owners September 27, 2018 21:55

This was referenced Sep 27, 2018

grpclb-in-DNS clients fall back too early when the LB server ends the stream right after sending a server list grpc/grpc-java#4887

Closed

Provide a fake grpclb balancer server that can be used for testing grpc/grpc-go#2338

Merged

apolcyn force-pushed the grpclb_interop_master branch 2 times, most recently from e315479 to c1320e4 Compare October 3, 2018 04:32

apolcyn added release notes: no Indicates if PR should not be in release notes lang/all wrapped languages labels Oct 3, 2018

This was referenced Oct 3, 2018

Edge case in grpclb: empty server list handled differently between Go/Java/C++ grpc/grpc-go#2347

Closed

grpclb with TLS sends a malformed SNI host name to the balancer grpc/grpc-java#4912

Closed

grpc deleted a comment from grpc-testing Oct 5, 2018

apolcyn changed the title ~~WIP: grpclb-in-DNS interop test suite~~ grpclb-in-DNS interop test suite Oct 5, 2018

apolcyn changed the title ~~grpclb-in-DNS interop test suite~~ Add a grpclb-in-DNS interop test suite Oct 5, 2018

apolcyn assigned markdroth and jtattermusch Oct 5, 2018

grpc deleted a comment from grpc-testing Oct 5, 2018

apolcyn force-pushed the grpclb_interop_master branch from ec8ddd2 to c175f2f Compare October 10, 2018 23:55

apolcyn force-pushed the grpclb_interop_master branch from c175f2f to 454a667 Compare October 11, 2018 01:06

zhangkun83 approved these changes Oct 12, 2018

View reviewed changes

Add a grpclb-in-DNS interop test suite

e52d87b

apolcyn force-pushed the grpclb_interop_master branch from 0f2fdc6 to e52d87b Compare October 23, 2018 07:00

apolcyn merged commit 2f92bde into grpc:master Oct 23, 2018

lock bot locked as resolved and limited conversation to collaborators Jan 21, 2019

Conversation

apolcyn commented Sep 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grpc-testing commented Oct 10, 2018

Uh oh!

grpc-testing commented Oct 10, 2018

Uh oh!

grpc-testing commented Oct 10, 2018

Uh oh!

grpc-testing commented Oct 10, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 11, 2018

Uh oh!

grpc-testing commented Oct 12, 2018

Uh oh!

grpc-testing commented Oct 12, 2018

Uh oh!

grpc-testing commented Oct 12, 2018

Uh oh!

grpc-testing commented Oct 12, 2018

Uh oh!

grpc-testing commented Oct 23, 2018

Uh oh!

grpc-testing commented Oct 23, 2018

Uh oh!

grpc-testing commented Oct 23, 2018

Uh oh!

grpc-testing commented Oct 23, 2018

Uh oh!

apolcyn commented Oct 23, 2018

Uh oh!

apolcyn commented Oct 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

apolcyn commented Sep 27, 2018 •

edited

Loading