Allow more advanced test case support semantic by lidizheng · Pull Request #27373 · grpc/grpc

lidizheng · 2021-09-16T22:54:45Z

After this PR, the framework can skip a test case for a specific language with a specific version range, or something more fancy.

Test runs:

menghanl · 2021-09-16T23:10:29Z

tools/run_tests/xds_k8s_test_driver/framework/xds_flags.py

+TESTING_VERSION = flags.DEFINE_string(
+    "testing_version",
+    default="master",
+    help="The testing gRPC version branch name.")


Add example values to the help string.

Done. Good idea.

menghanl · 2021-09-16T23:12:21Z

tools/run_tests/xds_k8s_test_driver/framework/xds_url_map_testcase.py


        Returns:
-          A tuple of strings contains the supported languages for this test.
+          A tuple of strings contains the supported versions for this test or


Is this doc right?

Awkward, updated.

menghanl · 2021-09-16T23:14:18Z

tools/run_tests/xds_k8s_test_driver/tests/url_map/retry_test.py

-        return 'cpp', 'java', 'go'
+    def is_supported(config: TestConfig) -> bool:
+        if config.client_lang in ['cpp', 'java', 'go']:
+            return config.version_ge('v1.40.x')


Retry should be the same as affinity. go version >= 1.41.x

menghanl · 2021-09-17T00:29:18Z

tools/run_tests/xds_k8s_test_driver/tests/url_map/retry_test.py

-        return 'cpp', 'java', 'go'
+    def is_supported(config: TestConfig) -> bool:
+        if config.client_lang in ['cpp', 'java', 'go']:
+            return config.version_ge('v1.40.x')


And this. Go >= 1.41.x

* Allow more advanced test case support semantic * Fix comments and docs * Correct the supported Golang version in retry test * Make isort happy

* Bump version to v1.41.0-pre1 (#27371) * Bump version to v1.41.0-pre1 * Regenerate projects * [Backport #27373] add testing_version flag (#27385) * Bump version to v1.41.0-pre2 (#27390) * Bump version to v1.41.0-pre2 * Regenerate projects * Core 19: bump core version from 18.0.0 to 19.0.0 (#27394) * Bump core version to 19.0.0 * Regenerate projects * fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398) * Final release: bump up version to 1.41.0 (#27476) * Bump version to 1.41.0 * Regenerate projects * xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580) * Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590) This reverts commit da0c7d6. * Update root pem certs (backport of #27539) (#27619) * Update boringssl to the latest (#27606) (#27625) * Change boringssl branch name * update submodule boringssl-with-bazel with origin/main-with-bazel * update boringssl dependency to main-with-bazel commit SHA * regenerate files * Increment podspec version * generate boringssl prefix headers * Bumping up version to v1.41.1 (#27699) * Bump version to v1.41.1 * Regenerate projects * [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695) Backports sourcing the test driver install script from master. This is a backport of #27389, #27462 and #27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in #27462 where this was missed. * Fix Python Interop (#27620) (#27703) * WIP. Attempt to fix interop * Yapf * Switch Python xDS Example Server to Listen on IPV4 Only (#27679) * Switch to IPV4 * Update to all hosts * Fix rvm ruby install failure (#27769) Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]>

* Bump version to v1.41.0-pre1 (grpc#27371) * Bump version to v1.41.0-pre1 * Regenerate projects * [Backport grpc#27373] add testing_version flag (grpc#27385) * Bump version to v1.41.0-pre2 (grpc#27390) * Bump version to v1.41.0-pre2 * Regenerate projects * Core 19: bump core version from 18.0.0 to 19.0.0 (grpc#27394) * Bump core version to 19.0.0 * Regenerate projects * fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (grpc#27398) * Final release: bump up version to 1.41.0 (grpc#27476) * Bump version to 1.41.0 * Regenerate projects * xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (grpc#27580) * Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (grpc#27580)" (grpc#27590) This reverts commit da0c7d6. * Update root pem certs (backport of grpc#27539) (grpc#27619) * Update boringssl to the latest (grpc#27606) (grpc#27625) * Change boringssl branch name * update submodule boringssl-with-bazel with origin/main-with-bazel * update boringssl dependency to main-with-bazel commit SHA * regenerate files * Increment podspec version * generate boringssl prefix headers * Bumping up version to v1.41.1 (grpc#27699) * Bump version to v1.41.1 * Regenerate projects * [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (grpc#27695) Backports sourcing the test driver install script from master. This is a backport of grpc#27389, grpc#27462 and grpc#27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (grpc#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (grpc#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in grpc#27462 where this was missed. * Fix Python Interop (grpc#27620) (grpc#27703) * WIP. Attempt to fix interop * Yapf * Switch Python xDS Example Server to Listen on IPV4 Only (grpc#27679) * Switch to IPV4 * Update to all hosts * Fix rvm ruby install failure (grpc#27769) Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]>

* Upgrade benchmark to 1.6.0 and remove hacks. Details: - GRPC currently uses an old version of benchmark (from Sept 2020). It should probably upgrade because downstream, in google3, everyone is already using 1.6.0) - Removed the hack added in PR/27629 to allow benchmarks in GRPC to continue to work with both pre-1.6.0 and 1.6.0 benchmarks. (This was needed to allow importing benchmarks 1.6.0 into google3 without breaking GRPC) * fix typo * update third_party/benchmark and check_submodules.sh * Upmerge from v1.41.x (#27821) * Bump version to v1.41.0-pre1 (#27371) * Bump version to v1.41.0-pre1 * Regenerate projects * [Backport #27373] add testing_version flag (#27385) * Bump version to v1.41.0-pre2 (#27390) * Bump version to v1.41.0-pre2 * Regenerate projects * Core 19: bump core version from 18.0.0 to 19.0.0 (#27394) * Bump core version to 19.0.0 * Regenerate projects * fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398) * Final release: bump up version to 1.41.0 (#27476) * Bump version to 1.41.0 * Regenerate projects * xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580) * Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590) This reverts commit da0c7d6. * Update root pem certs (backport of #27539) (#27619) * Update boringssl to the latest (#27606) (#27625) * Change boringssl branch name * update submodule boringssl-with-bazel with origin/main-with-bazel * update boringssl dependency to main-with-bazel commit SHA * regenerate files * Increment podspec version * generate boringssl prefix headers * Bumping up version to v1.41.1 (#27699) * Bump version to v1.41.1 * Regenerate projects * [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695) Backports sourcing the test driver install script from master. This is a backport of #27389, #27462 and #27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in #27462 where this was missed. * Fix Python Interop (#27620) (#27703) * WIP. Attempt to fix interop * Yapf * Switch Python xDS Example Server to Listen on IPV4 Only (#27679) * Switch to IPV4 * Update to all hosts * Fix rvm ruby install failure (#27769) Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]> * added perf_counters.cc manually since the script didn't work Co-authored-by: Mark D. Roth <[email protected]> Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]>

* Allow more advanced test case support semantic * Fix comments and docs * Correct the supported Golang version in retry test * Make isort happy

Allow more advanced test case support semantic

6decaa8

lidizheng added area/test release notes: no Indicates if PR should not be in release notes labels Sep 16, 2021

lidizheng requested a review from menghanl September 16, 2021 22:54

menghanl suggested changes Sep 16, 2021

View reviewed changes

Fix comments and docs

0b0ad1c

lidizheng marked this pull request as ready for review September 16, 2021 23:23

lidizheng assigned menghanl Sep 16, 2021

menghanl reviewed Sep 17, 2021

View reviewed changes

lidizheng added 2 commits September 16, 2021 17:42

Correct the supported Golang version in retry test

8dac394

Make isort happy

c8279e3

menghanl approved these changes Sep 17, 2021

View reviewed changes

lidizheng merged commit f99c193 into grpc:master Sep 17, 2021

lidizheng added a commit to lidizheng/grpc that referenced this pull request Sep 17, 2021

[Backport grpc#27373] add testing_version flag

082099f

lidizheng mentioned this pull request Sep 17, 2021

[Backport #27373] add testing_version flag #27385

Merged

lidizheng added a commit to lidizheng/grpc that referenced this pull request Sep 17, 2021

[Backport grpc#27373] add testing_version flag

58385cd

This was referenced Sep 17, 2021

[Backport #27373] add testing_version flag #27386

Merged

[Backport grpc#27373] add testing_version flag grpc/grpc-go#4782

Merged

copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label Sep 17, 2021

lidizheng added a commit that referenced this pull request Sep 17, 2021

[Backport #27373] add testing_version flag (#27385)

b31c03b

lidizheng added a commit to lidizheng/grpc-java that referenced this pull request Sep 17, 2021

[Backport grpc/grpc#27373] add testing_version flag

d20ad2a

lidizheng added a commit that referenced this pull request Sep 17, 2021

[Backport #27373] add testing_version flag (#27386)

07d9037

This was referenced Sep 17, 2021

Add testing_version flag grpc/grpc-java#8534

Merged

[Backport grpc#27373] add testing_version flag grpc/grpc-go#4783

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow more advanced test case support semantic#27373

Allow more advanced test case support semantic#27373
lidizheng merged 4 commits intogrpc:masterfrom
lidizheng:version-control

lidizheng commented Sep 16, 2021 •

edited

Loading

Uh oh!

menghanl Sep 16, 2021

Uh oh!

lidizheng Sep 16, 2021

Uh oh!

menghanl Sep 16, 2021

Uh oh!

lidizheng Sep 16, 2021

Uh oh!

menghanl Sep 16, 2021

Uh oh!

lidizheng Sep 16, 2021

Uh oh!

menghanl Sep 17, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lidizheng commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

menghanl Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

lidizheng Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

menghanl Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

lidizheng Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

menghanl Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

lidizheng Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

menghanl Sep 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lidizheng commented Sep 16, 2021 •

edited

Loading

menghanl Sep 17, 2021 •

edited

Loading