xds-k8s tests: Use test driver from master branch#27462
xds-k8s tests: Use test driver from master branch#27462sergiitk merged 38 commits intogrpc:masterfrom
Conversation
These changes allow the language specific test scripts to first clone the core repo to acquire thishis script before sourcing it.
These changes allow the language specific test scripts to first clone the core repo to acquire thishis script before sourcing it.
sergiitk
left a comment
There was a problem hiding this comment.
This looks great. I've noticed you missed url map tests somehow. There's another one like this for python too. Could you double check all files that source grpc_xds_k8s_install_test_driver.sh?
| GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-central1-a" | ||
| GKE_CLUSTER_ZONE="us-central1-a" | ||
| SECONDARY_GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-west1-b" | ||
| SECONDARY_GKE_CLUSTER_ZONE="us-west1-b" |
There was a problem hiding this comment.
If we need to recreate this clusters, these lines would have to be backported too all previous branches.
Instead, we could add a method to grpc_xds_k8s_install_test_driver.sh, that will export same variables based on cluster type. F.e.:
| GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-central1-a" | |
| GKE_CLUSTER_ZONE="us-central1-a" | |
| SECONDARY_GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-west1-b" | |
| SECONDARY_GKE_CLUSTER_ZONE="us-west1-b" | |
| activate_gke_cluster "psm-lb" | |
| activate_gke_cluster_secondary "psm-lb" |
and pseudocode for activate_gke_cluster:
switch cluster_type:
case "psm-lb":
readonly GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-central1-a"
readonly GKE_CLUSTER_ZONE="us-central1-a"
case "psm-security":
readonly GKE_CLUSTER_NAME="interop-test-psm-sec-v2-us-central1-a"
readonly GKE_CLUSTER_ZONE="us-central1-a"
...
and activate_gke_cluster_secondary:
switch cluster_type:
case "psm-lb":
readonly SECONDARY_GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-west1-b"
readonly SECONDARY_GKE_CLUSTER_ZONE="us-west1-b"
...
There was a problem hiding this comment.
+1. It would be great if we can update the cluster without backports.
There was a problem hiding this comment.
Agreed and the suggestion how to do it makes sense, went with that approach.
Enable cluster instead by calling a method in the test driver.
Enable cluster instead by calling a method in the test driver.
lidizheng
left a comment
There was a problem hiding this comment.
The overall logic looks good. Thanks for doing this work. I have some comments about choosing GKE cluster, and some discussion.
| GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-central1-a" | ||
| GKE_CLUSTER_ZONE="us-central1-a" | ||
| SECONDARY_GKE_CLUSTER_NAME="interop-test-psm-lb-v1-us-west1-b" | ||
| SECONDARY_GKE_CLUSTER_ZONE="us-west1-b" |
There was a problem hiding this comment.
+1. It would be great if we can update the cluster without backports.
| # Clone the test driver from the master branch using an external script. | ||
| # shellcheck source=tools/internal_ci/linux/grpc_xds_k8s_clone_driver_repo.sh | ||
| source "${script_dir}/grpc_xds_k8s_clone_driver_repo.sh" | ||
| clone_test_driver |
There was a problem hiding this comment.
optional: We might be able to do something fancy like
source /dev/stdin <<< "$(curl https://raw.githubusercontent.com/grpc/grpc/master/tools/internal_ci/linux/grpc_xds_k8s_install_test_driver.sh)"The URL of the test driver script can be stitched by environment vars for repo owner and branch, but that's the basic idea. This might be a smaller scale change to reach our end goal.
There was a problem hiding this comment.
That was my first thought as well, but Sergii had done some initial work with the clone approach so I went with that. Performance is not really a concern here and it doesn't hurt to have the whole master branch available.
There was a problem hiding this comment.
I think by going with a simple curl, we can save managing an extra file in java/go/node repos. This will be a significant improvement for reducing coupling between repos. E.g., in the worst scenario, if there is a small improvement we need to do in the clone logic, we will need to patch all release branches across repos. I agree performance is not the main issue here, my suggestion is more about reducing future maintenance toil.
@sergiitk What do you think about using curl to download the xds_k8s utility functions?
There was a problem hiding this comment.
Yes, my initial work was from some months ago, and I don't fully remember the reasoning for git clone vs curl. I remember talking to @ejona86 about it, and we concluded on cloning. Now, as it introduces an extra file to manage, I like the idea the curl oneliner.
There was a problem hiding this comment.
I generally prefer downloading the tar.gz/zip, as then I don't need a git dependency. Although the core repo tends to need submodules for lots of tasks, so it normally isn't an option for grpc/grpc, and a lot of the test tooling uses git directly, so nothing is gained there.
I don't remember the precise reasoning, but I think it was probably already using a git clone and I may have said something like, "if you are doing --depth=1, then you are already doing a lot better than most of the other scripts when they are written." And then we just left it at "good enough."
Grabbing just the one file we need sounds great, assuming it won't ever want to source other files.
There was a problem hiding this comment.
I initially liked the clone approach as it would allow us to potentially use other files in the future. But I can't really think what files those would be, so I'll switch to curl and eliminate the new clone script.
There was a problem hiding this comment.
I agree that Git clone is still necessary, but it will happen after the downloaded entrypoint script is sourced. It would look like following:
# shellcheck source=buildscripts/kokoro/xds-k8s-install-test-driver.sh
- source "${script_dir}/xds-k8s-install-test-driver.sh"
+ source /dev/stdin <<< "$(curl https://raw.githubusercontent.com/grpc/grpc/master/tools/internal_ci/linux/grpc_xds_k8s_install_test_driver.sh)"
set -x
if [[ -n "${KOKORO_ARTIFACTS_DIR}" ]]; then
kokoro_setup_test_driver "${GITHUB_REPOSITORY_NAME}"
else
local_setup_test_driver "${script_dir}"
fiThere was a problem hiding this comment.
The install script is now sourced and I removed the clone script, take another look please.
lidizheng
left a comment
There was a problem hiding this comment.
LGTM. fyi, you can run two more ad-hoc builds on Kokoro to valid your code:
- https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod:grpc%2Fcore%2Fmaster%2Flinux%2Fgrpc_xds_url_map
- https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod:grpc%2Fcore%2Fmaster%2Flinux%2Fgrpc_xds_k8s_lb
This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in #27462 where this was missed.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in grpc#27462 where this was missed.
…#27694) * Add missing quotation marks. (#27389) These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
…#27695) Backports sourcing the test driver install script from master. This is a backport of #27389, #27462 and #27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in #27462 where this was missed.
…#27681) * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
…#27691) * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
…#27692) * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
* Bump version to v1.41.0-pre1 (#27371) * Bump version to v1.41.0-pre1 * Regenerate projects * [Backport #27373] add testing_version flag (#27385) * Bump version to v1.41.0-pre2 (#27390) * Bump version to v1.41.0-pre2 * Regenerate projects * Core 19: bump core version from 18.0.0 to 19.0.0 (#27394) * Bump core version to 19.0.0 * Regenerate projects * fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398) * Final release: bump up version to 1.41.0 (#27476) * Bump version to 1.41.0 * Regenerate projects * xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580) * Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590) This reverts commit da0c7d6. * Update root pem certs (backport of #27539) (#27619) * Update boringssl to the latest (#27606) (#27625) * Change boringssl branch name * update submodule boringssl-with-bazel with origin/main-with-bazel * update boringssl dependency to main-with-bazel commit SHA * regenerate files * Increment podspec version * generate boringssl prefix headers * Bumping up version to v1.41.1 (#27699) * Bump version to v1.41.1 * Regenerate projects * [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695) Backports sourcing the test driver install script from master. This is a backport of #27389, #27462 and #27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in #27462 where this was missed. * Fix Python Interop (#27620) (#27703) * WIP. Attempt to fix interop * Yapf * Switch Python xDS Example Server to Listen on IPV4 Only (#27679) * Switch to IPV4 * Update to all hosts * Fix rvm ruby install failure (#27769) Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]>
* Bump version to v1.41.0-pre1 (grpc#27371) * Bump version to v1.41.0-pre1 * Regenerate projects * [Backport grpc#27373] add testing_version flag (grpc#27385) * Bump version to v1.41.0-pre2 (grpc#27390) * Bump version to v1.41.0-pre2 * Regenerate projects * Core 19: bump core version from 18.0.0 to 19.0.0 (grpc#27394) * Bump core version to 19.0.0 * Regenerate projects * fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (grpc#27398) * Final release: bump up version to 1.41.0 (grpc#27476) * Bump version to 1.41.0 * Regenerate projects * xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (grpc#27580) * Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (grpc#27580)" (grpc#27590) This reverts commit da0c7d6. * Update root pem certs (backport of grpc#27539) (grpc#27619) * Update boringssl to the latest (grpc#27606) (grpc#27625) * Change boringssl branch name * update submodule boringssl-with-bazel with origin/main-with-bazel * update boringssl dependency to main-with-bazel commit SHA * regenerate files * Increment podspec version * generate boringssl prefix headers * Bumping up version to v1.41.1 (grpc#27699) * Bump version to v1.41.1 * Regenerate projects * [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (grpc#27695) Backports sourcing the test driver install script from master. This is a backport of grpc#27389, grpc#27462 and grpc#27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (grpc#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (grpc#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in grpc#27462 where this was missed. * Fix Python Interop (grpc#27620) (grpc#27703) * WIP. Attempt to fix interop * Yapf * Switch Python xDS Example Server to Listen on IPV4 Only (grpc#27679) * Switch to IPV4 * Update to all hosts * Fix rvm ruby install failure (grpc#27769) Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]>
* Upgrade benchmark to 1.6.0 and remove hacks. Details: - GRPC currently uses an old version of benchmark (from Sept 2020). It should probably upgrade because downstream, in google3, everyone is already using 1.6.0) - Removed the hack added in PR/27629 to allow benchmarks in GRPC to continue to work with both pre-1.6.0 and 1.6.0 benchmarks. (This was needed to allow importing benchmarks 1.6.0 into google3 without breaking GRPC) * fix typo * update third_party/benchmark and check_submodules.sh * Upmerge from v1.41.x (#27821) * Bump version to v1.41.0-pre1 (#27371) * Bump version to v1.41.0-pre1 * Regenerate projects * [Backport #27373] add testing_version flag (#27385) * Bump version to v1.41.0-pre2 (#27390) * Bump version to v1.41.0-pre2 * Regenerate projects * Core 19: bump core version from 18.0.0 to 19.0.0 (#27394) * Bump core version to 19.0.0 * Regenerate projects * fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398) * Final release: bump up version to 1.41.0 (#27476) * Bump version to 1.41.0 * Regenerate projects * xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580) * Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590) This reverts commit da0c7d6. * Update root pem certs (backport of #27539) (#27619) * Update boringssl to the latest (#27606) (#27625) * Change boringssl branch name * update submodule boringssl-with-bazel with origin/main-with-bazel * update boringssl dependency to main-with-bazel commit SHA * regenerate files * Increment podspec version * generate boringssl prefix headers * Bumping up version to v1.41.1 (#27699) * Bump version to v1.41.1 * Regenerate projects * [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695) Backports sourcing the test driver install script from master. This is a backport of #27389, #27462 and #27658: * Add missing quatation marks. These were missed when creating the Python virtual env. * xds-k8s tests: Use test driver from master branch (#27462) Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch. This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches. All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch. * xds_k8s tests: Fix xlang install script sourcing. (#27658) This change sources the test driver install script correctly for the xlang tests. This fixes a mistake in #27462 where this was missed. * Fix Python Interop (#27620) (#27703) * WIP. Attempt to fix interop * Yapf * Switch Python xDS Example Server to Listen on IPV4 Only (#27679) * Switch to IPV4 * Update to all hosts * Fix rvm ruby install failure (#27769) Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]> * added perf_counters.cc manually since the script didn't work Co-authored-by: Mark D. Roth <[email protected]> Co-authored-by: donnadionne <[email protected]> Co-authored-by: Lidi Zheng <[email protected]> Co-authored-by: Jan Tattermusch <[email protected]> Co-authored-by: sanjaypujare <[email protected]> Co-authored-by: Sergii Tkachenko <[email protected]> Co-authored-by: Esun Kim <[email protected]> Co-authored-by: Terry Wilson <[email protected]> Co-authored-by: Richard Belleville <[email protected]>
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch.
This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches.
All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
Successful test runs (with the install script URL pointed to the feature branch):
grpc_xds_k8s.sh: https://fusion2.corp.google.com/invocations/eb7e7c6e-a133-47d1-8b4d-bab85954b833/targets
grpc_xds_k8s_lb.sh: https://fusion2.corp.google.com/invocations/adb0163c-f6a3-4974-b5a7-56035c81ef8f/targets
grpc_xds_url_map.sh: https://fusion2.corp.google.com/invocations/a3db9a8a-ae25-492c-9ed2-92bbb5f9c781/targets
@sergiitk