Skip to content

Commit 05ceb5d

Browse files
[CI][Backport][v1.83.x][Python] Fix PSM Interop xds-v3 Python continuous failure (#42997) (#43239)
Backport of #42997 to v1.83.x. --- The xds v3 Python tests have been failing continuously for the past couple of months with the following error: ``` 2026-07-14 12:40:57,738: DEBUG Invoking GetClientStats RPC to localhost:8079: 2026-07-14 12:51:57,744: ERROR Test case ping_pong failed Traceback (most recent call last): File "/var/local/git/grpc/tools/run_tests/run_xds_tests.py", line 4197, in <module> test_ping_pong(gcp, backend_service, instance_group) File "/var/local/git/grpc/tools/run_tests/run_xds_tests.py", line 959, in test_ping_pong wait_until_all_rpcs_go_to_given_backends( File "/var/local/git/grpc/tools/run_tests/run_xds_tests.py", line 591, in wait_until_all_rpcs_go_to_given_backends _verify_rpcs_to_given_backends( File "/var/local/git/grpc/tools/run_tests/run_xds_tests.py", line 565, in _verify_rpcs_to_given_backends stats = get_client_stats(num_rpcs, timeout_sec) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/local/git/grpc/tools/run_tests/run_xds_tests.py", line 447, in get_client_stats response = stub.GetClientStats( ^^^^^^^^^^^^^^^^^^^^ File "/tmp/tmp.XbXmPgcGvO/lib/python3.11/site-packages/grpc/_channel.py", line 1178, in __call__ return _end_unary_response_blocking(state, call, False, None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/tmp.XbXmPgcGvO/lib/python3.11/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking raise _InactiveRpcError(state) # pytype: disable=not-instantiable ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.DEADLINE_EXCEEDED details = "Deadline Exceeded" debug_error_string = "UNKNOWN:Error received from peer {grpc_status:4, grpc_message:"Deadline Exceeded"}" > ``` When reproducing locally, found that the client logs had the following error: ```sh File "/usr/local/google/home/ssreenithi/.cache/bazel/_bazel_ssreenithi/a79e574dae92a28db81879e89900fb50/execroot/_main/bazel-out/k8-fastbuild/bin/src/python/grpcio_tests/tests_py3_only/interop/xds_interop_client.runfiles/_main/py_xds_protos/envoy/service/status/v3/csds_pb2.py", line 28, in <module> from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2 ModuleNotFoundError: No module named 'google.api' ``` This is a recent problem as mentioned in https://github.com/grpc/grpc/blob/master/tools/remote_build/include/enable_bzlmod.bazelrc#L11-L12 and including this specific config along with the Bazel run commands should help resolve it. Internal ref: b/485429802 Closes #42997 COPYBARA_INTEGRATE_REVIEW=#42997 from sreenithi:fix_xds_v3_python cb7e14b PiperOrigin-RevId: 948909209 Co-authored-by: Sreenithi Sridharan <[email protected]>
1 parent dbc73b8 commit 05ceb5d

1 file changed

Lines changed: 3 additions & 2 deletions

File tree

‎tools/internal_ci/linux/grpc_xds_bazel_python_test_in_docker.sh‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ python -m grpc_tools.protoc \
7676
${HEALTH_PROTO_SOURCE_DIR}/health.proto
7777

7878
cd /var/local/jenkins/grpc/
79-
bazel build //src/python/grpcio_tests/tests_py3_only/interop:xds_interop_client
79+
bazel build --config=python //src/python/grpcio_tests/tests_py3_only/interop:xds_interop_client
8080

8181
# Run legacy ping_pong test. All tests are migrated to
8282
# https://github.com/grpc/psm-interop
@@ -91,4 +91,5 @@ GRPC_VERBOSITY=debug GRPC_TRACE=xds_client,xds_resolver,xds_cluster_manager_lb,c
9191
--gcp_suffix=$(date '+%s') \
9292
--verbose \
9393
${XDS_V3_OPT-} \
94-
--client_cmd='bazel run //src/python/grpcio_tests/tests_py3_only/interop:xds_interop_client -- --server=xds:///{server_uri} --stats_port={stats_port} --qps={qps} {rpcs_to_send} {metadata_to_send}'
94+
--client_cmd='bazel run --config=python //src/python/grpcio_tests/tests_py3_only/interop:xds_interop_client -- --server=xds:///{server_uri} --stats_port={stats_port} --qps={qps} {rpcs_to_send} {metadata_to_send}'
95+

0 commit comments

Comments
 (0)