samples/snippets.py: test_create_instance_with_processing_units flakes with 429 #416
Description
From Samples - Python 3.7 and Samples - Python 3.8 today:
def test_create_instance_with_processing_units(capsys):
processing_units = 500
> snippets.create_instance_with_processing_units(LCI_INSTANCE_ID, processing_units)
snippets_test.py:87:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
snippets.py:81: in create_instance_with_processing_units
operation = instance.create()
../../google/cloud/spanner_v1/instance.py:322: in create
metadata=metadata,
../../google/cloud/spanner_admin_instance_v1/services/instance_admin/client.py:825: in create_instance
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py:145: in __call__
return wrapped_func(*args, **kwargs)
.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
value = None
from_value = <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for...uests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1012616486416'.","grpc_status":8}"
>
> ???
E google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric 'Instance create requests' and limit 'Instance create requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1012616486416'.
<string>:3: ResourceExhausted
The fix would be to wrap the instance.create call with some form of exponential back-off, e.g. using the backoff package:
import backoff
w_backoff = backoff.on_exception(backoff.expo, exceptions.ResourceExhausted)
...
def test_create_instance_with_processing_units(capsys):
processing_units = 500
w_backoff(snippets.create_instance_with_processing_units)(LCI_INSTANCE_ID, processing_units)or the RetryException class from google-cloud-testutils.