PubSub Publisher 504 Deadline Exceeded when sending bulk messages #72
Description
Environment details
- OS type and version:
PRETTY_NAME="Debian GNU/Linux 10 (buster)" - Python version:
python --versionpython3.7 - pip version:
pip --version20.0.2 google-cloud-pubsubversion:pip show google-cloud-pubsub1.4.2
Steps to reproduce
- Send large number of messages to a PubSub Topic fails when I try to send 5000 without pausing and waiting, for example.
- Check logs
Code example
# I have also tried this using the callback and hit the same error. This is what I'm doing now though.
iter_size = 5000
iter_count = 0
futures = []
for row in rows:
data = json.dumps(row, default=lambda x: str(x)).encode('utf-8')
future = publish_client.publish(topic, data, origin='xxx')
futures.append(future)
published_count += 1
iter_count += 1
if iter_count >= iter_size:
[f.result() for f in futures]
futures = []
iter_count = 0
count += 1
[f.result() for f in futures]Stack trace
[2020-04-14 14:19:23,788] {{thread.py:264}} ERROR - Failed to publish 141 messages.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1586873963.056617563","description":"Error received from peer ipv4:108.177.122.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
Other Details:
I've found a similar issue here
googleapis/google-cloud-java#3867
Which references to this issue
googleapis/google-cloud-java#3003
Then
googleapis/java-pubsub#31
This will normally be running on GKE, so I understand me running it locally may have some variation of behavior due to network speed. However, we would prefer to avoid having this issue either way.
Is this something that is planned/intended to be supported or should I implement auto-throttling?