Details
-
Improvement
-
Status: Resolved
-
P3
-
Resolution: Duplicate
-
2.23.0, 2.24.0
-
None
Description
Hi, I am trying to test ordering_key (new beta feature of Google Pubsub). However, I seem to hit the wall here. Google Protobuff standard message has been described here
https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage . So, it seems that Apache Beam should have no issue to passing the new field to Pubsub if the format of the message is correct. It does not seem to be a case here. As far as, I have tried to go through the code, I believe that I would only need to change
// def _to_proto_str(self): msg = pubsub.types.pubsub_pb2.PubsubMessage() msg.data = self.data for key, value in iteritems(self.attributes): msg.attributes[key] = value msg.ordering_key = self.ordering_key return msg.SerializeToString()
And then set with_attributes=true. So, WriteToPubSub would allow me to send out a protobuff message to pubsub. However, at some point, the information (ordering_key) is tripped, although, I could set custom attributes. I guess it may be the dataflow_runner code, but it really difficult to navigate around the code in that area. I have gone as far as create a new version of write to WriteToPubSub, but it does not help. The field is still missing on the message when it hits Google Pubsub Topic.
Could anyone point me to the right direction please?
After digging a little bit deeper, I think the root reason probably because, Apache Beam depend on an old version of Pubsub (1.7.0) . Probably for support of Python 2. I am just not sure for how long? It seems that the support for Python will end after 2.24.0. However, the dependency for older version of google-cloud-pubsub is still there for 2.25.0 (
| 'google-cloud-pubsub>=0.39.0,<2', |
)
Attachments
Issue Links
- duplicates
-
BEAM-7819 PubsubMessage message parsing is lacking non-attribute fields
-
- Triage Needed
-