Hi @twmb. Thank you for this lib.
We have a scenario where we have N different brokers with topics and partitions, all replicated to one aggregated downstream broker node. Our goal is to replicate data from nodeUpstream1 to nodeDownstream, and in the event of nodeUpstream1 going down, we should be able to switch over to nodeUpstream2, nodeUpstream3, and so on in a round-robin fashion. This approach helps us deduplicate data downstream without requiring additional stores.
However, we have encountered few issues with our current implementation. I would like to discuss and confirm if we are following the right approach or if there are any better alternatives. The issues we are facing are as follows:
-
OnBrokerDisconnect() hook gets called multiple times when a broker goes down. It also occurs when idle connections are reaped internally. Is this behavior expected, or should the hook be called only once per broker disconnect event?
-
Close() cannot be called for the consumer group because it issues a fresh new request to LeaveGroup. This request gets stuck when the broker is already down. Is there a recommended way to gracefully handle this situation and ensure the consumer group is closed properly?
-
If autocommit is enabled, there seems to be no way to clean up the associated goroutines. Is there a recommended approach to clean up these goroutines when autocommit is enabled?
-
PurgeTopicsFromClient seems to be the closest cleanup option we found in the library. However, when we round-robin back to the same node, it causes an error (UNKNOWN_MEMBER_ID: The coordinator is not aware of this member.) when trying to commit offsets using kadm.CommitOffsets to resume the consumer group from where it was left off.
implementation: https://github.com/joeirimpan/kaf-relay/blob/fallbacks/consumer.go
Hi @twmb. Thank you for this lib.
We have a scenario where we have N different brokers with topics and partitions, all replicated to one aggregated downstream broker node. Our goal is to replicate data from nodeUpstream1 to nodeDownstream, and in the event of nodeUpstream1 going down, we should be able to switch over to nodeUpstream2, nodeUpstream3, and so on in a round-robin fashion. This approach helps us deduplicate data downstream without requiring additional stores.
However, we have encountered few issues with our current implementation. I would like to discuss and confirm if we are following the right approach or if there are any better alternatives. The issues we are facing are as follows:
OnBrokerDisconnect() hook gets called multiple times when a broker goes down. It also occurs when idle connections are reaped internally. Is this behavior expected, or should the hook be called only once per broker disconnect event?
Close() cannot be called for the consumer group because it issues a fresh new request to LeaveGroup. This request gets stuck when the broker is already down. Is there a recommended way to gracefully handle this situation and ensure the consumer group is closed properly?
If autocommit is enabled, there seems to be no way to clean up the associated goroutines. Is there a recommended approach to clean up these goroutines when autocommit is enabled?
PurgeTopicsFromClient seems to be the closest cleanup option we found in the library. However, when we round-robin back to the same node, it causes an error (UNKNOWN_MEMBER_ID: The coordinator is not aware of this member.) when trying to commit offsets using kadm.CommitOffsets to resume the consumer group from where it was left off.
implementation: https://github.com/joeirimpan/kaf-relay/blob/fallbacks/consumer.go