-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Proposal
I would like to have an option to limit number of replicas to the number of partitions with lag > 0 only.
Use-Case
When using Kafka key, the amount of records per key (partition) not always could be distributed evenly, and in some cases only some partitions have a lag while most of the partitions have no lag.
A special case is when only one partition has a lag which could easily scale to the max allowed number of replicas while all the consumers will be idle except one. In such case there is no need to scale over one replica.
In a general case, I would like to limit scaling to min(max allowed number of replicas as of today, number of partitions with lag > 0). Even though, the partitions could be distributed unevenly between consumers, sometimes it is more efficient way to allocate resources rather than scaling idle consumers.
Anything else?
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status