Skip to content

Kafka scaler: limit number of replicas to the number of partitions with lag. #3997

@alexakra

Description

@alexakra

Proposal

I would like to have an option to limit number of replicas to the number of partitions with lag > 0 only.

Use-Case

When using Kafka key, the amount of records per key (partition) not always could be distributed evenly, and in some cases only some partitions have a lag while most of the partitions have no lag.
A special case is when only one partition has a lag which could easily scale to the max allowed number of replicas while all the consumers will be idle except one. In such case there is no need to scale over one replica.
In a general case, I would like to limit scaling to min(max allowed number of replicas as of today, number of partitions with lag > 0). Even though, the partitions could be distributed unevenly between consumers, sometimes it is more efficient way to allocate resources rather than scaling idle consumers.

Anything else?

No response

Metadata

Metadata

Assignees

Labels

feature-requestAll issues for new features that have not been committed tohelp wantedLooking for support from communityneeds-discussion

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions