Final cost-based autoscaler knobs adjusments#19045
Conversation
|
Thanks for the PR, @Fly-Style ! Could you please share details on why the value 0.05 is better than 0.2? In general, for a new supervisor, what would be the steps to reach the optimal values for |
|
@kfaraz I updated PR description. Regarding the:
Technically, yes, but such a decision does sound logical, at least to my taste. The core idea is behind the juggling of the normalized cost fuction weights. Well, maybe my mind is not so agile 😅, and that may make sense. |
kfaraz
left a comment
kfaraz
left a comment
There was a problem hiding this comment.
Approved.
@Fly-Style , could you please share some thoughts on:
In general, for a new supervisor, what would be the steps to reach the optimal values for lagWeight and idleWeight?
It would be very helpful to allow users to experiment with and tune the new autoscaler.
|
@kfaraz sure. |
3 participants
This patch makes autoscaling during high lag even smoother, by reducing logarihmic mutliplier used a knob for that.
Old multiplier recommended to scale up too much tasks.
The result might be seen in the plot under the hood.
Details
Plot conditions were taken from real scenario: ~5M lag, 100k processing rate, 48 partitions (10 and 24 partitions are present here for differential analysis purposes).
This PR has: