Report
In our organisation, we have large clusters where workload autoscaling is being managed by KEDA where 350+ scaledobjects are present.
Lately, we have observed that the time taken for a change to the scaledobject to reflect in its child HPA takes around 10 min which is hampering the autoscaler responsiveness.
For example, if we update the scaledobject.spec.maxReplicaCount , it takes around 10 min to reflect in the hpa.spec.maxReplicas.
After debugging and analyzing the KEDA operator pod logs and the scaledobject_controller.go , we identified that ScaledObjects are continuously being reconciled when the child HPA is updated. This includes even upon the status updates to the HPA by other controllers such as HPA controller which is part of the kube controller manager. HPA controller keeps updating the hpa status with information such as conditions and the controller perceived resource metrics
(.status.currentMetrics). The volume of such updates is too high and this is causing unnecessary and redundant reconciles on the scaledobject further delaying any genuine updates.
Expected Behavior
Only changes to the hpa spec or labels/annotations should trigger scaledobject reconcilation and ignore any status updates.
Actual Behavior
In this controller initialisation part, any updates to the HPA are triggering scaledobject reconciles which can be redundant and unnecessary.
Steps to Reproduce the Problem
Have a cluster with 100+ scaledobjects being managed by KEDA. The time taken to update the HPA upon scaledobject spec updates is in the order of minutes.
Logs from KEDA operator
We are seeing multiple logs like this which are triggered by multiple HPA status updates (.status.currentMetrics).
2023-12-08T13:44:04.647+0530 INFO controllers.ScaledObject Reconciling ScaledObject {"ScaledObject.Namespace": "xx", "ScaledObject.Name": "yy"}
2023-12-08T13:51:08.931+0530 INFO controllers.ScaledObject Reconciling ScaledObject {"ScaledObject.Namespace": "xx", "ScaledObject.Name": "yy"}
KEDA Version
2.12.0
Kubernetes Version
1.26
Platform
Other
Scaler Details
CPU
Anything else?
This issue is reproducible in the older and the latest KEDA versions. Also, tweaking KEDA_SCALEDOBJECT_CTRL_MAX_RECONCILES will not completely resolve this issue as the redundant updates will still continue to be processed.
Report
In our organisation, we have large clusters where workload autoscaling is being managed by KEDA where 350+ scaledobjects are present.
Lately, we have observed that the time taken for a change to the scaledobject to reflect in its child HPA takes around 10 min which is hampering the autoscaler responsiveness.
For example, if we update the scaledobject.spec.maxReplicaCount , it takes around 10 min to reflect in the hpa.spec.maxReplicas.
After debugging and analyzing the KEDA operator pod logs and the scaledobject_controller.go , we identified that ScaledObjects are continuously being reconciled when the child HPA is updated. This includes even upon the status updates to the HPA by other controllers such as HPA controller which is part of the kube controller manager. HPA controller keeps updating the hpa status with information such as conditions and the controller perceived resource metrics
(.status.currentMetrics). The volume of such updates is too high and this is causing unnecessary and redundant reconciles on the scaledobject further delaying any genuine updates.
Expected Behavior
Only changes to the hpa spec or labels/annotations should trigger scaledobject reconcilation and ignore any status updates.
Actual Behavior
In this controller initialisation part, any updates to the HPA are triggering scaledobject reconciles which can be redundant and unnecessary.
Steps to Reproduce the Problem
Have a cluster with 100+ scaledobjects being managed by KEDA. The time taken to update the HPA upon scaledobject spec updates is in the order of minutes.
Logs from KEDA operator
We are seeing multiple logs like this which are triggered by multiple HPA status updates (.status.currentMetrics).
2023-12-08T13:44:04.647+0530 INFO controllers.ScaledObject Reconciling ScaledObject {"ScaledObject.Namespace": "xx", "ScaledObject.Name": "yy"}
2023-12-08T13:51:08.931+0530 INFO controllers.ScaledObject Reconciling ScaledObject {"ScaledObject.Namespace": "xx", "ScaledObject.Name": "yy"}
KEDA Version
2.12.0
Kubernetes Version
1.26
Platform
Other
Scaler Details
CPU
Anything else?
This issue is reproducible in the older and the latest KEDA versions. Also, tweaking KEDA_SCALEDOBJECT_CTRL_MAX_RECONCILES will not completely resolve this issue as the redundant updates will still continue to be processed.