Report
The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never close their underlying HTTP transport connections when Close() is called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client.
This is the same class of connection leak fixed for other scalers in #3387 and #5293, but the AWS scalers were never addressed. Many other scalers (Prometheus, Kafka, RabbitMQ, etc.) already call httpClient.CloseIdleConnections() in their Close() methods — the AWS scalers should do the same.
We hit this in production: the keda-operator pod accumulated idle connections over time, which eventually went stale and were reset by upstream infrastructure, causing repeated errors and readiness probe failures.
Expected Behavior
When a scaler is closed via Close(), its underlying HTTP transport connections should be cleaned up. Idle connections should not accumulate in the pool indefinitely.
Actual Behavior
Idle TCP connections from AWS SDK HTTP clients accumulate over time because CloseIdleConnections() is never called on the transport. Stale connections get reset by upstream load balancers, leading to cascading connection reset by peer errors and readiness probe failures.
Steps to Reproduce the Problem
- Deploy KEDA with multiple AWS-based ScaledObjects (e.g. SQS scalers)
- Over time, observe increasing TCP connections in TIME_WAIT / ESTABLISHED state from the KEDA operator pod
- Upstream load balancers eventually reset stale idle connections, causing
connection reset by peer errors on every polling interval
Logs from KEDA operator
{"level":"error","ts":"...","logger":"scale_handler","msg":"error getting scale decision", "scaledObject.Namespace":"...","scaledObject.Name":"...", "error":"read tcp ...: connection reset by peer"}
KEDA Version
2.19.0
Kubernetes Version
1.33
Platform
Amazon Web Services
Scaler Details
aws-sqs-queue
Would you be open to contributing a fix?
Yes
Anything else?
No response
Report
The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never close their underlying HTTP transport connections when
Close()is called.ClearAwsConfig()only removes credential cache entries but does not shut down the HTTP client.This is the same class of connection leak fixed for other scalers in #3387 and #5293, but the AWS scalers were never addressed. Many other scalers (Prometheus, Kafka, RabbitMQ, etc.) already call
httpClient.CloseIdleConnections()in theirClose()methods — the AWS scalers should do the same.We hit this in production: the keda-operator pod accumulated idle connections over time, which eventually went stale and were reset by upstream infrastructure, causing repeated errors and readiness probe failures.
Expected Behavior
When a scaler is closed via
Close(), its underlying HTTP transport connections should be cleaned up. Idle connections should not accumulate in the pool indefinitely.Actual Behavior
Idle TCP connections from AWS SDK HTTP clients accumulate over time because
CloseIdleConnections()is never called on the transport. Stale connections get reset by upstream load balancers, leading to cascadingconnection reset by peererrors and readiness probe failures.Steps to Reproduce the Problem
connection reset by peererrors on every polling intervalLogs from KEDA operator
KEDA Version
2.19.0
Kubernetes Version
1.33
Platform
Amazon Web Services
Scaler Details
aws-sqs-queue
Would you be open to contributing a fix?
Yes
Anything else?
No response