Skip to content

AWS scalers leak TCP connections. Close() does not shut down HTTP transport. #7756

Description

@visualphoenix

Report

The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never close their underlying HTTP transport connections when Close() is called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client.

This is the same class of connection leak fixed for other scalers in #3387 and #5293, but the AWS scalers were never addressed. Many other scalers (Prometheus, Kafka, RabbitMQ, etc.) already call httpClient.CloseIdleConnections() in their Close() methods — the AWS scalers should do the same.

We hit this in production: the keda-operator pod accumulated idle connections over time, which eventually went stale and were reset by upstream infrastructure, causing repeated errors and readiness probe failures.

Expected Behavior

When a scaler is closed via Close(), its underlying HTTP transport connections should be cleaned up. Idle connections should not accumulate in the pool indefinitely.

Actual Behavior

Idle TCP connections from AWS SDK HTTP clients accumulate over time because CloseIdleConnections() is never called on the transport. Stale connections get reset by upstream load balancers, leading to cascading connection reset by peer errors and readiness probe failures.

Steps to Reproduce the Problem

  1. Deploy KEDA with multiple AWS-based ScaledObjects (e.g. SQS scalers)
  2. Over time, observe increasing TCP connections in TIME_WAIT / ESTABLISHED state from the KEDA operator pod
  3. Upstream load balancers eventually reset stale idle connections, causing connection reset by peer errors on every polling interval

Logs from KEDA operator

{"level":"error","ts":"...","logger":"scale_handler","msg":"error getting scale decision", "scaledObject.Namespace":"...","scaledObject.Name":"...", "error":"read tcp ...: connection reset by peer"}

KEDA Version

2.19.0

Kubernetes Version

1.33

Platform

Amazon Web Services

Scaler Details

aws-sqs-queue

Would you be open to contributing a fix?

Yes

Anything else?

No response

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Ready To Ship

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions