fix(aws): close idle HTTP connections in AWS scaler Close() methods by visualphoenix · Pull Request #7740 · kedacore/keda

visualphoenix · 2026-05-16T02:15:38Z

The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never closed their underlying HTTP transport connections when Close() was called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client. This causes TCP connections to accumulate indefinitely, eventually exhausting file descriptors and causing the operator to lose connectivity.

Store an explicit *http.Client on each scaler struct, inject it into the AWS SDK client via options.HTTPClient, and call CloseIdleConnections() in Close(). This matches the pattern established in PR #5293 for non-AWS scalers.

Checklist

I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added (if applicable)
Ensure make generate-scalers-schema has been run to update any outdated generated files
Changelog has been updated and is aligned with our changelog requirements, only when the change impacts end users
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #7756

github-actions · 2026-05-16T02:15:48Z

Thank you for your contribution! 🙏

Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected.

While you are waiting, make sure to:

Add an entry in our changelog in alphabetical order and link related issue
Update the documentation, if needed
Add unit & e2e tests for your changes
GitHub checks are passing
Is the DCO check failing? Here is how you can fix DCO issues

Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient.

Learn more about our contribution guide.

snyk-io · 2026-05-16T02:15:54Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

rickbrouwer · 2026-05-16T11:30:43Z

/run-e2e aws*
Update: You can check the progress here

visualphoenix · 2026-05-16T14:06:46Z

Fwiw #3387 is about azure and I had a similar issue with the aws ones - this is addressing the same problem with aws sdk - if you want me to file a new bug about the aws leak I can do so

rickbrouwer · 2026-05-17T06:56:45Z

Fwiw #3387 is about azure and I had a similar issue with the aws ones - this is addressing the same problem with aws sdk - if you want me to file a new bug about the aws leak I can do so

Yes, that seems fine to me. You could then optionally link to the similar issue in your issue.

rickbrouwer · 2026-05-17T07:07:17Z

WDYT? Could we use awshttp.NewBuildableClient() (see https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-http.html) instead? It satisfies aws.HTTPClient, preserves the SDK defaults, gives us a per-scaler Transport, and exposes its own CloseIdleConnections().

visualphoenix · 2026-05-18T03:25:16Z

Good call... I switched to NewBuildableClient().GetTransport() to get a transport with defaults. Fwiw BuildableClient itself doesn't have CloseIdleConnections() so we can't quite do what you suggested, but I took the spirit of your suggestion... lmk what you think

visualphoenix · 2026-05-21T18:29:47Z

@rickbrouwer i don't think there is anything left to do here. CloseIdleConnections isnt in the upstream sdk - but I took the spirit of your ask and preserved the transport defaults. lmk if there is something else you want me to do.

rickbrouwer · 2026-05-21T19:20:45Z

LGTM. Thanks for iterating on this.

Maybe only one thing you also mentioned:

Fwiw #3387 is about azure and I had a similar issue with the aws ones - this is addressing the same problem with aws sdk - if you want me to file a new bug about the aws leak I can do so

Can you create this separate issue and link it in the CHANGELOG and description?

visualphoenix · 2026-05-21T19:22:15Z

will do so - tyvm - standby

rickbrouwer · 2026-05-21T20:01:27Z

/run-e2e aws*
Update: You can check the progress here

wozniakjan · 2026-05-22T14:16:53Z

/run-e2e aws*
Update: You can check the progress here

passed tests: 26

Execution of tests/scalers/aws/aws_managed_prometheus_pod_identity/aws_managed_prometheus_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_managed_prometheus/aws_managed_prometheus_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_secretmanager/aws_secretmanager_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_ignore_null_values_false/aws_cloudwatch_ignore_null_values_false_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_min_metric_value/aws_cloudwatch_min_metric_value_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_identity_external_id/aws_identity_external_id_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_identity_assume_role/aws_identity_assume_role_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_metric_stat/aws_cloudwatch_metric_stat_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch/aws_cloudwatch_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_sqs_queue_pod_identity_eks/aws_sqs_queue_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_pod_identity/aws_cloudwatch_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_pod_identity_eks/aws_dynamodb_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_sqs_queue/aws_sqs_queue_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_kinesis_stream_pod_identity_eks/aws_kinesis_stream_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_kinesis_stream_pod_identity/aws_kinesis_stream_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_pod_identity_eks/aws_cloudwatch_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_kinesis_stream/aws_kinesis_stream_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_cross_account/aws_cloudwatch_cross_account_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_pod_identity/aws_dynamodb_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_sqs_queue_pod_identity/aws_sqs_queue_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_expression/aws_cloudwatch_expression_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_secretmanager_pod_identity/aws_secretmanager_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb/aws_dynamodb_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_streams_pod_identity_eks/aws_dynamodb_streams_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_streams_pod_identity/aws_dynamodb_streams_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_streams/aws_dynamodb_streams_test.go, has passed after "one" attempts

failed tests: 0

Copilot

Pull request overview

This PR addresses a resource leak in KEDA’s AWS SDK-based scalers by ensuring their underlying HTTP connection pools are deterministically cleaned up when a scaler is closed, preventing long-lived TCP connections from accumulating in the operator.

Changes:

Add an explicit *http.Client to AWS scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) and inject it via options.HTTPClient.
Call httpClient.CloseIdleConnections() from each scaler’s Close() method.
Add/adjust unit tests to cover the new Close() behavior and update the changelog entry.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pkg/scalers/aws_sqs_queue_scaler.go	Adds per-scaler HTTP client, injects into AWS SQS client, closes idle connections on `Close()`
pkg/scalers/aws_sqs_queue_scaler_test.go	Updates struct literals and adds `Close()` HTTP cleanup tests (one added test is misleading)
pkg/scalers/aws_kinesis_stream_scaler.go	Adds per-scaler HTTP client, injects into AWS Kinesis client, closes idle connections on `Close()`
pkg/scalers/aws_kinesis_stream_scaler_test.go	Updates struct literals and adds `Close()` HTTP cleanup test
pkg/scalers/aws_dynamodb_streams_scaler.go	Adds per-scaler HTTP client, injects into DynamoDB + Streams clients, closes idle connections on `Close()`
pkg/scalers/aws_dynamodb_streams_scaler_test.go	Updates struct literals and adds `Close()` HTTP cleanup test
pkg/scalers/aws_dynamodb_scaler.go	Adds per-scaler HTTP client, injects into DynamoDB client, closes idle connections on `Close()`
pkg/scalers/aws_dynamodb_scaler_test.go	Updates struct literals and adds `Close()` HTTP cleanup test
pkg/scalers/aws_cloudwatch_scaler.go	Adds per-scaler HTTP client, injects into CloudWatch client, closes idle connections on `Close()`
pkg/scalers/aws_cloudwatch_scaler_test.go	Updates struct literals and adds `Close()` HTTP cleanup test
CHANGELOG.md	Documents the fix for AWS scaler TCP connection leak

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never closed their underlying HTTP transport connections when Close() was called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client. This causes TCP connections to accumulate indefinitely, eventually exhausting file descriptors and causing the operator to lose connectivity. Store an explicit *http.Client on each scaler struct, inject it into the AWS SDK client via options.HTTPClient, and call CloseIdleConnections() in Close(). This matches the pattern established in PR kedacore#5293 for non-AWS scalers. Signed-off-by: Raymond Barbiero <[email protected]>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

visualphoenix · 2026-05-22T23:44:50Z

look - AI reviews here are imho bad/annoying now - i'm not making any more changes to this code. either take it or leave it but i'm done fighting copilot.

rickbrouwer · 2026-05-23T06:21:33Z

/run-e2e aws*
Update: You can check the progress here

rickbrouwer · 2026-05-24T18:17:31Z

/run-e2e aws*
Update: You can check the progress here

passed tests: 26

Execution of tests/scalers/aws/aws_managed_prometheus/aws_managed_prometheus_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_managed_prometheus_pod_identity/aws_managed_prometheus_pod_identity_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_secretmanager/aws_secretmanager_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_secretmanager_pod_identity/aws_secretmanager_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_ignore_null_values_false/aws_cloudwatch_ignore_null_values_false_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_min_metric_value/aws_cloudwatch_min_metric_value_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_identity_external_id/aws_identity_external_id_test.go, has passed after "one" attempts
Execution of tests/secret-providers/aws_identity_assume_role/aws_identity_assume_role_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_pod_identity_eks/aws_cloudwatch_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_sqs_queue_pod_identity_eks/aws_sqs_queue_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_pod_identity/aws_cloudwatch_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_cross_account/aws_cloudwatch_cross_account_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_metric_stat/aws_cloudwatch_metric_stat_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch/aws_cloudwatch_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_kinesis_stream_pod_identity_eks/aws_kinesis_stream_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_pod_identity/aws_dynamodb_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_pod_identity_eks/aws_dynamodb_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_kinesis_stream/aws_kinesis_stream_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_sqs_queue_pod_identity/aws_sqs_queue_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_sqs_queue/aws_sqs_queue_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_cloudwatch_expression/aws_cloudwatch_expression_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_kinesis_stream_pod_identity/aws_kinesis_stream_pod_identity_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb/aws_dynamodb_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_streams_pod_identity_eks/aws_dynamodb_streams_pod_identity_eks_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_streams/aws_dynamodb_streams_test.go, has passed after "one" attempts
Execution of tests/scalers/aws/aws_dynamodb_streams_pod_identity/aws_dynamodb_streams_pod_identity_test.go, has passed after "one" attempts

failed tests: 0

rickbrouwer

Thanks!

wozniakjan · 2026-05-25T06:51:37Z

look - AI reviews here are imho bad/annoying now - i'm not making any more changes to this code. either take it or leave it but i'm done fighting copilot.

fair call, from my experience, even the best LLM models report something about 20% - 40% valid reviews and when retriggered on updates, past 5th iteration this ratio drops to 0

rickbrouwer · 2026-05-25T06:59:23Z

I completely agree. It indeed looks like the LLM always wants to give something back, no matter how far-fetched, even if the code is fine.

That’s why it’s good to decide at some point that it’s best to just let it go :)

…edacore#7740) The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never closed their underlying HTTP transport connections when Close() was called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client. This causes TCP connections to accumulate indefinitely, eventually exhausting file descriptors and causing the operator to lose connectivity. Store an explicit *http.Client on each scaler struct, inject it into the AWS SDK client via options.HTTPClient, and call CloseIdleConnections() in Close(). This matches the pattern established in PR kedacore#5293 for non-AWS scalers. Signed-off-by: Raymond Barbiero <[email protected]> Co-authored-by: Rick Brouwer <[email protected]> Signed-off-by: Yurii Shcherbak <[email protected]>

visualphoenix requested a review from a team as a code owner May 16, 2026 02:15

keda-automation requested a review from a team May 16, 2026 02:15

visualphoenix force-pushed the sqs-bug branch from f4232fb to a5a3749 Compare May 16, 2026 02:34

visualphoenix force-pushed the sqs-bug branch from a5a3749 to b412c0e Compare May 18, 2026 03:23

visualphoenix force-pushed the sqs-bug branch from b412c0e to cccc3c5 Compare May 21, 2026 19:57

rickbrouwer added waiting-for-e2e and removed waiting-for-e2e labels May 21, 2026

rickbrouwer approved these changes May 22, 2026

View reviewed changes

rickbrouwer added Awaiting/2nd-approval This PR needs one more approval review nice-to-have:keda-v2.20 Not strictly necessary, but nice if you can bring it along labels May 22, 2026

wozniakjan force-pushed the sqs-bug branch from cccc3c5 to 026e851 Compare May 22, 2026 14:16

keda-automation requested a review from a team May 22, 2026 14:17

wozniakjan approved these changes May 22, 2026

View reviewed changes

wozniakjan requested a review from Copilot May 22, 2026 14:17

rickbrouwer added the waiting-for-e2e label May 22, 2026

Copilot started reviewing on behalf of wozniakjan May 22, 2026 14:18 View session

wozniakjan enabled auto-merge (squash) May 22, 2026 14:18