fix: Prevented memory leak generated by not correctly cleaned HTTP resources by JorTurFer · Pull Request #5293 · kedacore/keda

JorTurFer · 2023-12-15T20:28:06Z

After profiling KEDA operator looking for this memory leak, I found a weird behavior.
When I deploy the OP given SO, during the first minutes, prometheus trigger returns 404 and the memory grows, but at some point, the memory usage becomes stable and then prometheus trigger start to produce timeouts.

This impacts in 2 way, significativelly increasing the allocation of HTTP resources

And also increasing the size of the in use memory due to stored rootCAs of the clients (we are cloning the current cached certs but we aren't modifying the cloned item, so doesn't make sense, at least atm)

In order to solve these issues, this PR does:

enforcing the Idle Connection closing during (scaler).Close(context.Context) to close them deterministically and not delegating this action to the garbage collector. This helps to reduce the allocated resources for stablishing connections
returning the rootCAs item directly instead of cloning it, as we don't need to modify it (at least for the moment) so doesn't make sense creating a copy every time when it's requested. This helps to reduce in use memory for storing the copies (one per HTTP Client can generate several instances until de GC collects them if the scaler fails all the time and KEDA regenerates the scaler cache all the time). If we need to modify it in the future, we should think about how to improve this

Checklist

Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #5248

…onnections Signed-off-by: Jorge Turrado <[email protected]>

github-actions · 2023-12-15T20:28:18Z

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

While you are waiting, make sure to:

Add an entry in our changelog in alphabetical order and link related issue
Update the documentation, if needed
Add unit & e2e tests for your changes
GitHub checks are passing
Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

Our contribution guide

Signed-off-by: Jorge Turrado Ferrero <[email protected]>

JorTurFer · 2023-12-15T20:29:44Z

/run-e2e
Update: You can check the progress here

Signed-off-by: Jorge Turrado <[email protected]>

JorTurFer · 2023-12-15T21:16:11Z

/run-e2e
Update: You can check the progress here

zroubalik

LGTM, great job!

…sources (kedacore#5293) Signed-off-by: Jorge Turrado <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]> Signed-off-by: anton.lysina <[email protected]>

… cleaned HTTP resources (kedacore#5293) Signed-off-by: Jorge Turrado <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

… cleaned HTTP resources (kedacore#5293) Signed-off-by: Jorge Turrado <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]> This wasn't a completely clean pick, I had to adjust the swift scaler client variable to be a pointer so it was nillable, but everything else was okay.

OCPBUGS-30145: fix: Prevented memory leak generated by not correctly cleaned HTTP resources (kedacore#5293)

rameshbabu009 · 2024-04-05T19:20:32Z

"scaler": "prometheusScaler", "error": "prometheus query api returned error. status: 403 response: {"message":"Credential should be scoped to a valid region. "}"} any suggestions on this error

JorTurFer · 2024-04-05T20:32:36Z

"scaler": "prometheusScaler", "error": "prometheus query api returned error. status: 403 response: {"message":"Credential should be scoped to a valid region. "}"} any suggestions on this error

This issue was closed 4 months ago and it's totally unrelated with your topic. I'd suggest opening a new issue for your case

The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never closed their underlying HTTP transport connections when Close() was called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client. This causes TCP connections to accumulate indefinitely, eventually exhausting file descriptors and causing the operator to lose connectivity. Store an explicit *http.Client on each scaler struct, inject it into the AWS SDK client via options.HTTPClient, and call CloseIdleConnections() in Close(). This matches the pattern established in PR kedacore#5293 for non-AWS scalers. Signed-off-by: Raymond Barbiero <[email protected]>

…7740) The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never closed their underlying HTTP transport connections when Close() was called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client. This causes TCP connections to accumulate indefinitely, eventually exhausting file descriptors and causing the operator to lose connectivity. Store an explicit *http.Client on each scaler struct, inject it into the AWS SDK client via options.HTTPClient, and call CloseIdleConnections() in Close(). This matches the pattern established in PR #5293 for non-AWS scalers. Signed-off-by: Raymond Barbiero <[email protected]> Co-authored-by: Rick Brouwer <[email protected]>

…edacore#7740) The AWS SDK-based scalers (SQS, Kinesis, DynamoDB, DynamoDB Streams, CloudWatch) never closed their underlying HTTP transport connections when Close() was called. ClearAwsConfig() only removes credential cache entries but does not shut down the HTTP client. This causes TCP connections to accumulate indefinitely, eventually exhausting file descriptors and causing the operator to lose connectivity. Store an explicit *http.Client on each scaler struct, inject it into the AWS SDK client via options.HTTPClient, and call CloseIdleConnections() in Close(). This matches the pattern established in PR kedacore#5293 for non-AWS scalers. Signed-off-by: Raymond Barbiero <[email protected]> Co-authored-by: Rick Brouwer <[email protected]> Signed-off-by: Yurii Shcherbak <[email protected]>

fix: Prevented memory leak generated by not correctly cleaning http c…

c6334a3

…onnections Signed-off-by: Jorge Turrado <[email protected]>

JorTurFer requested a review from a team as a code owner December 15, 2023 20:28

Merge branch 'main' into fix-leak

3388ff8

Signed-off-by: Jorge Turrado Ferrero <[email protected]>

semgrep-app Bot reviewed Dec 15, 2023

View reviewed changes

Comment thread pkg/scalers/kafka_scaler.go Outdated

JorTurFer added 2 commits December 15, 2023 21:32

fix static issue

f857ac7

Signed-off-by: Jorge Turrado <[email protected]>

return same rootCAs insteance instead of cloning it

fe19d3a

Signed-off-by: Jorge Turrado <[email protected]>

JorTurFer changed the title ~~fix: Prevented memory leak generated by not correctly cleaning http connections~~ fix: Prevented memory leak generated by not correctly cleaned HTTP resources Dec 15, 2023

JorTurFer mentioned this pull request Dec 15, 2023

keda-operator memory leak when prometheus scaler having errors #5248

Closed

zroubalik approved these changes Dec 18, 2023

View reviewed changes

zroubalik merged commit d3751e9 into kedacore:main Dec 18, 2023

JorTurFer deleted the fix-leak branch December 18, 2023 14:38

JorTurFer mentioned this pull request Jan 3, 2024

Leaked connections in Prometheus scaler on query error #5341

Closed

jkyros mentioned this pull request Mar 27, 2024

OCPBUGS-30145: fix: Prevented memory leak generated by not correctly cleaned HTTP resources (#5293) openshift/kedacore-keda#26

Merged

openshift-merge-bot Bot added a commit to openshift/kedacore-keda that referenced this pull request Mar 27, 2024

Merge pull request #26 from jkyros/http-fix-memory-leak

dc4c94b

OCPBUGS-30145: fix: Prevented memory leak generated by not correctly cleaned HTTP resources (kedacore#5293)

visualphoenix mentioned this pull request May 16, 2026

fix(aws): close idle HTTP connections in AWS scaler Close() methods #7740

Merged

5 tasks

visualphoenix mentioned this pull request May 21, 2026

AWS scalers leak TCP connections. Close() does not shut down HTTP transport. #7756

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: Prevented memory leak generated by not correctly cleaned HTTP resources#5293

fix: Prevented memory leak generated by not correctly cleaned HTTP resources#5293
zroubalik merged 4 commits into
kedacore:mainfrom
JorTurFer:fix-leak

JorTurFer commented Dec 15, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Dec 15, 2023

Uh oh!

JorTurFer commented Dec 15, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Uh oh!

JorTurFer commented Dec 15, 2023 •

edited by github-actions Bot

Loading

Uh oh!

zroubalik left a comment

Uh oh!

rameshbabu009 commented Apr 5, 2024

Uh oh!

JorTurFer commented Apr 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

JorTurFer commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

github-actions Bot commented Dec 15, 2023

Uh oh!

JorTurFer commented Dec 15, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JorTurFer commented Dec 15, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zroubalik left a comment

Choose a reason for hiding this comment

Uh oh!

rameshbabu009 commented Apr 5, 2024

Uh oh!

JorTurFer commented Apr 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JorTurFer commented Dec 15, 2023 •

edited

Loading

JorTurFer commented Dec 15, 2023 •

edited by github-actions Bot

Loading

JorTurFer commented Dec 15, 2023 •

edited by github-actions Bot

Loading