Merging to release-5.13: TT-17147: fix: remove unconditional break in memorycache cleanup timer (#8180)#8227
Merged
mativm02 merged 1 commit intoMay 18, 2026
Conversation
#8180) ## Problem / Task The `internal/memorycache.Cache.startCleanupTimer` exits after one cleanup due to an unconditional `break` inside the for/select loop. This causes `BucketStorage` entries used by `SessionLimiter.limitDRL` to accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB live heap) when using rate limiting by client IP. ## Who is affected? Users are potentially affected by this memory leak if they meet **all** of the following conditions: - Running Tyk Gateway version **5.8.x** (or any version where `internal/memorycache` was introduced/embedded). - Using **Rate Limiting** (specifically the default distributed rate limiter `limitDRL`, which relies on `BucketStorage` backed by `memorycache`). - Experiencing a high cardinality of rate limit keys (e.g., rate limiting by **Client IP** or using custom gRPC plugins that generate many unique rate limit keys). ## Changes - Removed the unconditional `break` statement in `internal/memorycache/cache.go` inside `startCleanupTimer` so the loop continues to run and clean up expired items. - Added `TestCache_CleanupTimer` in `internal/memorycache/cache_test.go` to verify the cleanup timer runs multiple times and correctly removes expired items. - Added `BenchmarkCache_MemoryLeak` in `internal/memorycache/cache_test.go` to validate memory usage and performance of the cache over time, simulating the memory leak scenario. ## Benchmark Results To validate the fix, we ran `BenchmarkCache_MemoryLeak` which reports an `items_left` metric showing how many items remain in the cache at the end of the run. **Before the fix (with the unconditional break):** ```text BenchmarkCache_MemoryLeak-16 3703683 1384 ns/op 456112 items_left 101 B/op 3 allocs/op ``` *Result: 456,112 items left in the cache. The cleanup goroutine was dead, causing items to accumulate indefinitely.* **After the fix (unconditional break removed):** ```text BenchmarkCache_MemoryLeak-16 3742993 1387 ns/op 1.000 items_left 90 B/op 3 allocs/op ``` *Result: Only 1 item left in the cache. The cleanup timer runs continuously and successfully evicts expired items.* ## Testing - Ran `go test -v -run TestCache_CleanupTimer ./internal/memorycache` which passes. - Ran `go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$ ./internal/memorycache` which passes and shows stable memory usage. - Verified the build passes with `make build`. --------- Co-authored-by: Leonid Bugaev <[email protected]> Co-authored-by: Matias <[email protected]> (cherry picked from commit 114bc3c)
Contributor
🚨 Jira Linter FailedCommit: The Jira linter failed to validate your PR. Please check the error details below: 🔍 Click to view error detailsNext Steps
This comment will be automatically deleted once the linter passes. |
Contributor
Author
Contributor
Author
mativm02
enabled auto-merge (squash)
May 15, 2026 11:50
|
mativm02
deleted the
merge/release-5.13/114bc3c369906f1e0a533a85499a9ef94b6fc986/TT-17147
branch
May 18, 2026 07:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



TT-17147: fix: remove unconditional break in memorycache cleanup timer (#8180)
Problem / Task
The
internal/memorycache.Cache.startCleanupTimerexits after onecleanup due to an unconditional
breakinside the for/select loop. Thiscauses
BucketStorageentries used bySessionLimiter.limitDRLtoaccumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.
Who is affected?
Users are potentially affected by this memory leak if they meet all
of the following conditions:
internal/memorycachewas introduced/embedded).limiter
limitDRL, which relies onBucketStoragebacked bymemorycache).limiting by Client IP or using custom gRPC plugins that generate
many unique rate limit keys).
Changes
breakstatement ininternal/memorycache/cache.goinsidestartCleanupTimerso the loopcontinues to run and clean up expired items.
TestCache_CleanupTimerininternal/memorycache/cache_test.goto verify the cleanup timer runs multiple times and correctly removes
expired items.
BenchmarkCache_MemoryLeakininternal/memorycache/cache_test.goto validate memory usage andperformance of the cache over time, simulating the memory leak scenario.
Benchmark Results
To validate the fix, we ran
BenchmarkCache_MemoryLeakwhich reports anitems_leftmetric showing how many items remain in the cache at theend of the run.
Before the fix (with the unconditional break):
Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.
After the fix (unconditional break removed):
Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.
Testing
go test -v -run TestCache_CleanupTimer ./internal/memorycachewhich passes.
go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$ ./internal/memorycachewhich passes and shows stable memory usage.make build.Co-authored-by: Leonid Bugaev [email protected]
Co-authored-by: Matias [email protected]
Ticket Details
TT-17147
Generated at: 2026-05-15 11:47:23