Skip to content

Merging to release-5.13: TT-17147: fix: remove unconditional break in memorycache cleanup timer (#8180)#8227

Merged
mativm02 merged 1 commit into
release-5.13from
merge/release-5.13/114bc3c369906f1e0a533a85499a9ef94b6fc986/TT-17147
May 18, 2026
Merged

Merging to release-5.13: TT-17147: fix: remove unconditional break in memorycache cleanup timer (#8180)#8227
mativm02 merged 1 commit into
release-5.13from
merge/release-5.13/114bc3c369906f1e0a533a85499a9ef94b6fc986/TT-17147

Conversation

@probelabs

@probelabs probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

TT-17147: fix: remove unconditional break in memorycache cleanup timer (#8180)

Problem / Task

The internal/memorycache.Cache.startCleanupTimer exits after one
cleanup due to an unconditional break inside the for/select loop. This
causes BucketStorage entries used by SessionLimiter.limitDRL to
accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.

Who is affected?

Users are potentially affected by this memory leak if they meet all
of the following conditions:

  • Running Tyk Gateway version 5.8.x (or any version where
    internal/memorycache was introduced/embedded).
  • Using Rate Limiting (specifically the default distributed rate
    limiter limitDRL, which relies on BucketStorage backed by
    memorycache).
  • Experiencing a high cardinality of rate limit keys (e.g., rate
    limiting by Client IP or using custom gRPC plugins that generate
    many unique rate limit keys).

Changes

  • Removed the unconditional break statement in
    internal/memorycache/cache.go inside startCleanupTimer so the loop
    continues to run and clean up expired items.
  • Added TestCache_CleanupTimer in internal/memorycache/cache_test.go
    to verify the cleanup timer runs multiple times and correctly removes
    expired items.
  • Added BenchmarkCache_MemoryLeak in
    internal/memorycache/cache_test.go to validate memory usage and
    performance of the cache over time, simulating the memory leak scenario.

Benchmark Results

To validate the fix, we ran BenchmarkCache_MemoryLeak which reports an
items_left metric showing how many items remain in the cache at the
end of the run.

Before the fix (with the unconditional break):

BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op

Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.

After the fix (unconditional break removed):

BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op

Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.

Testing

  • Ran go test -v -run TestCache_CleanupTimer ./internal/memorycache
    which passes.
  • Ran go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$ ./internal/memorycache which passes and shows stable memory usage.
  • Verified the build passes with make build.

Co-authored-by: Leonid Bugaev [email protected]
Co-authored-by: Matias [email protected]

Ticket Details

TT-17147
Status Closed
Summary Memory Leak in internal/memorycache.Cache.startCleanupTimer

Generated at: 2026-05-15 11:47:23

#8180)

## Problem / Task
The `internal/memorycache.Cache.startCleanupTimer` exits after one
cleanup due to an unconditional `break` inside the for/select loop. This
causes `BucketStorage` entries used by `SessionLimiter.limitDRL` to
accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.

## Who is affected?
Users are potentially affected by this memory leak if they meet **all**
of the following conditions:
- Running Tyk Gateway version **5.8.x** (or any version where
`internal/memorycache` was introduced/embedded).
- Using **Rate Limiting** (specifically the default distributed rate
limiter `limitDRL`, which relies on `BucketStorage` backed by
`memorycache`).
- Experiencing a high cardinality of rate limit keys (e.g., rate
limiting by **Client IP** or using custom gRPC plugins that generate
many unique rate limit keys).

## Changes
- Removed the unconditional `break` statement in
`internal/memorycache/cache.go` inside `startCleanupTimer` so the loop
continues to run and clean up expired items.
- Added `TestCache_CleanupTimer` in `internal/memorycache/cache_test.go`
to verify the cleanup timer runs multiple times and correctly removes
expired items.
- Added `BenchmarkCache_MemoryLeak` in
`internal/memorycache/cache_test.go` to validate memory usage and
performance of the cache over time, simulating the memory leak scenario.

## Benchmark Results
To validate the fix, we ran `BenchmarkCache_MemoryLeak` which reports an
`items_left` metric showing how many items remain in the cache at the
end of the run.

**Before the fix (with the unconditional break):**
```text
BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op
```
*Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.*

**After the fix (unconditional break removed):**
```text
BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op
```
*Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.*

## Testing
- Ran `go test -v -run TestCache_CleanupTimer ./internal/memorycache`
which passes.
- Ran `go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$
./internal/memorycache` which passes and shows stable memory usage.
- Verified the build passes with `make build`.

---------

Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
(cherry picked from commit 114bc3c)
@github-actions

Copy link
Copy Markdown
Contributor

🚨 Jira Linter Failed

Commit: e45df57
Failed at: 2026-05-15 11:47:25 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details
failed to validate Jira issue: jira ticket TT-17147 has status 'Closed' but must be one of: In Dev, In Code Review, Ready For Dev, Dod Check

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@probelabs

probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor Author
\n\n \n\n \n\n

Powered by Visor from Probelabs

Last updated: 2026-05-15T11:47:29.317Z | Triggered by: pr_opened | Commit: e45df57

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs

probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

Powered by Visor from Probelabs

Last updated: 2026-05-15T11:47:30.106Z | Triggered by: pr_opened | Commit: e45df57

💡 TIP: You can chat with Visor using /visor ask <your question>

@mativm02
mativm02 enabled auto-merge (squash) May 15, 2026 11:50
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@mativm02
mativm02 merged commit 5a8b101 into release-5.13 May 18, 2026
49 of 82 checks passed
@mativm02
mativm02 deleted the merge/release-5.13/114bc3c369906f1e0a533a85499a9ef94b6fc986/TT-17147 branch May 18, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant