Skip to content

TT-17147: fix: remove unconditional break in memorycache cleanup timer#8180

Merged
mativm02 merged 7 commits into
masterfrom
fix-memorycache-leak
May 15, 2026
Merged

TT-17147: fix: remove unconditional break in memorycache cleanup timer#8180
mativm02 merged 7 commits into
masterfrom
fix-memorycache-leak

Conversation

@probelabs

@probelabs probelabs Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

Problem / Task

The internal/memorycache.Cache.startCleanupTimer exits after one cleanup due to an unconditional break inside the for/select loop. This causes BucketStorage entries used by SessionLimiter.limitDRL to accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB live heap) when using rate limiting by client IP.

Who is affected?

Users are potentially affected by this memory leak if they meet all of the following conditions:

  • Running Tyk Gateway version 5.8.x (or any version where internal/memorycache was introduced/embedded).
  • Using Rate Limiting (specifically the default distributed rate limiter limitDRL, which relies on BucketStorage backed by memorycache).
  • Experiencing a high cardinality of rate limit keys (e.g., rate limiting by Client IP or using custom gRPC plugins that generate many unique rate limit keys).

Changes

  • Removed the unconditional break statement in internal/memorycache/cache.go inside startCleanupTimer so the loop continues to run and clean up expired items.
  • Added TestCache_CleanupTimer in internal/memorycache/cache_test.go to verify the cleanup timer runs multiple times and correctly removes expired items.
  • Added BenchmarkCache_MemoryLeak in internal/memorycache/cache_test.go to validate memory usage and performance of the cache over time, simulating the memory leak scenario.

Benchmark Results

To validate the fix, we ran BenchmarkCache_MemoryLeak which reports an items_left metric showing how many items remain in the cache at the end of the run.

Before the fix (with the unconditional break):

BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op

Result: 456,112 items left in the cache. The cleanup goroutine was dead, causing items to accumulate indefinitely.

After the fix (unconditional break removed):

BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op

Result: Only 1 item left in the cache. The cleanup timer runs continuously and successfully evicts expired items.

Testing

  • Ran go test -v -run TestCache_CleanupTimer ./internal/memorycache which passes.
  • Ran go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$ ./internal/memorycache which passes and shows stable memory usage.
  • Verified the build passes with make build.

@probelabs

probelabs Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

This PR addresses a critical memory leak in the internal/memorycache component by removing an unconditional break statement from its cleanup timer loop. This bug caused the cleanup goroutine to exit after a single run, preventing the eviction of expired items and leading to unbounded memory growth, especially when using high-cardinality rate limiting (e.g., by client IP).

The fix ensures the cleanup timer runs continuously as intended. To validate this and prevent regressions, the PR introduces comprehensive tests: a unit test (TestCache_CleanupTimer) to verify the timer's periodic execution, a benchmark (BenchmarkCache_MemoryLeak) to confirm memory stability under churn, and a GC-aware test (TestCache_MemoryLeak_GC) to explicitly verify memory reclamation.

Files Changed Analysis

  • internal/memorycache/cache.go: The core change is the removal of a single break statement in the startCleanupTimer function, allowing the cleanup loop to run indefinitely as intended.
  • internal/memorycache/cache_test.go: 99 lines of new test code have been added. This includes TestCache_CleanupTimer to ensure the timer runs multiple times, BenchmarkCache_MemoryLeak to simulate the leak scenario and confirm stable memory usage, and TestCache_MemoryLeak_GC to verify that Go's garbage collector reclaims memory from expired items.

Architecture & Impact Assessment

  • What this PR accomplishes: It resolves a significant memory leak in the gateway's core in-memory caching utility, directly improving the stability and reliability of features that depend on it.
  • Key technical changes introduced:
    1. Bug Fix: Removed the unconditional break in startCleanupTimer to enable continuous cache cleanup.
    2. Test Validation: Added a new unit test to confirm the timer runs repeatedly.
    3. Memory Verification: Added benchmark and GC-aware tests to prove the memory leak is resolved and memory is reclaimed.
  • Affected system components: The primary impact is on the Session Limiter (limitDRL), which uses memorycache for distributed rate limiting. However, because internal/memorycache is a foundational component, this fix also hardens other critical gateway functions that rely on it, including session management and certificate caching.
  • Flow Visualization: The change corrects the control flow of the cleanup timer from a single-run to a continuous loop.
graph TD
    subgraph "Before"
        A[Start Timer] --> B{Select};
        B -- ticker.C --> C["cleanup()"];
        C --> D[break];
        D --> E[Exit Loop];
    end

    subgraph "After"
        F[Start Timer] --> G{Select};
        G -- ticker.C --> H["cleanup()"];
        H --> G;
        G -- ctx.Done --> I[Exit Loop];
    end
Loading

Scope Discovery & Context Expansion

The bug in internal/memorycache has a broad impact radius. The PR correctly identifies the SessionLimiter as the most visible area affected by the leak due to the high churn of rate-limiting keys.

Further analysis confirms that this generic caching component is also used by:

  • Session Management: The core authentication and authorization layer relies on this cache for session state.
  • Certificate Management: Used for caching TLS certificates and JWKS (JSON Web Key Sets), which is critical for JWT validation performance.

By fixing the underlying cache implementation, this PR provides a crucial stability improvement across multiple high-throughput features of the gateway, preventing potential memory-related outages.

Metadata
  • Review Effort: 2 / 5
  • Primary Label: bug

Powered by Visor from Probelabs

Last updated: 2026-05-15T10:42:24.170Z | Triggered by: pr_updated | Commit: 8549fa3

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs

probelabs Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

✅ Security Check Passed

No security issues found – changes LGTM.

✅ Security Check Passed

No security issues found – changes LGTM.

\n\n

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

✅ Performance Check Passed

No performance issues found – changes LGTM.

Quality Issues (1)

Severity Location Issue
🟡 Warning internal/memorycache/cache_test.go:128
The comment on the preceding line mentions a 2MB memory overhead, but the test uses a hardcoded delta of 5MB. This discrepancy, combined with the use of a large, unexplained "magic number", weakens the test's precision and could mask smaller memory leaks.
💡 SuggestionUpdate the comment to match the 5MB value and add a brief justification for this specific overhead allowance, or adjust the code to use the 2MB value mentioned in the comment if that is sufficient. The code and comment should be consistent.

Powered by Visor from Probelabs

Last updated: 2026-05-15T10:42:13.456Z | Triggered by: pr_updated | Commit: 8549fa3

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs probelabs Bot changed the title fix: remove unconditional break in memorycache cleanup timer TT-17147: fix: remove unconditional break in memorycache cleanup timer May 6, 2026
@mativm02
mativm02 enabled auto-merge (squash) May 14, 2026 13:36
@github-actions

github-actions Bot commented May 14, 2026

Copy link
Copy Markdown
Contributor

🎯 Recommended Merge Targets

Based on JIRA ticket TT-17147: Memory Leak in internal/memorycache.Cache.startCleanupTimer

Fix Version: Tyk 5.8.14

Required:

  • release-5.8.14 - Exact version branch for Tyk 5.8.14 - specific patch release
  • release-5.8 - Minor version branch for 5.8.x patches - required for creating Tyk 5.8.14
  • master - Main development branch - ensures fix is in all future releases

Fix Version: Tyk 5.13.0

Required:

  • release-5.13.0 - Exact version branch for Tyk 5.13.0 - specific patch release
  • master - Main development branch - ensures fix is in all future releases

Recommended:

  • release-5.13 - Minor version branch for 5.13.x releases

📋 Workflow

  1. Merge this PR to master first

  2. Cherry-pick to release branches by commenting on the merged PR:

    • /release to release-5.8.14
    • /release to release-5.8
    • /release to release-5.13.0
    • /release to release-5.13
  3. Automated backport - The bot will automatically create backport PRs to the specified release branches

mativm02 and others added 2 commits May 14, 2026 10:52
Remove trailing whitespace on two blank lines and add the missing
final newline. Fixes golangci-lint gci finding flagged on PR #8180.
@github-actions

Copy link
Copy Markdown
Contributor

🚨 Jira Linter Failed

Commit: 8549fa3
Failed at: 2026-05-15 10:41:16 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details
failed to validate branch and PR title rules: branch name 'fix-memorycache-leak' must contain a valid Jira ticket ID (e.g., ABC-123)

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@mativm02
mativm02 merged commit 114bc3c into master May 15, 2026
53 of 55 checks passed
@mativm02
mativm02 deleted the fix-memorycache-leak branch May 15, 2026 11:46
@lghiur

lghiur commented May 15, 2026

Copy link
Copy Markdown
Collaborator

/release to release-5.13

@lghiur

lghiur commented May 15, 2026

Copy link
Copy Markdown
Collaborator

/release to release-5.13.0

@probelabs

probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

✅ Cherry-pick successful. A PR was created: #8227

@probelabs

probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

✅ Cherry-pick successful. A PR was created: #8228

@lghiur

lghiur commented May 15, 2026

Copy link
Copy Markdown
Collaborator

/release to release-5.8

@lghiur

lghiur commented May 15, 2026

Copy link
Copy Markdown
Collaborator

/release to release-5.8.14

@probelabs

probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

✅ Cherry-pick successful. A PR was created: #8229

@probelabs

probelabs Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

✅ Cherry-pick successful. A PR was created: #8230

mativm02 added a commit that referenced this pull request May 15, 2026
…in memorycache cleanup timer (#8180) (#8228)

TT-17147: fix: remove unconditional break in memorycache cleanup timer
(#8180)

## Problem / Task
The `internal/memorycache.Cache.startCleanupTimer` exits after one
cleanup due to an unconditional `break` inside the for/select loop. This
causes `BucketStorage` entries used by `SessionLimiter.limitDRL` to
accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.

## Who is affected?
Users are potentially affected by this memory leak if they meet **all**
of the following conditions:
- Running Tyk Gateway version **5.8.x** (or any version where
`internal/memorycache` was introduced/embedded).
- Using **Rate Limiting** (specifically the default distributed rate
limiter `limitDRL`, which relies on `BucketStorage` backed by
`memorycache`).
- Experiencing a high cardinality of rate limit keys (e.g., rate
limiting by **Client IP** or using custom gRPC plugins that generate
many unique rate limit keys).

## Changes
- Removed the unconditional `break` statement in
`internal/memorycache/cache.go` inside `startCleanupTimer` so the loop
continues to run and clean up expired items.
- Added `TestCache_CleanupTimer` in `internal/memorycache/cache_test.go`
to verify the cleanup timer runs multiple times and correctly removes
expired items.
- Added `BenchmarkCache_MemoryLeak` in
`internal/memorycache/cache_test.go` to validate memory usage and
performance of the cache over time, simulating the memory leak scenario.

## Benchmark Results
To validate the fix, we ran `BenchmarkCache_MemoryLeak` which reports an
`items_left` metric showing how many items remain in the cache at the
end of the run.

**Before the fix (with the unconditional break):**
```text
BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op
```
*Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.*

**After the fix (unconditional break removed):**
```text
BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op
```
*Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.*

## Testing
- Ran `go test -v -run TestCache_CleanupTimer ./internal/memorycache`
which passes.
- Ran `go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$
./internal/memorycache` which passes and shows stable memory usage.
- Verified the build passes with `make build`.

---------

Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-17147" title="TT-17147"
target="_blank">TT-17147</a>
</summary>

|         |    |
|---------|----|
| Status  | Closed |
| Summary | Memory Leak in internal/memorycache.Cache.startCleanupTimer
|

Generated at: 2026-05-15 11:47:30

</details>

<!---TykTechnologies/jira-linter ends here-->

Co-authored-by: probelabs[bot] <219682034+probelabs[bot]@users.noreply.github.com>
Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
mativm02 added a commit that referenced this pull request May 18, 2026
… memorycache cleanup timer (#8180) (#8227)

TT-17147: fix: remove unconditional break in memorycache cleanup timer
(#8180)

## Problem / Task
The `internal/memorycache.Cache.startCleanupTimer` exits after one
cleanup due to an unconditional `break` inside the for/select loop. This
causes `BucketStorage` entries used by `SessionLimiter.limitDRL` to
accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.

## Who is affected?
Users are potentially affected by this memory leak if they meet **all**
of the following conditions:
- Running Tyk Gateway version **5.8.x** (or any version where
`internal/memorycache` was introduced/embedded).
- Using **Rate Limiting** (specifically the default distributed rate
limiter `limitDRL`, which relies on `BucketStorage` backed by
`memorycache`).
- Experiencing a high cardinality of rate limit keys (e.g., rate
limiting by **Client IP** or using custom gRPC plugins that generate
many unique rate limit keys).

## Changes
- Removed the unconditional `break` statement in
`internal/memorycache/cache.go` inside `startCleanupTimer` so the loop
continues to run and clean up expired items.
- Added `TestCache_CleanupTimer` in `internal/memorycache/cache_test.go`
to verify the cleanup timer runs multiple times and correctly removes
expired items.
- Added `BenchmarkCache_MemoryLeak` in
`internal/memorycache/cache_test.go` to validate memory usage and
performance of the cache over time, simulating the memory leak scenario.

## Benchmark Results
To validate the fix, we ran `BenchmarkCache_MemoryLeak` which reports an
`items_left` metric showing how many items remain in the cache at the
end of the run.

**Before the fix (with the unconditional break):**
```text
BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op
```
*Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.*

**After the fix (unconditional break removed):**
```text
BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op
```
*Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.*

## Testing
- Ran `go test -v -run TestCache_CleanupTimer ./internal/memorycache`
which passes.
- Ran `go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$
./internal/memorycache` which passes and shows stable memory usage.
- Verified the build passes with `make build`.

---------

Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-17147" title="TT-17147"
target="_blank">TT-17147</a>
</summary>

|         |    |
|---------|----|
| Status  | Closed |
| Summary | Memory Leak in internal/memorycache.Cache.startCleanupTimer
|

Generated at: 2026-05-15 11:47:23

</details>

<!---TykTechnologies/jira-linter ends here-->

Co-authored-by: probelabs[bot] <219682034+probelabs[bot]@users.noreply.github.com>
Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
mativm02 added a commit that referenced this pull request May 18, 2026
…in memorycache cleanup timer (#8180) (#8230)

TT-17147: fix: remove unconditional break in memorycache cleanup timer
(#8180)

## Problem / Task
The `internal/memorycache.Cache.startCleanupTimer` exits after one
cleanup due to an unconditional `break` inside the for/select loop. This
causes `BucketStorage` entries used by `SessionLimiter.limitDRL` to
accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.

## Who is affected?
Users are potentially affected by this memory leak if they meet **all**
of the following conditions:
- Running Tyk Gateway version **5.8.x** (or any version where
`internal/memorycache` was introduced/embedded).
- Using **Rate Limiting** (specifically the default distributed rate
limiter `limitDRL`, which relies on `BucketStorage` backed by
`memorycache`).
- Experiencing a high cardinality of rate limit keys (e.g., rate
limiting by **Client IP** or using custom gRPC plugins that generate
many unique rate limit keys).

## Changes
- Removed the unconditional `break` statement in
`internal/memorycache/cache.go` inside `startCleanupTimer` so the loop
continues to run and clean up expired items.
- Added `TestCache_CleanupTimer` in `internal/memorycache/cache_test.go`
to verify the cleanup timer runs multiple times and correctly removes
expired items.
- Added `BenchmarkCache_MemoryLeak` in
`internal/memorycache/cache_test.go` to validate memory usage and
performance of the cache over time, simulating the memory leak scenario.

## Benchmark Results
To validate the fix, we ran `BenchmarkCache_MemoryLeak` which reports an
`items_left` metric showing how many items remain in the cache at the
end of the run.

**Before the fix (with the unconditional break):**
```text
BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op
```
*Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.*

**After the fix (unconditional break removed):**
```text
BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op
```
*Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.*

## Testing
- Ran `go test -v -run TestCache_CleanupTimer ./internal/memorycache`
which passes.
- Ran `go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$
./internal/memorycache` which passes and shows stable memory usage.
- Verified the build passes with `make build`.

---------

Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-17147" title="TT-17147"
target="_blank">TT-17147</a>
</summary>

|         |    |
|---------|----|
| Status  | Closed |
| Summary | Memory Leak in internal/memorycache.Cache.startCleanupTimer
|

Generated at: 2026-05-15 11:48:26

</details>

<!---TykTechnologies/jira-linter ends here-->

Co-authored-by: probelabs[bot] <219682034+probelabs[bot]@users.noreply.github.com>
Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
mativm02 added a commit that referenced this pull request May 18, 2026
…memorycache cleanup timer (#8180) (#8229)

TT-17147: fix: remove unconditional break in memorycache cleanup timer
(#8180)

## Problem / Task
The `internal/memorycache.Cache.startCleanupTimer` exits after one
cleanup due to an unconditional `break` inside the for/select loop. This
causes `BucketStorage` entries used by `SessionLimiter.limitDRL` to
accumulate indefinitely, leading to a massive memory leak (e.g., 1.9 GB
live heap) when using rate limiting by client IP.

## Who is affected?
Users are potentially affected by this memory leak if they meet **all**
of the following conditions:
- Running Tyk Gateway version **5.8.x** (or any version where
`internal/memorycache` was introduced/embedded).
- Using **Rate Limiting** (specifically the default distributed rate
limiter `limitDRL`, which relies on `BucketStorage` backed by
`memorycache`).
- Experiencing a high cardinality of rate limit keys (e.g., rate
limiting by **Client IP** or using custom gRPC plugins that generate
many unique rate limit keys).

## Changes
- Removed the unconditional `break` statement in
`internal/memorycache/cache.go` inside `startCleanupTimer` so the loop
continues to run and clean up expired items.
- Added `TestCache_CleanupTimer` in `internal/memorycache/cache_test.go`
to verify the cleanup timer runs multiple times and correctly removes
expired items.
- Added `BenchmarkCache_MemoryLeak` in
`internal/memorycache/cache_test.go` to validate memory usage and
performance of the cache over time, simulating the memory leak scenario.

## Benchmark Results
To validate the fix, we ran `BenchmarkCache_MemoryLeak` which reports an
`items_left` metric showing how many items remain in the cache at the
end of the run.

**Before the fix (with the unconditional break):**
```text
BenchmarkCache_MemoryLeak-16    	 3703683	      1384 ns/op	    456112 items_left	     101 B/op	       3 allocs/op
```
*Result: 456,112 items left in the cache. The cleanup goroutine was
dead, causing items to accumulate indefinitely.*

**After the fix (unconditional break removed):**
```text
BenchmarkCache_MemoryLeak-16    	 3742993	      1387 ns/op	         1.000 items_left	      90 B/op	       3 allocs/op
```
*Result: Only 1 item left in the cache. The cleanup timer runs
continuously and successfully evicts expired items.*

## Testing
- Ran `go test -v -run TestCache_CleanupTimer ./internal/memorycache`
which passes.
- Ran `go test -bench=BenchmarkCache_MemoryLeak -benchmem -run=^$
./internal/memorycache` which passes and shows stable memory usage.
- Verified the build passes with `make build`.

---------

Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-17147" title="TT-17147"
target="_blank">TT-17147</a>
</summary>

|         |    |
|---------|----|
| Status  | Closed |
| Summary | Memory Leak in internal/memorycache.Cache.startCleanupTimer
|

Generated at: 2026-05-15 11:48:19

</details>

<!---TykTechnologies/jira-linter ends here-->

Co-authored-by: probelabs[bot] <219682034+probelabs[bot]@users.noreply.github.com>
Co-authored-by: Leonid Bugaev <[email protected]>
Co-authored-by: Matias <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants