Skip to content

Revert TT-16259#8307

Merged
lghiur merged 1 commit into
masterfrom
TT-16259-security-unable-to-delete-session-that-uses-a-quota-and-is-under-heavy-load-revert
Jun 15, 2026
Merged

Revert TT-16259#8307
lghiur merged 1 commit into
masterfrom
TT-16259-security-unable-to-delete-session-that-uses-a-quota-and-is-under-heavy-load-revert

Conversation

@shults

@shults shults commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

…ota and is under heavy load [Plan A] (#8254)"

This reverts commit 989c668.

Description

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

@shults
shults requested review from MaciekMis and kofoworola June 11, 2026 09:28
@probelabs

probelabs Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

This pull request reverts commit 989c668b, which was introduced in PR #8254 to address issue TT-16259. The original change aimed to fix a race condition where a session token, particularly one with a quota under heavy load, could be recreated by an in-flight request immediately after being deleted. The fix involved introducing a conditional "set-if-exists" operation (SetKeyEx) at the storage layer.

This revert completely removes the SetKeyEx logic and its associated code, suggesting the original implementation may have caused unintended side effects or stability issues. The session update mechanism is now returned to a simpler, non-atomic "create-or-overwrite" operation, which unfortunately re-introduces the original race condition.

Files Changed Analysis

The changes are overwhelmingly deletions (~1084 lines removed vs. ~56 added), which is characteristic of a revert. Key changes include:

  • Removal of SetKeyEx and SetRawKeyEx: These methods have been removed from the storage.Handler interface and all its implementations (redis_cluster.go, rpc_storage_handler.go, dummy.go, mdcb_storage.go).
  • Reverted Session Update Logic: In gateway/auth_manager.go, the UpdateSession method no longer distinguishes between new and existing sessions based on an isRestored flag and unconditionally uses SetKey.
  • Simplified Session State: The isRestored flag and its related methods have been removed from user/session.go.
  • Test and Utility Removal: A significant number of tests related to the reverted functionality have been deleted. The log/hook.go utility file and its tests have also been removed entirely.
  • Dependency Downgrade: go.mod shows a downgrade of github.com/TykTechnologies/storage from v1.3.2 to v1.3.1, aligning with the removal of the feature from the storage dependency.

Architecture & Impact Assessment

  • What this PR accomplishes: It rolls back the fix for the session deletion race condition, effectively re-introducing the bug from TT-16259.
  • Key technical changes introduced: The primary change is the removal of the atomic "update-if-exists" operation from the storage layer. Session updates will now always perform a "create-or-overwrite" operation.
  • Affected system components: The main components affected are Session Management (gateway/auth_manager.go) and the Storage Abstraction Layer (storage/). The behavior of session persistence under high concurrent loads is directly impacted.

Session Update Flow Change

This revert simplifies the session update logic but re-introduces the race condition. The diagram below illustrates the change.

Before (Logic being reverted):

sequenceDiagram
    participant SM as SessionManager
    participant SS as SessionState
    participant Store as Storage
    SM->>SS: IsRestored()?
    alt Existing Session
        SS-->>SM: true
        SM->>Store: SetKeyEx(key, session) # Update only if key exists
    else New Session
        SS-->>SM: false
        SM->>Store: SetKey(key, session) # Create or overwrite key
    end
Loading

After (This PR's Logic):

sequenceDiagram
    participant SM as SessionManager
    participant Store as Storage
    SM->>Store: SetKey(key, session) # Always create or overwrite
Loading

Scope Discovery & Context Expansion

  • The most significant impact of this revert is the reintroduction of the bug from TT-16259. The system is again vulnerable to a race condition where a session deleted by an administrator can be recreated by an in-flight request, effectively bypassing the deletion. This has security and resource management implications.
  • The revert was likely necessary due to unforeseen issues with the original implementation, such as performance degradation, deadlocks, or bugs in specific storage backends (the removal of complex retry logic in rpc_storage_handler.go might be a clue).
  • The next step for the team should be to investigate a new, more stable solution for the original race condition, as the problem of atomically updating sessions without resurrecting them remains a critical issue to be solved.
Metadata
  • Review Effort: 3 / 5
  • Primary Label: chore

Powered by Visor from Probelabs

Last updated: 2026-06-15T09:36:22.245Z | Triggered by: pr_updated | Commit: 844afc8

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs

probelabs Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Security Issues (1)

Severity Location Issue
🔴 Critical gateway/auth_manager.go:139-145
This change re-introduces a race condition vulnerability (previously tracked as TT-16259) that allows for an access control bypass. By reverting to a simple 'create-or-overwrite' (SetKey/SetRawKey) operation for session updates, an in-flight request can recreate a session that has just been deleted by an administrator. This makes session revocation unreliable under load, as a compromised key could be resurrected by a concurrent request, allowing continued unauthorized access.
💡 SuggestionThe original fix using a 'set-if-exists' operation (SetKeyEx/SetRawKeyEx) should be restored or replaced with an alternative solution that atomically updates sessions without recreating them if they have been deleted. If the original implementation had side effects, a different approach to mitigate the race condition is necessary before merging. Reverting this security fix without a replacement is highly discouraged.

Performance Issues (2)

Severity Location Issue
🟠 Error gateway/session_manager.go:499
The removed comment `// todo: change to lua script, it does not works like author suppose` highlighted a bug in the Redis pipelining logic for quota management. The code checks the result of an `INCR` command within the pipeline function to conditionally add an `EXPIRE` command. This does not work as intended because the result is not available until the pipeline executes. Consequently, the condition `res.Val() == 1` is likely always false, preventing `pipe.Expire` from being called. This causes quota keys to persist indefinitely, leading to a memory leak.
💡 SuggestionThe underlying code, which causes a memory leak, should be fixed. The logic needs to be atomic. Using a Lua script is the recommended approach to correctly implement a conditional `EXPIRE` after an `INCR` operation in Redis. At a minimum, the comment highlighting this critical bug should be restored until a fix is implemented.
🟡 Warning gateway/auth_manager.go:130-148
The revert to an unconditional `SetKey` (create or overwrite) for session updates re-introduces a race condition. Under heavy load, an in-flight request can recreate a session immediately after it has been deleted. This leads to resource management issues, as sessions intended for deletion are resurrected, continuing to consume memory in the session store and allowing continued use of gateway resources.
💡 SuggestionWhile the original fix was reverted, the underlying race condition remains a risk. A new approach for atomic session updates should be implemented. Using a Lua script for Redis operations could provide the necessary atomicity (e.g., update-if-exists) without the performance or stability issues of the reverted implementation.

Quality Issues (2)

Severity Location Issue
🔴 Critical gateway/auth_manager.go:125-144
This PR reverts a fix for a race condition (TT-16259) where a deleted session can be recreated by an in-flight request. The `UpdateSession` function now uses a non-atomic "create-or-overwrite" operation (`SetKey`/`SetRawKey`) instead of a "set-if-exists" operation. This re-introduces a security vulnerability where session deletion can be bypassed under heavy load.
💡 SuggestionWhile this revert may be necessary to address other issues, the underlying race condition is critical and should be addressed with high priority. A new fix that is more stable than the reverted one should be implemented.
🟡 Warning gateway/api_definition.go:1999-2003
The `sessionStore` parameter is added to the `APISpec.Init` function signature but is not used within the function's body. Call sites in `api_loader.go` and tests are updated to pass this parameter, but it is then discarded.
💡 SuggestionIf the `sessionStore` is intended to be used for session management, it should be passed to the relevant component's initializer (e.g., `a.AuthManager.Init`). If it is not needed, it should be removed from the function signature and all call sites to avoid confusion and code smell.

Powered by Visor from Probelabs

Last updated: 2026-06-15T09:36:15.302Z | Triggered by: pr_updated | Commit: 844afc8

💡 TIP: You can chat with Visor using /visor ask <your question>

@kofoworola kofoworola added the deps-reviewed Dependency changes reviewed and approved for CI execution label Jun 11, 2026
@kofoworola
kofoworola force-pushed the TT-16259-security-unable-to-delete-session-that-uses-a-quota-and-is-under-heavy-load-revert branch from 7bfc7c3 to 03ec7c1 Compare June 11, 2026 16:11
@radkrawczyk radkrawczyk reopened this Jun 12, 2026
@kofoworola
kofoworola force-pushed the TT-16259-security-unable-to-delete-session-that-uses-a-quota-and-is-under-heavy-load-revert branch 2 times, most recently from 75220f1 to f4ea04f Compare June 12, 2026 16:30
…ota and is under heavy load [Plan A] (#8254)"

This reverts commit 989c668.
@kofoworola
kofoworola force-pushed the TT-16259-security-unable-to-delete-session-that-uses-a-quota-and-is-under-heavy-load-revert branch from f4ea04f to 844afc8 Compare June 15, 2026 09:33
@kofoworola
kofoworola enabled auto-merge (squash) June 15, 2026 09:33
@sentinelone-cnapp-eu1

Copy link
Copy Markdown

SentinelOne CNS Hardcoded Secret Detector
✅ Congratulations, your code is safe

SentinelOne CNS is a cloud-agnostic, agentless CSPM & CWPP solution that continuously detects and prevents vulnerabilities that have the highest probability of being exploited in Azure, AWS, Google Cloud, and Kubernetes.

@github-actions

Copy link
Copy Markdown
Contributor

🚨 Jira Linter Failed

Commit: 844afc8
Failed at: 2026-06-15 09:34:27 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details
failed to get Jira issue: failed to fetch Jira issue TT-16259: Issue does not exist or you do not have permission to see it.: request failed. Please analyze the request body for more details. Status code: 404

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@lghiur
lghiur disabled auto-merge June 15, 2026 11:30
@lghiur
lghiur merged commit 2675376 into master Jun 15, 2026
48 of 56 checks passed
@lghiur
lghiur deleted the TT-16259-security-unable-to-delete-session-that-uses-a-quota-and-is-under-heavy-load-revert branch June 15, 2026 11:30
@lghiur

lghiur commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

@kofoworola asked for this to be merged since this is an agreed revert inside the team

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deps-reviewed Dependency changes reviewed and approved for CI execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants