[TT-3738] Implement rate limit smoothing#6295
Conversation
|
API Changes --- prev.txt 2024-05-30 09:59:19.462649629 +0000
+++ current.txt 2024-05-30 09:59:16.090633675 +0000
@@ -1666,6 +1666,81 @@
Value string `bson:"value" json:"value"`
}
+type RateLimitSmoothing struct {
+ // Enabled indicates if rate limit smoothing is active.
+ Enabled bool `json:"enabled" bson:"enabled"`
+
+ // Threshold is the request rate above which smoothing is applied.
+ Threshold int64 `json:"threshold" bson:"threshold"`
+
+ // Trigger is the step factor determining when smoothing events trigger.
+ Trigger float64 `json:"trigger" bson:"trigger"`
+
+ // Step is the increment/decrement for adjusting the rate limit.
+ Step int64 `json:"step" bson:"step"`
+
+ // Delay is the minimum time between rate limit changes (in seconds).
+ Delay int64 `json:"delay" bson:"delay"`
+}
+ RateLimitSmoothing holds the rate smoothing configuration.
+
+ Rate Limit Smoothing is a mechanism to dynamically adjust the request rate
+ limits based on the current traffic patterns. It helps in managing request
+ spikes by gradually increasing or decreasing the rate limit instead of
+ making abrupt changes or blocking requests excessively.
+
+ Once the rate limit smoothing triggers an allowance change, one of the
+ following events is emitted:
+
+ - `RateLimitSmoothingUp` when the allowance increases -
+ `RateLimitSmoothingDown` when the allowance decreases
+
+ Events are emitted based on the configuration:
+
+ - `enabled` (boolean) to enable or disable rate limit smoothing -
+ `threshold` after which to apply smoothing (minimum rate for window) -
+ `trigger` configures at which fraction of a step a smoothing event is
+ emitted - `step` is the value by which the rate allowance will get adjusted
+ - `delay` is the amount of seconds between smoothing updates
+
+ This is used to compute a request allowance. The request allowance will
+ be smoothed between `threshold`, and the defined rate limits (maximum).
+ The request allowance will be updated internally every `delay` seconds.
+
+ The `step * trigger` value is substracted from the request allowance, and
+ if your request rate goes above that, then a RateLimitSmoothingUp event is
+ emitted and the allowance is increased by `step`. A RateLimitSmoothingDown
+ event is emitted when the request rate drops one step below that, and the
+ allowance then decreases by step.
+
+ For any allowance, events are emitted based on the following calculations:
+
+ - When the request rate rises above `allowance - (step * trigger)`, a
+ RateLimitSmoothingUp event is emitted and allowance increases by `step`.
+ - When the request rate falls below `allowance - (step + step * trigger)`,
+ a RateLimitSmoothingDown event is emitted and allowance decreases by
+ `step`.
+
+ Example: Allowance: 600, Current rate: 500, Step: 100, Trigger: 0.5
+
+ - To trigger a RateLimitSmoothingUp event, the request rate must exceed:
+ Allowance - (Step * Trigger) Calculation: 600 - (100 * 0.5) = 550
+ Exceeding a request rate of 550 will increase the allowance to 700
+ (Allowance + Step).
+
+ - To trigger a RateLimitSmoothingDown event, the request rate must fall
+ below: Allowance - (Step + (Step * Trigger)) Calculation: 600 - (100
+ + (100 * 0.5)) = 450 As the request rate falls below 450, that will
+ decrease the allowance to 500 (Allowance - Step).
+
+func (r *RateLimitSmoothing) Err() error
+ Err checks the rate limit smoothing configuration for validity and returns
+ an error if it is not valid. It checks for a nil value, the enabled flag and
+ valid values for each setting.
+
+func (r *RateLimitSmoothing) Valid() bool
+ Valid will return true if the rate limit smoothing should be applied.
+
type RequestHeadersRewriteConfig struct {
Value string `json:"value" bson:"value"`
Remove bool `json:"remove" bson:"remove"`
@@ -5854,6 +5929,10 @@
// The standard rate limiter offers similar performance as the sentinel-based limiter. This is disabled by default.
EnableSentinelRateLimiter bool `json:"enable_sentinel_rate_limiter"`
+ // EnableRateLimitSmoothing enables or disables rate smoothing. The rate smoothing is only supported on the
+ // Redis Rate Limiter, or the Sentinel Rate Limiter, as both algorithms implement a sliding log.
+ EnableRateLimitSmoothing bool `json:"enable_rate_limit_smoothing"`
+
// An enhancement for the Redis and Sentinel rate limiters, that offers a significant improvement in performance by not using transactions on Redis rate-limit buckets.
EnableNonTransactionalRateLimiter bool `json:"enable_non_transactional_rate_limiter"`
@@ -6752,6 +6831,7 @@
const (
SessionData Key = iota
+ // Deprecated: UpdateSession was used to trigger a session update, use *SessionData.Touch instead.
UpdateSession
AuthToken
HashedAuthToken
@@ -7021,9 +7101,6 @@
const (
// EventQuotaExceeded is an alias maintained for backwards compatibility.
EventQuotaExceeded = event.QuotaExceeded
- // EventRateLimitExceeded is an alias maintained for backwards compatibility.
- EventRateLimitExceeded = event.RateLimitExceeded
-
// EventAuthFailure is an alias maintained for backwards compatibility.
EventAuthFailure = event.AuthFailure
// EventKeyExpired is an alias maintained for backwards compatibility.
@@ -7254,7 +7331,7 @@
EnsureTransport sanitizes host/protocol pairs and returns a valid URL.
func GenerateTestBinaryData() (buf *bytes.Buffer)
-func GetAccessDefinitionByAPIIDOrSession(currentSession *user.SessionState, api *APISpec) (accessDef *user.AccessDefinition, allowanceScope string, err error)
+func GetAccessDefinitionByAPIIDOrSession(session *user.SessionState, api *APISpec) (accessDef *user.AccessDefinition, allowanceScope string, err error)
func GetTLSClient(cert *tls.Certificate, caCert []byte) *http.Client
func GetTLSConfig(cert *tls.Certificate, caCert []byte) *tls.Config
func InitTestMain(ctx context.Context, m *testing.M) int
@@ -9723,12 +9800,12 @@
func (l *SessionLimiter) Context() context.Context
-func (l *SessionLimiter) ForwardMessage(r *http.Request, currentSession *user.SessionState, rateLimitKey string, quotaKey string, store storage.Handler, enableRL, enableQ bool, api *APISpec, dryRun bool) sessionFailReason
+func (l *SessionLimiter) ForwardMessage(r *http.Request, session *user.SessionState, rateLimitKey string, quotaKey string, store storage.Handler, enableRL, enableQ bool, api *APISpec, dryRun bool) sessionFailReason
ForwardMessage will enforce rate limiting, returning a non-zero
sessionFailReason if session limits have been exceeded. Key values to manage
rate are Rate and Per, e.g. Rate of 10 messages Per 10 seconds
-func (l *SessionLimiter) RedisQuotaExceeded(r *http.Request, currentSession *user.SessionState, quotaKey, scope string, limit *user.APILimit, store storage.Handler, hashKeys bool) bool
+func (l *SessionLimiter) RedisQuotaExceeded(r *http.Request, session *user.SessionState, quotaKey, scope string, limit *user.APILimit, store storage.Handler, hashKeys bool) bool
type SlaveDataCenter struct {
SlaveOptions config.SlaveOptionsConfig
@@ -11920,11 +11997,22 @@
QuotaRemaining int64 `json:"quota_remaining" msg:"quota_remaining"`
QuotaRenewalRate int64 `json:"quota_renewal_rate" msg:"quota_renewal_rate"`
SetBy string `json:"-" msg:"-"`
+
+ // Smoothing contains rate limit smoothing settings.
+ Smoothing *apidef.RateLimitSmoothing `json:"smoothing" bson:"smoothing"`
}
APILimit stores quota and rate limit on ACL level (per API)
+func (g *APILimit) Duration() time.Duration
+ Duration returns the time between two allowed requests at the defined rate.
+ It's used to decide which rate limit has a bigger allowance.
+
func (limit APILimit) IsEmpty() bool
+func (g *APILimit) Less(in APILimit) bool
+ Less will return true if the receiver has a smaller duration between
+ requests than `in`.
+
type AccessDefinition struct {
APIName string `json:"api_name" msg:"api_name"`
APIID string `json:"api_id" msg:"api_id"`
@@ -12000,9 +12088,14 @@
LastUpdated string `bson:"last_updated" json:"last_updated"`
MetaData map[string]interface{} `bson:"meta_data" json:"meta_data"`
GraphQL map[string]GraphAccessDefinition `bson:"graphql_access_rights" json:"graphql_access_rights"`
+
+ // Smoothing contains rate limit smoothing settings.
+ Smoothing *apidef.RateLimitSmoothing `json:"smoothing" bson:"smoothing"`
}
Policy represents a user policy swagger:model
+func (p *Policy) APILimit() APILimit
+
type PolicyPartitions struct {
Quota bool `bson:"quota" json:"quota"`
RateLimit bool `bson:"rate_limit" json:"rate_limit"`
@@ -12053,6 +12146,10 @@
SessionLifetime int64 `bson:"session_lifetime" json:"session_lifetime"`
KeyID string `json:"-"`
+
+ // Smoothing contains rate limit smoothing settings.
+ Smoothing *apidef.RateLimitSmoothing `json:"smoothing" bson:"smoothing"`
+
// Has unexported fields.
}
SessionState objects represent a current API session, mainly used
@@ -12065,6 +12162,9 @@
func NewSessionState() *SessionState
+func (s *SessionState) APILimit() APILimit
+ APILimit returns an user.APILimit from the session data.
+
func (s SessionState) Clone() SessionState
Clone returns a fresh copy of s
@@ -12077,6 +12177,10 @@
func (s *SessionState) IsBasicAuth() bool
IsBasicAuth returns whether the key is basic auth or not.
+func (s *SessionState) IsModified() bool
+ IsModified will return true if session has been modified to trigger an
+ update.
+
func (s *SessionState) KeyHash() string
func (s *SessionState) KeyHashEmpty() bool
@@ -12100,6 +12204,9 @@
For backwards compatibility reasons, this falls back to ApplyPolicyID if
ApplyPolicies is empty.
+func (s *SessionState) Reset()
+ Reset marks the session as not modified, skipping related updates.
+
func (s *SessionState) SetCustomPolicies(list []Policy)
func (s *SessionState) SetKeyHash(hash string)
@@ -12109,3 +12216,6 @@
func (s *SessionState) TagsFromMetadata(data map[string]interface{}) (updateSession bool)
TagsFromMetadata updates the session state with the tags from the metadata.
+func (s *SessionState) Touch()
+ Touch marks the session as modified, indicating that it should be updated.
+ |
PR Review 🔍
Code feedback:
|
PR Code Suggestions ✨
|
| log.Debug("Renewal Date is: ", renewalDate) | ||
| log.Debug("As epoch: ", quotaRenews) | ||
| log.Debug("Session: ", currentSession) | ||
| log.Debug("Session: ", session) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information
PR Agent Walkthrough 🤖Welcome to the PR Agent, an AI-powered tool for automated pull request analysis, feedback, suggestions and more. Here is a list of tools you can use to interact with the PR Agent:
(1) Note that each tool be triggered automatically when a new PR is opened, or called manually by commenting on a PR. (2) Tools marked with [*] require additional parameters to be passed. For example, to invoke the |
|
/add_docs |
48bd76c to
78be36f
Compare
ecfe25e to
282c373
Compare
💥 CI tests failed 🙈git-stateall okPlease look at the run or in the Checks tab. |
1 similar comment
💥 CI tests failed 🙈git-stateall okPlease look at the run or in the Checks tab. |
|
### **User description**
The PR does the following:
- adds a hook to the sliding log implementation to evaluate rate limit
smoothing
- adds event functions which allow to trigger events via request
context, to bubble them up (avoid globals)
- most relevant changes needed to be done in `session_manager.go`, a
breakdown below:
**Data model changes:**
- Extend apidef with `RateLimitSmoothing` type, holding some data model
logic (validation, allowance)
- Extend config with EnableRateLimitSmoothing flag, update cli/linter
schema
- Extend user.SessionData with Touch/Reset/IsModified, update old
context code
- ctx removes UpdateSession
- gateway removes ctx* scheduled session update functions
- Add RateLimitSmoothingUp/Down events in internal/event
**Internal changes:**
- `internal/event` adds APIs to attach events to requests/contexts,
event readable names
- `internal/rate/sliding_log.go` adds hooks into our sliding log
implementation (`SmoothingFn`). This is required to evaluate smooting.
- `internal/rate/smoothing.go` adds a standalone Smoothing() function
implementing smoothing logic and triggering a session update if the
allowance changes
**Gateway pkg changes:**
- Most relevant changes in `session_manager.go`.
- Updated `middleware.go`, `mw_api_rate_limit.go` and
`mw_rate_limiting.go` to emit new events, deduplicate some code around
handling RateLimitExceeded events.
- Updated `event_handler*` to add error handling, fixes a test issue
where webhook event template rendering would fail if event.Event would
implement a fmt.Stringer.
- Remove now unused context schedule session update functions, update
usage of SessionData.
___
### **PR Type**
Enhancement, Other
___
### **Description**
- Added `RateLimitSmoothing` struct and validation methods to hold and
validate rate smoothing configuration.
- Introduced `EnableRateLimitSmoothing` configuration option in
`config/rate_limit.go`.
- Integrated rate limit smoothing in `SessionLimiter` by adding
`rateLimitSmoothing` method and modifying `doRollingWindowWrite`.
- Added new rate limiter events `RateLimitSmoothingUp` and
`RateLimitSmoothingDown` in `internal/event/event.go`.
- Implemented smoothing function in `SlidingLog` rate limiter by adding
`SmoothingFn` type and modifying `SlidingLog` struct and methods.
- Updated `cli/linter/schema.json` to include
`enable_rate_limit_smoothing` configuration option.
___
### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement
</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>rate_limit_smoothing.go</strong><dd><code>Add rate
limit smoothing configuration and validation</code>
</dd></summary>
<hr>
apidef/rate_limit_smoothing.go
<li>Added <code>RateLimitSmoothing</code> struct to hold rate smoothing
configuration.<br> <li> Implemented validation methods
<code>Valid</code> and <code>Err</code> for rate smoothing
<br>configuration.<br>
</details>
</td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6295/files#diff-479528161031eaa34a0cc82305e13e9603d770b5f069314d7615ec99361790c1">+53/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>rate_limit.go</strong><dd><code>Add configuration
option for rate limit smoothing</code>
</dd></summary>
<hr>
config/rate_limit.go
- Added `EnableRateLimitSmoothing` configuration option.
</details>
</td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6295/files#diff-375bf116f8d6527c50d7591d7cb01e8f821b22df4a4ca18b4da4c6f0d526f18e">+4/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>session_manager.go</strong><dd><code>Integrate rate
limit smoothing in session manager</code>
</dd></summary>
<hr>
gateway/session_manager.go
<li>Added <code>rateLimitSmoothing</code> method to
<code>SessionLimiter</code>.<br> <li> Modified
<code>doRollingWindowWrite</code> to use the new smoothing function.<br>
</details>
</td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6295/files#diff-e6b40a285464cd86736e970c4c0b320b44c75b18b363d38c200e9a9d36cdabb6">+19/-24</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>event.go</strong><dd><code>Add new rate limiter events
for smoothing</code>
</dd></summary>
<hr>
internal/event/event.go
<li>Added new rate limiter events: <code>RateLimitSmoothingUp</code> and
<br><code>RateLimitSmoothingDown</code>.<br>
</details>
</td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6295/files#diff-3d64b81c3937b899f363a9ce6bd4dbca0325ce8c20a67b8fb763d0c798cef93e">+12/-2</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>sliding_log.go</strong><dd><code>Implement smoothing
function in sliding log rate limiter</code> </dd></summary>
<hr>
internal/rate/sliding_log.go
<li>Added <code>SmoothingFn</code> type for rate limiter decision.<br>
<li> Modified <code>SlidingLog</code> to include smoothing function.<br>
<li> Implemented <code>Do</code> method to use smoothing function for
rate limiting.<br>
</details>
</td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6295/files#diff-c2300c4284e73a74e69f500492a1f9b15cac918acf19f05d105d3134e246b450">+25/-5</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>schema.json</strong><dd><code>Add rate limit smoothing
option to schema</code>
</dd></summary>
<hr>
cli/linter/schema.json
<li>Added <code>enable_rate_limit_smoothing</code> configuration option
to schema.<br>
</details>
</td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6295/files#diff-103cec746d3e61d391c5a67c171963f66fea65d651d704d5540e60aa5d574f46">+3/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
___
> 💡 **PR-Agent usage**:
>Comment `/help` on the PR to get a list of all available PR-Agent tools
and their descriptions
---------
Co-authored-by: Tit Petric <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: andyo-tyk <[email protected]>
Co-authored-by: dcs3spp <[email protected]>




User description
The PR does the following:
session_manager.go, a breakdown below:Data model changes:
RateLimitSmoothingtype, holding some data model logic (validation, allowance)Internal changes:
internal/eventadds APIs to attach events to requests/contexts, event readable namesinternal/rate/sliding_log.goadds hooks into our sliding log implementation (SmoothingFn). This is required to evaluate smooting.internal/rate/smoothing.goadds a standalone Smoothing() function implementing smoothing logic and triggering a session update if the allowance changesGateway pkg changes:
session_manager.go.middleware.go,mw_api_rate_limit.goandmw_rate_limiting.goto emit new events, deduplicate some code around handling RateLimitExceeded events.event_handler*to add error handling, fixes a test issue where webhook event template rendering would fail if event.Event would implement a fmt.Stringer.PR Type
Enhancement, Other
Description
RateLimitSmoothingstruct and validation methods to hold and validate rate smoothing configuration.EnableRateLimitSmoothingconfiguration option inconfig/rate_limit.go.SessionLimiterby addingrateLimitSmoothingmethod and modifyingdoRollingWindowWrite.RateLimitSmoothingUpandRateLimitSmoothingDownininternal/event/event.go.SlidingLograte limiter by addingSmoothingFntype and modifyingSlidingLogstruct and methods.cli/linter/schema.jsonto includeenable_rate_limit_smoothingconfiguration option.Changes walkthrough 📝
rate_limit_smoothing.go
Add rate limit smoothing configuration and validationapidef/rate_limit_smoothing.go
RateLimitSmoothingstruct to hold rate smoothing configuration.ValidandErrfor rate smoothingconfiguration.
rate_limit.go
Add configuration option for rate limit smoothingconfig/rate_limit.go
EnableRateLimitSmoothingconfiguration option.session_manager.go
Integrate rate limit smoothing in session managergateway/session_manager.go
rateLimitSmoothingmethod toSessionLimiter.doRollingWindowWriteto use the new smoothing function.event.go
Add new rate limiter events for smoothinginternal/event/event.go
RateLimitSmoothingUpandRateLimitSmoothingDown.sliding_log.go
Implement smoothing function in sliding log rate limiterinternal/rate/sliding_log.go
SmoothingFntype for rate limiter decision.SlidingLogto include smoothing function.Domethod to use smoothing function for rate limiting.schema.json
Add rate limit smoothing option to schemacli/linter/schema.json
enable_rate_limit_smoothingconfiguration option to schema.