feat: Handle rate limit errors by respecting X-RateLimit-Reset and Retry-After headers and returning cached queue length by andrewhibbert · Pull Request #7684 · kedacore/keda

andrewhibbert · 2026-04-24T14:47:30Z

Provide a description of what has been changed

Checklist

[N/A] When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added (if applicable)
Ensure make generate-scalers-schema has been run to update any outdated generated files
Changelog has been updated and is aligned with our changelog requirements, only when the change impacts end users
[N/A] A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #7683

Relates to: kedacore/keda-docs#1749

Follow on from #7192 and comment #7192 (review)

…g cached queue length Signed-off-by: andrewhibbert <[email protected]>

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

snyk-io · 2026-04-24T14:47:45Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

github-actions · 2026-04-24T14:47:51Z

Thank you for your contribution! 🙏

Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected.

While you are waiting, make sure to:

Add an entry in our changelog in alphabetical order and link related issue
Update the documentation, if needed
Add unit & e2e tests for your changes
GitHub checks are passing
Is the DCO check failing? Here is how you can fix DCO issues

Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient.

Learn more about our contribution guide.

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

rickbrouwer · 2026-04-25T06:56:42Z

/run-e2e github*
Update: You can check the progress here

rickbrouwer · 2026-04-25T10:15:09Z

+		s.logger.V(1).Info(fmt.Sprintf(
+			"Github API rate limit exceeded. Cached queue length: %d, last checked at %s",
+			s.previousQueueLength,
+			s.previousQueueLengthTime,
+		))


One thing I was wondering about, when we fall back to the cached queue length, the only signal to the operator is a V(1) log line. Could we also create an event? That way tools like Argo CD would pick it up and show it on the resource directly, which feels like a much better fit here. Not every user has access to the KEDA operator logs (or knows to look there), and I think this is actually an important signal. Without it, unexpected scaling behaviour can be hard to explain. Wdyt?

(As a smaller secondary thing, it might
be worth bumping the V(1) log to V(0) so there's continuous visibility in the logs while we're rate-limited, not just at the transition. But the event is the main thing.)

could the log be spammy? IDK how often this happens so maybe it's too much? (I have 0 idea, just asking) the event is 100% worth it ❤️

JorTurFer · 2026-04-26T11:30:56Z

+func (s *githubRunnerScaler) getRateLimit(header http.Header) (RateLimit, error) {
+	var retryAfterTime time.Time
+
+	remainingStr := header.Get("X-RateLimit-Remaining")


is this header always set? Do we need to check if it's present before trying to cast it?

JorTurFer · 2026-04-26T11:31:00Z

+		return RateLimit{}, fmt.Errorf("failed to parse X-RateLimit-Remaining header: %w", err)
+	}
+
+	resetStr := header.Get("X-RateLimit-Reset")


is this header always set? Do we need to check if it's present before trying to cast it?

JorTurFer · 2026-04-26T11:34:09Z

+		s.logger.V(1).Info(fmt.Sprintf(
+			"Github API rate limit exceeded. Cached queue length: %d, last checked at %s",
+			s.previousQueueLength,
+			s.previousQueueLengthTime,
+		))


could the log be spammy? IDK how often this happens so maybe it's too much? (I have 0 idea, just asking) the event is 100% worth it ❤️

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

rickbrouwer · 2026-04-27T15:27:12Z

Thanks for adding the event! The event fires on every reconcile while we're rate-limited. That could be a lot of events, because the message includes "last checked at ", so each event is unique, so Kubernetes' event aggregation won't dedupe them. Would it be worth only emitting on state transitions? So once when we go into
the rate-limited state and possibly once when we recover? That would also make the event a more meaningful signal.

Further, does EventTypeWarning fit the cache-hit path? The scaler is working as designed there (degraded but functional), so I was wondering if EventTypeNormal might be a better match. The no-cache path does feel like a real warning though. And on that no-cache path, there we're already returning an error which the reconciler surfaces as an event itself, right? So we might be double-emitting there.

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

andrewhibbert · 2026-04-27T15:53:18Z

Thanks for adding the event! The event fires on every reconcile while we're rate-limited. That could be a lot of events, because the message includes "last checked at ", so each event is unique, so Kubernetes' event aggregation won't dedupe them. Would it be worth only emitting on state transitions? So once when we go into the rate-limited state and possibly once when we recover? That would also make the event a more meaningful signal.

Further, does EventTypeWarning fit the cache-hit path? The scaler is working as designed there (degraded but functional), so I was wondering if EventTypeNormal might be a better match. The no-cache path does feel like a real warning though. And on that no-cache path, there we're already returning an error which the reconciler surfaces as an event itself, right? So we might be double-emitting there.

The "last checked at " is the last time it last cached inside GetWorkflowQueueLength, so once rate limited this time won't change.

I have changed the event to be EventTypeNormal and removed the no cache event

rickbrouwer · 2026-04-27T16:49:16Z

Thanks for the explanation. But a separate thing I hadn't spotted before: is "last checked at" correct or is a bit misleading? Would something like "last successful cache at" be more accurate?

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

andrewhibbert · 2026-04-27T17:00:50Z

Thanks for the explanation. But a separate thing I hadn't spotted before: is "last checked at" correct or is a bit misleading? Would something like "last successful cache at" be more accurate?

Agreed, I have changed it to "last successful cache at"

rickbrouwer · 2026-04-27T17:08:38Z

/run-e2e github*
Update: You can check the progress here

rickbrouwer · 2026-04-30T12:58:35Z

/run-e2e github*
Update: You can check the progress here

rickbrouwer · 2026-04-30T13:58:56Z

/run-e2e github*
Update: You can check the progress here

JorTurFer · 2026-05-03T18:19:10Z

loooooool. merging kedacore/keda-docs#1749 I've closed directly this one 🤦

andrewhibbert added 2 commits April 24, 2026 15:45

feat: Handle rate limit errors by respecting and headers and returnin…

5bf281c

…g cached queue length Signed-off-by: andrewhibbert <[email protected]>

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

8fd46f5

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

andrewhibbert requested a review from a team as a code owner April 24, 2026 14:47

keda-automation requested a review from a team April 24, 2026 14:47

andrewhibbert changed the title ~~Feat rate limit handling~~ feat: Handle rate limit errors by respecting X-RateLimit-Reset and Retry-After headers and returning cached queue length Apr 24, 2026

andrewhibbert mentioned this pull request Apr 24, 2026

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Retry-After headers and returning cached queue length kedacore/keda-docs#1749

Merged

1 task

rickbrouwer reviewed Apr 24, 2026

View reviewed changes

Comment thread pkg/scalers/github_runner_scaler.go Outdated

Comment thread pkg/scalers/github_runner_scaler.go Outdated

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

f500579

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

keda-automation requested a review from a team April 24, 2026 17:03

JorTurFer reviewed Apr 24, 2026

View reviewed changes

Comment thread pkg/scalers/github_runner_scaler.go Outdated

Comment thread pkg/scalers/github_runner_scaler.go Outdated

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

3c20d5d

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

keda-automation requested a review from a team April 24, 2026 18:12

rickbrouwer reviewed Apr 25, 2026

View reviewed changes

JorTurFer reviewed Apr 26, 2026

View reviewed changes

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

3479beb

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

keda-automation requested a review from a team April 27, 2026 11:10

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

bb09b12

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

andrewhibbert added 2 commits April 27, 2026 16:44

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

9dfd176

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

fa452b4

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

feat: Handle rate limit errors by respecting X-RateLimit-Reset and Re…

2986536

…try-After headers and returning cached queue length Signed-off-by: andrewhibbert <[email protected]>

rickbrouwer approved these changes Apr 27, 2026

View reviewed changes

rickbrouwer added the Awaiting/2nd-approval This PR needs one more approval review label Apr 27, 2026

rickbrouwer self-requested a review April 30, 2026 12:58

rickbrouwer removed the Awaiting/2nd-approval This PR needs one more approval review label Apr 30, 2026

rickbrouwer approved these changes Apr 30, 2026

View reviewed changes

rickbrouwer added the Awaiting/2nd-approval This PR needs one more approval review label Apr 30, 2026

JorTurFer approved these changes May 3, 2026

View reviewed changes

JorTurFer removed the request for review from a team May 3, 2026 18:18

JorTurFer enabled auto-merge (squash) May 3, 2026 18:18

JorTurFer closed this in kedacore/keda-docs#1749 May 3, 2026

auto-merge was automatically disabled May 3, 2026 18:18
Pull request was closed

JorTurFer reopened this May 3, 2026

keda-automation requested review from a team May 3, 2026 18:19

JorTurFer merged commit 5f89276 into kedacore:main May 3, 2026
41 checks passed

rickbrouwer removed the Awaiting/2nd-approval This PR needs one more approval review label May 6, 2026

Uh oh!

Conversation

andrewhibbert commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

snyk-io Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rickbrouwer commented Apr 25, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickbrouwer Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JorTurFer Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

JorTurFer Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

JorTurFer Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

JorTurFer Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

rickbrouwer commented Apr 27, 2026

Uh oh!

andrewhibbert commented Apr 27, 2026

Uh oh!

rickbrouwer commented Apr 27, 2026

Uh oh!

andrewhibbert commented Apr 27, 2026

Uh oh!

rickbrouwer commented Apr 27, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickbrouwer commented Apr 30, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickbrouwer commented Apr 30, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JorTurFer commented May 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewhibbert commented Apr 24, 2026 •

edited

Loading

snyk-io Bot commented Apr 24, 2026 •

edited

Loading

rickbrouwer commented Apr 25, 2026 •

edited by github-actions Bot

Loading

rickbrouwer Apr 25, 2026 •

edited

Loading

rickbrouwer commented Apr 27, 2026 •

edited by github-actions Bot

Loading

rickbrouwer commented Apr 30, 2026 •

edited by github-actions Bot

Loading

rickbrouwer commented Apr 30, 2026 •

edited by github-actions Bot

Loading