Skip to content

Add an option for disabling the workqueue bucket rate limiter#6

Merged
tfujiwar merged 1 commit intomercari-masterfrom
tfujiwar-disable-workqueue-bucket-rate-limiter
Nov 29, 2024
Merged

Add an option for disabling the workqueue bucket rate limiter#6
tfujiwar merged 1 commit intomercari-masterfrom
tfujiwar-disable-workqueue-bucket-rate-limiter

Conversation

@tfujiwar
Copy link
Copy Markdown
Collaborator

@tfujiwar tfujiwar commented Nov 29, 2024

We observed that many EpehemralRunners were waiting for reconciliation after a JITConfig secret was created. This is likely because the next reconciliation is not enqueued by returning Result{Requeue: true} and such requeued reconciliations are rate limited by the workqueue rate limiter.
https://github.com/actions/actions-runner-controller/blob/8b36ea90ebe81710fcdcb4f96424b43203d24f1e/controllers/actions.github.com/ephemeralrunner_controller.go#L681-L682

The default rate limiter is defined by a combination of NewTypedItemExponentialFailureRateLimiter and TypedBucketRateLimiter.
https://github.com/kubernetes/client-go/blob/37045084c2aa82927b0e5ffc752861430fd7e4ab/util/workqueue/default_rate_limiters.go#L50-L56

This PR adds an option for disabling the TypedBucketRateLimiter.

@tfujiwar tfujiwar marked this pull request as ready for review November 29, 2024 01:20
if autoScalingRunnerSetOnly {
var newWorkqueueRateLimiter func() workqueue.RateLimiter
if disableWorkqueueBucketRateLimiter {
newWorkqueueRateLimiter = workqueue.DefaultItemBasedRateLimiter
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what version of workqueue this code is on, but in latest versions DefaultItemBasedRateLimiter is considered deprecated: "Deprecated: DefaultItemBasedRateLimiter is deprecated, use DefaultTypedItemBasedRateLimiter instead."

if disableWorkqueueBucketRateLimiter {
newWorkqueueRateLimiter = workqueue.DefaultItemBasedRateLimiter
} else {
newWorkqueueRateLimiter = workqueue.DefaultControllerRateLimiter
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same ^

Copy link
Copy Markdown

@djahandarie djahandarie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM since it's just to test hypothesis, can clean it up later as needed

@tfujiwar tfujiwar merged commit eea49ea into mercari-master Nov 29, 2024
@tfujiwar tfujiwar deleted the tfujiwar-disable-workqueue-bucket-rate-limiter branch November 29, 2024 04:07
@azrsh azrsh mentioned this pull request Oct 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants