Skip to content

perf(appsec): make rate limiter synchronous to remove idle goroutine cost#4956

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
eliottness/rework-limiter
Jun 25, 2026
Merged

perf(appsec): make rate limiter synchronous to remove idle goroutine cost#4956
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
eliottness/rework-limiter

Conversation

@eliottness

@eliottness eliottness commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Replaces the AppSec rate limiter's background-goroutine token bucket with a synchronous implementation backed by golang.org/x/time/rate.Limiter.

  • Removes the per-instance background goroutine and hard-coded 500µs time.Ticker that woke ~2000×/sec on every pod, even idle ones.
  • Allow() now refills the bucket lazily on call via rate.Limiter, so idle pods incur zero limiter CPU cost.
  • Deletes the TokenTicker struct and its Start()/Stop() lifecycle. The NewTokenTicker / NewTokenTickerWithInterval constructors now return the limiter.Limiter interface backed directly by *rate.Limiter (which already satisfies Allow() bool).
  • Drops the now-unused Start()/Stop() calls in the WAF listener and the apisec proxy sampler. As a side effect, the proxy sampler's previously-leaked goroutine (the Sampler interface had no Stop) is eliminated.
  • Migrates the limiter unit tests from an injected fake clock to the standard testing/synctest package (deterministic virtual time); concurrency and benchmark tests remain on real time. Also fixes BenchmarkLimiter, which no longer built on modern Go (B.Loop called with timer stopped).

No change to rate-limit values, defaults, env vars, or the Limiter interface contract; behavior is preserved.

Motivation

The limiter's background goroutine consumed roughly 0.3% of a CPU core per instance continuously — even on idle pods with no traffic — which adds up meaningfully across the fleet. golang.org/x/time/rate is already a direct dependency and is fully synchronous (no goroutine, no ticker), so it gives zero idle cost while preserving the existing token-bucket behavior. Per-call cost (~50ns uncontended) is irrelevant given Allow() is invoked at most ~100/s.

Performance notes

Rough local microbenchmarks (single machine, quick comparison — directional only, not a rigorous A/B) informed the choice of rate.Limiter:

  • Idle cost (what we actually care about): the old background ticker burned on the order of ~0.3% of a CPU core per limiter instance, continuously, even with zero traffic (it woke ~2000×/s regardless of load). The synchronous rate.Limiter does no background work, so idle cost drops to ~zero.
  • Per-call Allow(): the old lock-free atomic path was a few ns uncontended; rate.Limiter is in the tens of ns uncontended, and the two are roughly comparable under contention. Both are allocation-free. At our call frequency (Allow() fires at most ~100/s) the difference is noise.
  • Why rate.Limiter vs a bespoke lazy bucket: it's already a direct dependency, fully synchronous, and well-tested upstream — not worth hand-rolling and maintaining an equivalent for a sub-microsecond, low-frequency path.

Net trade: a negligible per-call overhead in exchange for eliminating a continuous, fleet-wide idle CPU cost.

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added.
  • New code is free of linting errors. You can check this by running make lint locally.
  • New code doesn't break existing tests. You can check this by running make test locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • All generated files are up to date. You can check this by running make generate locally.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running make fix-modules locally.

eliottness and others added 3 commits June 25, 2026 13:41
…icker goroutine)

Replaces the per-instance background goroutine and 500µs time.Ticker with a synchronous golang.org/x/time/rate.Limiter that lazily refills via an injected clock. Eliminates idle-pod CPU cost and the apisec proxy-sampler goroutine leak while preserving the public API (TokenTicker, constructors, Start/Stop no-ops, Allow) and behavioral test assertions. Also fixes the broken BenchmarkLimiter.
…ly and test via synctest

Removes the TokenTicker wrapper struct, its Start()/Stop() no-op methods, and the injected now func() time.Time clock. The NewTokenTicker/NewTokenTickerWithInterval constructors now build a golang.org/x/time/rate.Limiter and return it as the limiter.Limiter interface (which *rate.Limiter already satisfies via Allow()).

Drops all Start()/Stop() calls in the WAF listener and apisec proxy sampler; the WAF feature field is now limiter.Limiter.

Migrates limiter unit tests from an injected fake clock to the standard testing/synctest package (deterministic virtual time); concurrency and benchmark tests remain on real time.

No behavior change to rate limiting; public limiter behavior and all assertions preserved.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <[email protected]>
Resolves the golangci-lint 'modernize' finding by replacing the manual if-clamp with the min builtin. No behavior change.
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jun 25, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 81.25%
Overall Coverage: 62.74% (+10.55%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 24dbd17 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 25, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-25 12:59:35

Comparing candidate commit 24dbd17 in PR branch eliottness/rework-limiter with baseline commit 2f4d3ca in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 297 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

@eliottness
eliottness marked this pull request as ready for review June 25, 2026 13:08
@eliottness
eliottness requested a review from a team as a code owner June 25, 2026 13:08
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit cedfb42 into main Jun 25, 2026
204 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the eliottness/rework-limiter branch June 25, 2026 13:59
darccio pushed a commit that referenced this pull request Jun 25, 2026
…cost (#4956)

### What does this PR do?

Replaces the AppSec rate limiter's background-goroutine token bucket with a synchronous implementation backed by `golang.org/x/time/rate.Limiter`.

- Removes the per-instance background goroutine and hard-coded 500µs `time.Ticker` that woke ~2000×/sec on **every** pod, even idle ones.
- `Allow()` now refills the bucket lazily on call via `rate.Limiter`, so idle pods incur **zero** limiter CPU cost.
- Deletes the `TokenTicker` struct and its `Start()`/`Stop()` lifecycle. The `NewTokenTicker` / `NewTokenTickerWithInterval` constructors now return the `limiter.Limiter` interface backed directly by `*rate.Limiter` (which already satisfies `Allow() bool`).
- Drops the now-unused `Start()`/`Stop()` calls in the WAF listener and the apisec proxy sampler. As a side effect, the proxy sampler's previously-leaked goroutine (the `Sampler` interface had no `Stop`) is eliminated.
- Migrates the limiter unit tests from an injected fake clock to the standard `testing/synctest` package (deterministic virtual time); concurrency and benchmark tests remain on real time. Also fixes `BenchmarkLimiter`, which no longer built on modern Go (`B.Loop called with timer stopped`).

No change to rate-limit values, defaults, env vars, or the `Limiter` interface contract; behavior is preserved.

### Motivation

The limiter's background goroutine consumed roughly 0.3% of a CPU core per instance **continuously** — even on idle pods with no traffic — which adds up meaningfully across the fleet. `golang.org/x/time/rate` is already a direct dependency and is fully synchronous (no goroutine, no ticker), so it gives zero idle cost while preserving the existing token-bucket behavior. Per-call cost (~50ns uncontended) is irrelevant given `Allow()` is invoked at most ~100/s.

### Performance notes

Rough local microbenchmarks (single machine, quick comparison — directional only, not a rigorous A/B) informed the choice of `rate.Limiter`:

- **Idle cost (what we actually care about):** the old background ticker burned on the order of ~0.3% of a CPU core *per limiter instance, continuously*, even with zero traffic (it woke ~2000×/s regardless of load). The synchronous `rate.Limiter` does no background work, so idle cost drops to ~zero.
- **Per-call `Allow()`:** the old lock-free atomic path was a few ns uncontended; `rate.Limiter` is in the tens of ns uncontended, and the two are roughly comparable under contention. Both are allocation-free. At our call frequency (`Allow()` fires at most ~100/s) the difference is noise.
- **Why `rate.Limiter` vs a bespoke lazy bucket:** it's already a direct dependency, fully synchronous, and well-tested upstream — not worth hand-rolling and maintaining an equivalent for a sub-microsecond, low-frequency path.

Net trade: a negligible per-call overhead in exchange for eliminating a continuous, fleet-wide idle CPU cost.

### Reviewer's Checklist

- [x] Changed code has unit tests for its functionality at or near 100% coverage.
- [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag.
- [x] There is a benchmark for any new code, or changes to existing code.
- [ ] If this interacts with the agent in a new way, a system test has been added.
- [x] New code is free of linting errors. You can check this by running `make lint` locally.
- [x] New code doesn't break existing tests. You can check this by running `make test` locally.
- [ ] Add an appropriate team label so this PR gets put in the right place for the release notes.
- [x] All generated files are up to date. You can check this by running `make generate` locally.
- [x] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally.



Co-authored-by: eliott.bouhana <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants