Skip to content

benchmarking: pace boomer to a target rate and add the resumepaused user class - #2350

Merged
Max Smythe (maxsmythe) merged 2 commits into
agent-substrate:mainfrom
haiyanmeng:boomer-load-generator
Oct 13, 2026
Merged

Max Smythe (maxsmythe) merged 2 commits into
agent-substrate:mainfrom
haiyanmeng:boomer-load-generator

Conversation

@haiyanmeng

@haiyanmeng haiyanmeng commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

This PR lets boomer hold a fixed request rate for each ladder step, and adds a user class, resumepaused, that drives ate-api-server's cold resume path at that rate. It is needed for ate-api-server capacity runs: a closed-loop user can't hold a rate, because the offered load drops whenever the server slows down. Benchmark-only, with no production code changes.

Commits

  1. Pace boomer to a target request rate per ladder step. A new dynconfig field, target_rps, is set with --target-rps or per ladder step as users:duration@rps (e.g. 200:3m@4000). Boomer picks up a change within 10 s. Only user classes that pace themselves honor it, which today is resumepaused.
  2. Add the resumepaused user class. Each user owns one actor. On its first iteration the user creates the actor and makes it cold with a resume and a pause; every later iteration resumes it (ResumeActorCold, paced to target_rps) and pauses it again (PauseActor). The warm-up is unpaced and recorded under its own names (CreateActor, ResumeActorWarmup, PauseActorWarmup) in the step that added the user. Every actor is deleted on exit. It sends no traffic to the actors, so it runs on the fake data plane (fake-dataplane: let the fake data plane host actors #2300).

Notes for reviewers

  • A capacity resumepaused finds is one of resume and pause cycles, not of resumes alone.
  • Keep users at or above the rate times the p99 cycle time, or a step falls short of its rate.
  • target_rps paces one boomer worker.
  • A step that adds many users carries their warm-up load while they join at the spawn rate, so trim the start of each step. When a step lowers the user count, the stopped users' actors stay paused until boomer exits.
  • The tests.yaml capacity tests come in a follow-up PR.

Details are in the ResumePaused section of benchmarking/README.md.

Testing

  • go test -race ./internal/benchmarking/boomer/...
  • python3 benchmarking/locust/unit_tests/test_ladder_shape.py (needs locust)
  • python3 benchmarking/automation/test_orchestrator.py

Addresses #2261

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Comment thread benchmarking/locust/tests/resumecold.py Resolved
Comment thread benchmarking/automation/tests.yaml Resolved
@haiyanmeng
haiyanmeng force-pushed the boomer-load-generator branch 3 times, most recently from 7b2173d to 80ece49 Compare October 8, 2026 19:06
Comment thread benchmarking/README.md Resolved
Comment thread benchmarking/README.md Resolved
@haiyanmeng
haiyanmeng force-pushed the boomer-load-generator branch from 80ece49 to afc96d1 Compare October 11, 2026 21:09
@haiyanmeng haiyanmeng changed the title benchmarking: pace boomer to a target rate and add the resumecold user class benchmarking: pace boomer to a target rate and add the resumepaused user class Oct 11, 2026
@haiyanmeng
haiyanmeng force-pushed the boomer-load-generator branch from afc96d1 to 839e7f7 Compare October 11, 2026 21:14
A capacity run holds each request rate for a step and reads the cost at
that rate. Stepping users does not hold a rate: a closed-loop user sends
its next request when the last one returns, so when the server slows the
offered load drops with it.

dynconfig gains target_rps, the requests per second a whole boomer worker
paces to, for user classes that pace themselves; 0 means unpaced and a
negative value is rejected. Like the other fields, the master can change
it while the run continues. locust gains --target-rps to set it, and a
--ladder step can end in @target_rps, e.g. 200:3m@4000, to change it as
the step starts. boomer reads a rate-only change at its next poll of the
master's config.
@haiyanmeng
haiyanmeng force-pushed the boomer-load-generator branch 2 times, most recently from 6ab42f1 to b63f7e6 Compare October 11, 2026 21:57
@haiyanmeng
haiyanmeng force-pushed the boomer-load-generator branch from b63f7e6 to e54598b Compare October 13, 2026 15:43
resumepaused drives ate-api-server's cold resume path at a set rate.
Each user owns one actor. On its first iteration the user creates the
actor from the glutton template and makes it cold with a resume and a
pause; every later iteration resumes the actor and pauses it again.
Boomer gives a task no user identity, so the actor is keyed by the
user's goroutine ID, as in the glutton classes. The warm-up is not paced
and is recorded under its own names, in the step that added the user. It
adds load only while users join at the spawn rate, a stretch of each
step that is trimmed anyway. An actor that fails to warm is deleted and
the user tries again on its next iteration.

Only the resume is paced, to dynconfig's target_rps, through one limiter
the users share with a burst of 1, so resumes arrive evenly rather than
in a burst at the start of each second. The target is therefore cold
resumes per second. The pauses follow at the same rate, so a capacity
the class finds is one of resume and pause cycles.

A resume that finds its actor running fails the sample, as no cold
resume happened. A failed cycle pauses the actor again so it is cold for
the next one; that pause is recorded as PauseActorRecovery, apart from
the measured pauses, and counts as a success when it finds the actor
already paused. If the atespace cannot be ensured, the worker exits at
startup. Shutdown waits for the cycles and setups in flight and deletes
every actor. They get half of its time budget; any still running then
are canceled, so a hung RPC cannot leave the deletes no time. Once
shutdown begins, samples go to Prometheus only: boomer may have stopped
draining its stats channel, and a send to it would block once the
buffer fills.

resumepaused sends the actors themselves no requests, so it runs on the
fake data plane, where it measures what a cold resume costs
ate-api-server and Postgres without the router or the restore in the
way. benchmarking/README.md documents its knobs and metrics.
@haiyanmeng
haiyanmeng force-pushed the boomer-load-generator branch from e54598b to a05ff95 Compare October 13, 2026 16:20

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe
Max Smythe (maxsmythe) added this pull request to the merge queue Oct 13, 2026
Merged via the queue into agent-substrate:main with commit e1847fb Oct 13, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants