benchmarking/locust: Add sweperf trajectory-based benchmarking workload by pmandewalkar · Pull Request #1694 · agent-substrate/substrate

Parnika Mandewalkar (pmandewalkar) · 2026-09-17T00:34:38Z

Adds a SWE-Perf benchmarking workload to the benchmarking suite, exercising
actor suspend/resume against a real SWE-bench task image (astropy-7336)
rather than a synthetic load.

The workload runs a recorded agent trajectory in configurable cycles,
suspending and resuming the actor between each, so the benchmark measures
lifecycle cost under a realistic in-sandbox server.

What's here

Area	Change
`internal/benchmarking/boomer/sweperf/`	New `SweperfUser` boomer user class (+ tests)
`internal/benchmarking/boomer/dynconfig/`	`sweperf_template`, `sweperf_total_steps`, `sweperf_num_cycles` knobs
`benchmarking/locust/common/sweperf_config.py`	`--sweperf-*` Locust flags
`benchmarking/locust/tests/sweperf.py`	Stub user class so the master attributes boomer's stats rows
`benchmarking/workloads/manifests/`	`swebench-astropy-7336` ActorTemplate

Defaults: 21 trace steps partitioned into 4 cycles.

Verification

hack/verify-all.sh — pass
go test -race ./internal/benchmarking/... ./cmd/benchmarking/... — pass
End-to-end on a GKE dev cluster (gVisor), 1 user:

state: running | users: 1 | fail_ratio: 0.0

NAME                     REQ  FAIL   AVG_ms
CreateActor                1     0        1
CreateAtespace             1     0        1
ResumeActor               20     0      495
SuspendActor              20     0      947
Workload_Cycle_1           5     0     1996
Workload_Cycle_2           5     0      964
Workload_Cycle_3           5     0     4877
Workload_Cycle_4           5     0     3022

164 requests total, 0 failures, 0 errors.

The ActorTemplate image is currently pinned to a personal Artifact Registry
repo; a shared public registry for these SWE-Perf task images is planned.

Tests pass
Appropriate changes to documentation are included in the PR

…kload - Add sweperf_template, sweperf_total_steps, and sweperf_num_cycles to Boomer dynconfig - Update sweperf workload runtime to resolve config dynamically - Add sweperf arguments to Locust web UI / common config - Cleanup static env vars from Locust manifests and deploy scripts

Max Smythe (maxsmythe)

Thank you for PR!

Biggest questions are about the repo we are drawing binaries from and how VU config values are sourced.

Max Smythe (maxsmythe) · 2026-09-17T17:14:23Z

+  # test; the command below runs that entrypoint against that trace, so the
+  # image has to be the one built for this task. A shared public registry of 
+  # these images is planned; for now this is our working version. 
+  image: us-central1-docker.pkg.dev/nishanth-gke-dev/agent-sandbox/swe-astropy-7336@sha256:944924a9a4571c0fac72ac5d1c9be71ff5e98ef2cdcd1034ceaa48d47f755ced


for security, this should point to an official repo

Agreed. Waiting until the SWE-Perf image repo is public, then I'll repoint this to the official location.

Public now, added the repo

Max Smythe (maxsmythe) · 2026-09-17T21:22:48Z

+
+// dynamicWait is the think time between cycles: a uniform draw from
+// [MinWait, MaxWait), or MinWait when the range is empty.
+func (r *sweperfRuntime) dynamicWait() time.Duration {


Can we turn this into a shared function? I think every class uses a similar function.

i wanted to check scope before I do this. there are three copies rn: taskRuntime.dynamicWait + liveWait (lifecycle.go), durDirRuntime.dynamicWait (durdir.go), and this one. lifecycle.go already has a package-private
uniformWait(lo, hi) that durdir doesn't use.

Options I see:

Move uniformWait to boomerutil.UniformWait(lo, hi). Each runtime would then have a 2-line dynamicWait() that picks MinWait/MaxWait (liveWait picks MinLive/MaxLive). this shares the math function but leaves 3 wrappers.

same, but delete the wrappers and call boomerutil.UniformWait(dyn.MinWait, dyn.MaxWait) at the 6 sleep sites. this has no wrappers, but the which window choice gets repeated at each call site.

we could accessors on dynconfig.Config [cfg.DrawWait() / cfg.DrawLive()] over one shared helper. this gets rid of the copies and keeps the field choice next to the fields, which helps if dynconfig gets redone like you were suggesting.

I think 3 makes sense, or 1 if you'd rather keep dynconfig free of actions. what do u think?

Also, these would touch glutton. i'm happy to do it here, or I can put it in a separate PR with the shared setActorRouting change so this one stays unblocked and sweperf-only, up to you!

discussed, will put shared function extraction in a diff pr

Point the module docstring at internal/benchmarking/boomer/sweperf/, where the load is actually implemented, rather than the boomer-worker binary that only registers it. Also fix a copy-paste in the /boomer-config comment that referred to boomer-glutton, and restore the no-op caveat the glutton and durdir stubs carry.

resolveConfig consulted dynconfig, then WORKLOAD_TEMPLATE/WORKLOAD_REPO/ TOTAL_STEPS/NUM_CYCLES from the environment, then built-in defaults. No deployment set those variables, and the operator-facing equivalents (LOCUST_SWEPERF_*) already reach the worker through dynconfig, so the env layer was unreachable and duplicated the defaults beneath it. Drop it, leaving dynconfig with a built-in fallback, matching how the glutton workloads read their config. The env-fallback unit test goes with it; the defaults path it exercised is already covered by the default fallback values subtest.

Drop the pinned image from a personal dev registry and leave the field empty, so the manifest carries no reference a reader cannot pull. The comment now names the repo the image is built from.

Haowei Cai (Roy) (roycaihw)

LGTM. Thanks!

Haowei Cai (Roy) (roycaihw) · 2026-09-21T19:08:35Z

+// templateNS is the namespace holding the actor templates this workload
+// instantiates from.
+const (
+	templateNS = "benchmark-workloads"


Shall we randomize this namespace / make it configurable? Or is this (benchmark-workloads) a common pattern shared by other benchmarks?

it's shared! mirror's glutton's convention.

Haowei Cai (Roy) (roycaihw) · 2026-09-21T19:11:48Z

+// generateDynamicChunks splits totalSteps into numCycles contiguous chunks.
+// The remainder is spread over the earliest cycles, so chunk lengths differ by
+// at most one.
+func generateDynamicChunks(totalSteps int, numCycles int) []chunk {


nit: the chunking logic can be put in a separate util.go file and be unit-tested there

makes sense! i think Max also suggested a bit of library-fying, so ideally that will go in a separate PR and I can address your two comments there. I want to get the workload itself to Nishanth as soon as I can and will handle the refactoring after :D

Why would we have the chunking logic in a util.go file? Are we expecting to reuse the chunks for other code? Seems pretty use-case specific to me.

Max Smythe (maxsmythe)

LGTM

…ad (agent-substrate#1694) Fixes agent-substrate#1692 Adds a SWE-Perf benchmarking workload to the benchmarking suite, exercising actor suspend/resume against a real SWE-bench task image (`astropy-7336`) rather than a synthetic load. The workload runs a recorded agent trajectory in configurable cycles, suspending and resuming the actor between each, so the benchmark measures lifecycle cost under a realistic in-sandbox server. ### What's here | Area | Change | |---|---| | `internal/benchmarking/boomer/sweperf/` | New `SweperfUser` boomer user class (+ tests) | | `internal/benchmarking/boomer/dynconfig/` | `sweperf_template`, `sweperf_total_steps`, `sweperf_num_cycles` knobs | | `benchmarking/locust/common/sweperf_config.py` | `--sweperf-*` Locust flags | | `benchmarking/locust/tests/sweperf.py` | Stub user class so the master attributes boomer's stats rows | | `benchmarking/workloads/manifests/` | `swebench-astropy-7336` ActorTemplate | Defaults: 21 trace steps partitioned into 4 cycles. ### Verification - `hack/verify-all.sh` — pass - `go test -race ./internal/benchmarking/... ./cmd/benchmarking/...` — pass - End-to-end on a GKE dev cluster (gVisor), 1 user: ``` state: running | users: 1 | fail_ratio: 0.0 NAME REQ FAIL AVG_ms CreateActor 1 0 1 CreateAtespace 1 0 1 ResumeActor 20 0 495 SuspendActor 20 0 947 Workload_Cycle_1 5 0 1996 Workload_Cycle_2 5 0 964 Workload_Cycle_3 5 0 4877 Workload_Cycle_4 5 0 3022 ``` 164 requests total, 0 failures, 0 errors. The ActorTemplate image is currently pinned to a personal Artifact Registry repo; a shared public registry for these SWE-Perf task images is planned. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR

…substrate#1889) Related: agent-substrate#1694 The `swebench-astropy-7336` ActorTemplate can't be deployed as shipped: 1. `image: ""` must be hand-edited into a tracked file before every run. 2. The command points at `/opt/swebench/replay.py` and `/opt/swebench/astropy_trace.json`, which the sweperf image generator never produces. It writes `/replay.py` and `/trace.json` at the image root (also the image's ENTRYPOINT), so the container exits immediately. This PR: - takes the image from a `SWEPERF_IMAGE` env var, substituted by `workloads/deploy.sh` like the other template variables; - fixes both paths; - documents the minimum sweperf commit (`c30c0d6`), since older images have no HTTP server and fail on the first `GET /status`. Usage: ```sh SWEPERF_IMAGE=<registry>/sweperf-astropy@sha256:... \ WORKLOAD_TEMPLATES="swebench-astropy-7336" \ benchmarking/workloads/deploy.sh --deploy ... ``` Verified on GKE with both gVisor and microVM worker pools: 30-minute sweperf runs at 1 user, all 4 cycles, 0 failures. - [x] Tests pass (`bash -n`; exercised end-to-end as above) - [x] Appropriate changes to documentation are included in the PR (template comments)

Parnika Mandewalkar (pmandewalkar) added 5 commits September 17, 2026 16:40

Add sweperf benchmarking workload

f9df71a

add unit tests for sweperf workload

b0f7052

route sweperf requests via TargetActorHeader instead of Host

a450780

docs(benchmarking): explain astropy task image requirement

5c5be8d

Parnika Mandewalkar (pmandewalkar) force-pushed the sweperf-workload branch from bfc205b to 5c5be8d Compare September 17, 2026 17:04

Max Smythe (maxsmythe) reviewed Sep 17, 2026

View reviewed changes

Bowei Du (bowei) added kind/feature An enhancement / feature request or implementation area/benchmarking labels Sep 17, 2026

Parnika Mandewalkar (pmandewalkar) added 2 commits September 18, 2026 22:44

Parnika Mandewalkar (pmandewalkar) requested a review from Max Smythe (maxsmythe) September 18, 2026 23:11

docs(benchmarking): point astropy task image at the public sweperf repo

5e1d176

Drop the pinned image from a personal dev registry and leave the field empty, so the manifest carries no reference a reader cannot pull. The comment now names the repo the image is built from.

Haowei Cai (Roy) (roycaihw) approved these changes Sep 21, 2026

View reviewed changes

Max Smythe (maxsmythe) approved these changes Sep 21, 2026

View reviewed changes

Tim Hockin (thockin) added this pull request to the merge queue Sep 22, 2026

Merged via the queue into agent-substrate:main with commit cdac9ba Sep 22, 2026
13 of 15 checks passed

Conversation

Parnika Mandewalkar (pmandewalkar) commented Sep 17, 2026

What's here

Verification

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Parnika Mandewalkar (pmandewalkar) Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Haowei Cai (Roy) (roycaihw) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Parnika Mandewalkar (pmandewalkar) Sep 18, 2026 •

edited

Loading