Skip to content

fix(benchmarking): make the SWE-Perf ActorTemplate deployable - #1889

Merged
Max Smythe (maxsmythe) merged 1 commit into
agent-substrate:mainfrom
roycaihw:fix/sweperf-actortemplate-image-and-command
Sep 28, 2026
Merged

fix(benchmarking): make the SWE-Perf ActorTemplate deployable#1889
Max Smythe (maxsmythe) merged 1 commit into
agent-substrate:mainfrom
roycaihw:fix/sweperf-actortemplate-image-and-command

Conversation

@roycaihw

@roycaihw Haowei Cai (Roy) (roycaihw) commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Related: #1694

The swebench-astropy-7336 ActorTemplate can't be deployed as shipped:

  1. image: "" must be hand-edited into a tracked file before every run.
  2. The command points at /opt/swebench/replay.py and
    /opt/swebench/astropy_trace.json, which the sweperf image generator never
    produces. It writes /replay.py and /trace.json at the image root (also
    the image's ENTRYPOINT), so the container exits immediately.

This PR:

  • takes the image from a SWEPERF_IMAGE env var, substituted by
    workloads/deploy.sh like the other template variables;
  • fixes both paths;
  • documents the minimum sweperf commit (c30c0d6), since older images have no
    HTTP server and fail on the first GET /status.

Usage:

SWEPERF_IMAGE=<registry>/sweperf-astropy@sha256:... \
WORKLOAD_TEMPLATES="swebench-astropy-7336" \
  benchmarking/workloads/deploy.sh --deploy ...

Verified on GKE with both gVisor and microVM worker pools: 30-minute sweperf
runs at 1 user, all 4 cycles, 0 failures.

  • Tests pass (bash -n; exercised end-to-end as above)
  • Appropriate changes to documentation are included in the PR (template comments)

@roycaihw
fix(benchmarking): make the SWE-Perf ActorTemplate deployable
43456e0 
The swebench-astropy-7336 ActorTemplate cannot be deployed as shipped.
Two independent problems:

1. `image: ""` — the field is empty with a comment saying to fill it in
   before deploying. There is no way to supply it without editing a
   tracked file, so every run dirties the tree (and the substrate version
   label derived from it).

2. The command points at `/opt/swebench/replay.py` and
   `/opt/swebench/astropy_trace.json`. Neither path exists. The image
   generator in https://github.com/gke-labs/sweperf writes the replay
   server to `/replay.py` and the trajectory to `/trace.json` at the
   image root — which is also the image's own ENTRYPOINT. Nothing in
   that repo produces `/opt/swebench/`, so the container exits
   immediately with "can't open file '/opt/swebench/replay.py'".

Supply the image through a SWEPERF_IMAGE environment variable,
substituted by workloads/deploy.sh alongside the existing template
variables, and correct the two paths.

Also documents the minimum sweperf commit (c30c0d6, 2026-09-21). The
async HTTP execution API this benchmark drives landed 2026-08-31 and
`execution_duration_ms` on 2026-09-21; images built earlier ship a
replay.py with no HTTP server and fail on the client's first
`GET /status`. Verified against a rebuilt image: a 21-step / 4-cycle
run completes with 0 failures.
@roycaihw

Haowei Cai (Roy) (roycaihw) commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

/cc Nishanth Kotla (@Nishanth29) Parnika Mandewalkar (@pmandewalkar)

@Nishanth29

Nishanth Kotla (Nishanth29) commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Thanks for putting this together, Haowei. I think #1848 already covers this in 42b892db (same command, digest-pinned images for all 5 templates).

@roycaihw

Haowei Cai (Roy) (roycaihw) commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for putting this together, Haowei. I think #1848 already covers this in 42b892db (same command, digest-pinned images for all 5 templates).

SG! We can close this as duplicate if #1848 gets merged first. This is a focus fix to unblock the deployment

@roycaihw

Haowei Cai (Roy) (roycaihw) commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Max Smythe (@maxsmythe) This is a small fix to make swe-perf runnable OOTB. PTAL. This will help unblock the prowjob setup.

@Nishanth29

Nishanth Kotla (Nishanth29) commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

LGTM. A few thoughts for later Haowei Cai (Roy) (@roycaihw):

  • a single SWEPERF_IMAGE works for one template, but once the other four swebench templates land each needs its own image, so we may want per-template vars or a repo var + pinned digest.
  • if SWEPERF_IMAGE is unset the template gets image: "" and only fails later; a quick empty check in deploy.sh would make that clearer.
  • we could default to the pinned images in us-docker.pkg.dev/gke-substrate-release/benchmark (all five tasks are there) so people don't have to build their own.

@roycaihw

Haowei Cai (Roy) (roycaihw) commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Yes Parnika Mandewalkar (@pmandewalkar) we have plan to use multiple sweperf images and dynamically populate the template, instead of using one specific image as entry point.

@pmandewalkar

Parnika Mandewalkar (pmandewalkar) commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Haowei Cai (Roy) (@roycaihw) I think dynamically populating the template though substrate as an idea is getting abandoned. One of our earlier conversations resulted in us agreeing on a small lightweight script that'll generate those templates, and this removes any additional substrate overhead. Let me know if that is still ok; cc Nishanth Kotla (@Nishanth29)

@roycaihw

Haowei Cai (Roy) (roycaihw) commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Haowei Cai (Roy) (Haowei Cai (Roy) (@roycaihw)) I think dynamically populating the template though substrate as an idea is getting abandoned. One of our earlier conversations resulted in us agreeing on a small lightweight script that'll generate those templates, and this removes any additional substrate overhead. Let me know if that is still ok; cc Nishanth Kotla (Nishanth Kotla (@Nishanth29))

Discussed offline. We are still aligned that we need a benchmark entry point that "run actors that uses a distribution of all available sweperf trajectories" instead of what we have now ("run actors that uses this single sweperf trajectory")

The implementation detail of how to generate the actor templates for all the available sweperf trajectories isn't critical. We can have a script to generate them on the fly, or checked-in if substrate maintainers are okay with that.

maxsmythe
Max Smythe (maxsmythe) approved these changes Sep 28, 2026
View reviewed changes

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe
Max Smythe (maxsmythe) added this pull request to the merge queue Sep 28, 2026
Merged via the queue into agent-substrate:main with commit f7ec6ac Sep 28, 2026
11 checks passed
@roycaihw
Haowei Cai (Roy) (roycaihw) deleted the fix/sweperf-actortemplate-image-and-command branch September 29, 2026 00:10
@Nishanth29 Nishanth Kotla (Nishanth29) mentioned this pull request Sep 29, 2026
Merged
2 tasks done
@bowei Bowei Du (bowei) added area/benchmarking kind/bug Something isn't working / bugfixes labels Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxsmythe Max Smythe (maxsmythe) maxsmythe approved these changes

Assignees

No one assigned

Labels

area/benchmarking kind/bug Something isn't working / bugfixes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@roycaihw @Nishanth29 @pmandewalkar @maxsmythe @bowei