Skip to content

Commit 4e079e7

Browse files
benchmarking(agentsession): define the session script in YAML, selectable per run (agent-substrate#2046)
## What this is Follow-up to agent-substrate#1934, as discussed there: the agent-session script moves out of Go and into YAML, so a workload can be read, edited, and selected without touching the driver. - The default script is now `internal/benchmarking/boomer/agentsession/scripts/coding-session.yaml`, embedded in the boomer worker. It was generated from the old Go table and round-trips byte-for-byte on every op. - `--agentsession-script <name>` picks a built-in variant (any `scripts/<name>.yaml`). Resolved on the worker's first iteration (the knob arrives with the first spawn message, after Init); a bad name is logged and retried each iteration, so fixing the knob and re-swarming recovers. - Decoding is strict and runs the same invariants the old tests enforced: unknown op kinds or fields, arguments a kind does not take, reads before writes, walks before fills, duplicate steps, non-positive think times. - Each script declares `min_actor_memory`. The worker reads the glutton template's memory limit at start and refuses to run against a smaller actor, with a message naming `--actor-memory`. That replaces the README's advice with a guard. - README gains a "Writing an agent-session script" section with the format and the op table. ## Not in this PR Loading a script from a file on the worker (ConfigMap mount) is the next PR, on top of this loader. ## Testing `go test -race` on the benchmarking packages, boilerplate, gofmt, and golangci-lint pass. New tests: every embedded script loads; default script shape (20 steps, 1Gi floor, declared bytes under budget); encode/decode round trip; 14 rejection cases; the template memory guard in refuse, exact, and no-limit modes. Co-authored-by: Aditya Shantanu <[email protected]>
1 parent 3f789b3 commit 4e079e7

12 files changed

Lines changed: 1080 additions & 344 deletions

File tree

‎benchmarking/README.md‎

Lines changed: 59 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -160,12 +160,13 @@ step's think time, and the next step's first request **wakes it through the
160160
atenet router** (request parking). Mostly-idle sessions plus fast wake is
161161
exactly the oversubscription story this measures.
162162

163-
The entire workload is the declarative script in
164-
[`internal/benchmarking/boomer/agentsession/script.go`](../internal/benchmarking/boomer/agentsession/script.go)
165-
— one table entry per step, naming what the agent is doing and the resource
166-
ops that act it out. To change the workload, edit the table. Each session is
167-
an actor from the stock `glutton` template; steps are sequences of glutton
168-
RPCs:
163+
The entire workload is a YAML script. The default,
164+
[`internal/benchmarking/boomer/agentsession/scripts/coding-session.yaml`](../internal/benchmarking/boomer/agentsession/scripts/coding-session.yaml),
165+
has one entry per step naming what the agent is doing and the resource ops
166+
that act it out. To change the workload, edit it or add a sibling file and
167+
select it with `--agentsession-script` (see [Writing a
168+
script](#writing-an-agent-session-script)). Each session is an actor from
169+
the stock `glutton` template; steps are sequences of glutton RPCs:
169170

170171
| Step | The agent is… | Sandbox effect |
171172
|---|---|---|
@@ -193,9 +194,10 @@ RPCs:
193194
The script needs bigger actors than the 256Mi default: it holds ~96Mi of
194195
RAM arrays + ~110Mi of tmpfs files, and the observed guest peak with
195196
allocator transients is ~320Mi (512Mi OOMs). Deploy the workloads with
196-
`--actor-memory 1Gi`. `TestSessionBudgets` bounds the script-declared bytes
197-
only, so script edits that grow the working set fail the test and force this
198-
guidance to be revisited.
197+
`--actor-memory 1Gi`. Every script declares that floor as
198+
`min_actor_memory`; the worker reads the `glutton` template's memory limit
199+
at start and refuses to run a script against a smaller actor, so a too-small
200+
deployment fails loudly instead of showing up as OOM-flaky steps.
199201

200202
```sh
201203
./benchmarking/deploy_locust.sh --deploy --sandbox-class gvisor --actor-memory 1Gi
@@ -204,6 +206,10 @@ guidance to be revisited.
204206

205207
#### Agent-Session Configuration Knobs
206208

209+
* `--agentsession-script` — built-in script variant to run, by file name
210+
under `internal/benchmarking/boomer/agentsession/scripts/` (default
211+
`coding-session`). Read once, on the worker's first iteration after the
212+
swarm starts, then fixed for the worker's lifetime.
207213
* `--agentsession-think-scale` — multiplier on every think gap; 0.5 makes the
208214
fleet twice as chatty, 4.0 models slow reasoning models (default 1.0). Each
209215
gap gets ±20% jitter so sessions don't move in lockstep.
@@ -212,6 +218,50 @@ guidance to be revisited.
212218
* `--lifecycle-mode suspend|pause` — durable suspend (default) or node-local
213219
pause between steps.
214220

221+
#### Writing an agent-session script
222+
223+
A script is a named step list with a memory floor:
224+
225+
```yaml
226+
name: coding-session
227+
min_actor_memory: 1Gi
228+
steps:
229+
- name: 01_read_task
230+
agent: Boots, reads the task prompt, loads its context window
231+
think: 2s
232+
ops:
233+
- fill_ram: {key: agent_context, size: 32Mi}
234+
- ping: {}
235+
- name: 02_clone_repo
236+
agent: git clone of the target repository
237+
think: 3s
238+
ops:
239+
- ingest: {key: repo_tarball, size: 16Mi}
240+
- burn_cpu: {millis: 500, parallel: 1}
241+
```
242+
243+
`think` is the suspended gap before the step (a Go duration, scaled by
244+
`--agentsession-think-scale`); `name` keys the `Step_<name>` stats row.
245+
Sizes are Kubernetes quantities. Each op is one glutton RPC:
246+
247+
| Op | Arguments | Sandbox effect |
248+
|---|---|---|
249+
| `ingest` | `key`, `size` | bytes cross the network from the driver and land in a file: a download |
250+
| `burn_cpu` | `millis`, `parallel` (default 1) | spins `parallel` goroutines for `millis` of wall clock |
251+
| `write_disk` | `key`, `size` | writes locally generated bytes to a file |
252+
| `read_disk_digest` | `key` | reads and hashes a file; nothing ships back |
253+
| `read_disk_data` | `key` | reads a file and returns its bytes: disk read plus egress |
254+
| `fill_ram` | `key`, `size` | allocates a resident RAM array |
255+
| `churn_ram` | `key`, `size` | re-randomizes part of an array in place, dirtying pages |
256+
| `walk_ram` | `key` | touches one byte per page: demand-paging cost after a resume |
257+
| `ping` | none | a minimal round trip through the router |
258+
259+
Loading is strict: unknown op kinds or fields, an argument a kind does not
260+
take, a read of a file nothing wrote, a walk of an array nothing filled, a
261+
duplicate step name, or a `min_actor_memory` below the declared RAM plus
262+
disk all fail before any actor is created. Built-in variants are checked by
263+
`TestEmbeddedScriptsAreValid`, so a broken file cannot merge.
264+
215265
#### Agent-Session Reported Metrics
216266

217267
* `WakeFirstTouch`: latency of a dedicated ping sent before each step's ops —

‎benchmarking/locust/common/agentsession_config.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,18 @@
2020

2121
@events.init_command_line_parser.add_listener
2222
def add_agentsession_arguments(parser: LocustArgumentParser) -> None:
23+
parser.add_argument(
24+
"--agentsession-script",
25+
type=str,
26+
default="coding-session",
27+
env_var="LOCUST_AGENTSESSION_SCRIPT",
28+
help=(
29+
"Built-in agent-session script variant to run; one of the YAML "
30+
"files under internal/benchmarking/boomer/agentsession/scripts/ "
31+
"(default: coding-session)"
32+
),
33+
include_in_web_ui=True,
34+
)
2335
parser.add_argument(
2436
"--agentsession-think-scale",
2537
type=float,

‎benchmarking/locust/common/boomer_config.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@
6767
"--sweperf-total-steps": int,
6868
"--sweperf-num-cycles": int,
6969
"--sweperf-poll-interval-ms": int,
70+
"--agentsession-script": str,
7071
"--agentsession-think-scale": float,
7172
}
7273

‎internal/benchmarking/boomer/agentsession/agentsession.go‎

Lines changed: 114 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,9 @@
1313
// limitations under the License.
1414

1515
// Package agentsession implements the AgentSessionUser locust test: each VU
16-
// is one coding-agent session replaying the scripted steps in script.go
17-
// against a glutton actor. After every step the actor is suspended for the
16+
// is one coding-agent session replaying a scripted step sequence (a YAML
17+
// file under scripts/, selected by --agentsession-script) against a glutton
18+
// actor. After every step the actor is suspended for the
1819
// step's "LLM thinking" gap; by default the next step's first request wakes
1920
// it through the atenet router (request parking), so the benchmark measures
2021
// the user-visible wake latency Substrate's oversubscription story rests on.
@@ -55,6 +56,7 @@ import (
5556
"google.golang.org/grpc/metadata"
5657
"google.golang.org/grpc/status"
5758
"google.golang.org/protobuf/proto"
59+
"k8s.io/apimachinery/pkg/api/resource"
5860
)
5961

6062
const (
@@ -100,8 +102,7 @@ func initAgentSession(cfg *userclass.Config) (taskFn func(), shutdown func(conte
100102
if cfg.Tracer == nil {
101103
cfg.Tracer = otel.Tracer("substrate-boomer/agentsession")
102104
}
103-
rt := &runtime{cfg: cfg, steps: Session()}
104-
rt.ingestBuf = makeIngestBuf(rt.steps)
105+
rt := &runtime{cfg: cfg}
105106
return rt.iterate, rt.shutdown
106107
}
107108

@@ -132,10 +133,101 @@ func makeIngestBuf(steps []Step) []byte {
132133
// goroutine keeps its own session in users, keyed by goroutine ID, because
133134
// boomer offers no per-VU context.
134135
type runtime struct {
135-
cfg *userclass.Config
136-
steps []Step
137-
ingestBuf []byte // shared read-only ingest payload, sized to the largest ingest op
138-
users sync.Map // goroutineID -> *sessionUser
136+
cfg *userclass.Config
137+
users sync.Map // goroutineID -> *sessionUser
138+
139+
// script is resolved by loadScript on the first iteration and fixed for
140+
// the worker's lifetime; ingestBuf is sized to it.
141+
scriptMu sync.Mutex
142+
script *Script
143+
ingestBuf []byte // shared read-only ingest payload, sized to the largest ingest op
144+
145+
// The template memory check runs once per worker. A fetch failure
146+
// leaves it unanswered so the next startUser retries; a verdict is
147+
// final until the workloads are redeployed.
148+
templateMu sync.Mutex
149+
templateOK bool
150+
templateErr error
151+
}
152+
153+
// loadScript resolves the run's script on first use. It cannot happen at
154+
// Init: the operator's knob values arrive with the first spawn message,
155+
// after Init, so a load there would only ever see --config-json. A failed
156+
// load is retried on the next iteration, so correcting the knob in the web
157+
// UI and starting a new swarm recovers without a pod restart.
158+
func (r *runtime) loadScript() (*Script, error) {
159+
r.scriptMu.Lock()
160+
defer r.scriptMu.Unlock()
161+
if r.script != nil {
162+
return r.script, nil
163+
}
164+
name := r.cfg.Dyn.Load().AgentSessionScript
165+
if name == "" {
166+
name = DefaultScript
167+
}
168+
s, err := Load(name)
169+
if err != nil {
170+
return nil, err
171+
}
172+
b := Budgets(s.Steps)
173+
slog.Info("agentsession: loaded script",
174+
slog.String("script", s.Name),
175+
slog.Int("steps", len(s.Steps)),
176+
slog.String("declared_ram", formatSize(b.RAM)),
177+
slog.String("declared_disk", formatSize(b.Disk)),
178+
slog.String("min_actor_memory", formatSize(s.MinActorMemory)))
179+
r.script = s
180+
r.ingestBuf = makeIngestBuf(s.Steps)
181+
return s, nil
182+
}
183+
184+
// checkTemplateMemory refuses to start sessions against a template whose
185+
// memory limit is below the script's min_actor_memory. A too-small actor
186+
// OOMs partway through the script and shows up as flaky steps, which is far
187+
// harder to read than a refusal at start. A template with no memory limit
188+
// is allowed through with a warning.
189+
func (r *runtime) checkTemplateMemory(ctx context.Context) error {
190+
r.templateMu.Lock()
191+
defer r.templateMu.Unlock()
192+
if r.templateOK {
193+
return nil
194+
}
195+
if r.templateErr != nil {
196+
return r.templateErr
197+
}
198+
ref := &ateapipb.ObjectRef{Atespace: templateNS, Name: templateName}
199+
tmpl, err := r.cfg.APIStub.GetActorTemplate(ctx, &ateapipb.GetActorTemplateRequest{ActorTemplate: ref})
200+
if err != nil {
201+
return fmt.Errorf("GetActorTemplate %s/%s: %w", templateNS, templateName, err)
202+
}
203+
limit, ok := memoryLimit(tmpl)
204+
switch {
205+
case !ok:
206+
slog.Warn("agentsession: template sets no memory limit; cannot check it against the script",
207+
slog.String("template", templateName),
208+
slog.String("min_actor_memory", formatSize(r.script.MinActorMemory)))
209+
case limit < r.script.MinActorMemory:
210+
r.templateErr = fmt.Errorf("template %s/%s memory limit %s is below script %q min_actor_memory %s; redeploy the workloads with --actor-memory %s",
211+
templateNS, templateName, formatSize(limit), r.script.Name, formatSize(r.script.MinActorMemory), formatSize(r.script.MinActorMemory))
212+
return r.templateErr
213+
}
214+
r.templateOK = true
215+
return nil
216+
}
217+
218+
// memoryLimit reads the template's memory limit in bytes, if it has one.
219+
func memoryLimit(tmpl *ateapipb.ActorTemplate) (int64, bool) {
220+
for _, l := range tmpl.GetResources().GetLimits() {
221+
if l.GetName() != "memory" {
222+
continue
223+
}
224+
q, err := resource.ParseQuantity(l.GetQuantity())
225+
if err != nil {
226+
return 0, false
227+
}
228+
return q.AsInt64()
229+
}
230+
return 0, false
139231
}
140232

141233
// think returns the suspended "LLM thinking" gap before step s, which is the
@@ -155,6 +247,15 @@ func (r *runtime) think(s Step) time.Duration {
155247
// script completes, so one actor keeps producing suspend/resume cycles for
156248
// the whole run.
157249
func (r *runtime) iterate() {
250+
script, err := r.loadScript()
251+
if err != nil {
252+
slog.Error("agentsession: cannot load script; goroutine will retry next iter",
253+
slog.String("err", err.Error()))
254+
time.Sleep(2 * time.Second)
255+
return
256+
}
257+
steps := script.Steps
258+
158259
gid := boomerutil.GoroutineID()
159260
val, loaded := r.users.Load(gid)
160261
if !loaded {
@@ -169,13 +270,13 @@ func (r *runtime) iterate() {
169270
}
170271
u := val.(*sessionUser)
171272

172-
step := r.steps[u.stepIndex]
273+
step := steps[u.stepIndex]
173274

174275
// The LLM is "thinking": the actor stays suspended for the gap.
175276
time.Sleep(r.think(step))
176277

177278
if u.runStep(context.Background(), step) {
178-
u.stepIndex = (u.stepIndex + 1) % len(r.steps)
279+
u.stepIndex = (u.stepIndex + 1) % len(steps)
179280
if u.stepIndex == 0 {
180281
slog.Info("agent session completed the full script; starting over",
181282
slog.String("actor", u.actorName))
@@ -195,6 +296,9 @@ func (r *runtime) iterate() {
195296
// then suspends it so the first scripted step begins — like every later
196297
// step — with a wake from suspension.
197298
func (r *runtime) startUser(ctx context.Context) (*sessionUser, error) {
299+
if err := r.checkTemplateMemory(ctx); err != nil {
300+
return nil, err
301+
}
198302
u := &sessionUser{
199303
cfg: r.cfg,
200304
actorName: "agent-" + uuid.NewString(),

0 commit comments

Comments
 (0)