Repository navigation
benchmarking: add a turnover mode to agentsession - #3421
Merged
Max Smythe (maxsmythe) merged 3 commits intoOct 30, 2026
Merged
Max Smythe (maxsmythe) merged 3 commits into
Max Smythe (maxsmythe) merged 3 commits into
Conversation
Chenyi Wang (chw120)
force-pushed
the
agentsession-turnover-steps
branch
from
October 29, 2026 02:13
51eb21b to
89dea4f
Compare
When --agentsession-awake > 0, each session registers by creating its actor, running step 1, and parking it. Once registration is idle for 1 minute, the conductor wakes --agentsession-awake sessions and swaps --target-rps idle awake sessions with parked ones each second, tracking counts in agentsession_turnover_sessions. Signed-off-by: Chenyi Wang <[email protected]>
Chenyi Wang (chw120)
force-pushed
the
agentsession-turnover-steps
branch
from
October 30, 2026 00:24
89dea4f to
af3d3b2
Compare
Max Smythe (maxsmythe)
left a comment
Max Smythe (maxsmythe)
left a comment
Collaborator
There was a problem hiding this comment.
couple more comments
…urnover stats Move conductor.step and runTurnCmd back to agentsession.go as runtime.iterateTurnover so turnover.go holds only conductor methods. Run step 1 at start in --lifecycle-mode none so resident sandboxes hold initial memory and disk state before step 1's think gap. At think scale 1, personal-assistant-light waits 55m before step 1, which outlasts a 20m resident density run. Record turnover registration parks as SuspendActor_Register (or SuspendActorAsync_Register) so SuspendActor and SuspendActorAsync hold only turnover parks. Parse boomer's "agentsession: turnover fill done" log line in runner.py to start the server_telemetry steady-state window there and append a turnover_window row to stats.jsonl with per-request counts, failures, and rates from stats_history.csv. Signed-off-by: Chenyi Wang <[email protected]>
…starts Point turnover swap-rate analysis in docs/guides/benchmarking.md at rps_TimeToWake, which records one sample per wake in both implicit and explicit resume modes, and note WakeFirstTouch alongside TimeToWake for fill wakes. Log when steady_not_before falls outside the detected steady-state window in server_telemetry.start_steady_window_at so runs whose telemetry still covers registration say so in the log. Signed-off-by: Chenyi Wang <[email protected]>
Collaborator
|
benchmarking only change, merging |
Max Smythe (maxsmythe)
enabled auto-merge
October 30, 2026 05:27
1 of 2 tasks
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Nov 2, 2026
### Summary
Adds `--agentsession-awake` (paired with `--target-rps`) to
`agentsession` so benchmarks can maintain a large pool of registered
sessions while keeping a fixed number resident and swapping idle awake
sessions with parked ones at a controlled rate.
### How It Works
- **Registration (`createSession`)**: When `--agentsession-awake > 0` or
`--lifecycle-mode none`, each session runs step 1 as soon as its actor
serves so the initial actor/snapshot holds real memory and disk state.
In `--lifecycle-mode none` at `--agentsession-think-scale 1`,
`personal-assistant-light` waits 55m before step 1, so a 20m resident
density run would otherwise measure empty sandboxes.
- **Fill**: Once session registration has been idle for 1 minute
(`registrationQuiet`), the conductor wakes `--agentsession-awake`
sessions (`fillConcurrency = 8` at a time) and logs `agentsession:
turnover fill done`.
- **Turnover (`--target-rps`)**: Each second, the conductor swaps
`--target-rps` idle awake sessions (`cmdPark`) with parked sessions
(`cmdWake`). Each woken session immediately runs its next scripted step
while marked `busy` and returns to `idle` when the step finishes.
- **Measurement isolation**:
- Registration parks record under `SuspendActor_Register` /
`SuspendActorAsync_Register` so `SuspendActor` and `SuspendActorAsync`
hold only turnover parks.
- `runner.py` parses `agentsession: turnover fill done`
(`locust/turnover_window.py`), starts `server_telemetry`'s steady-state
window there, and appends a `turnover_window` row to `stats.jsonl` with
per-request counts, failures, and rates from `stats_history.csv` over
the turnover phase.
- **Observability**: Exports
`agentsession_turnover_sessions{state="awake|parked|waking|parking"}`.
### Testing
```bash
go test -race -count=3 -shuffle=on ./internal/benchmarking/boomer/agentsession/...
python3 -P -m unittest discover -s benchmarking/locust/unit_tests -p "test_*.py"
```
---------
Signed-off-by: Chenyi Wang <[email protected]>
(cherry picked from commit 5e22427)
Signed-off-by: Eitan Yarmush <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
--agentsession-awake(paired with--target-rps) toagentsessionso benchmarks can maintain a large pool of registered sessions while keeping a fixed number resident and swapping idle awake sessions with parked ones at a controlled rate.How It Works
createSession): When--agentsession-awake > 0or--lifecycle-mode none, each session runs step 1 as soon as its actor serves so the initial actor/snapshot holds real memory and disk state. In--lifecycle-mode noneat--agentsession-think-scale 1,personal-assistant-lightwaits 55m before step 1, so a 20m resident density run would otherwise measure empty sandboxes.registrationQuiet), the conductor wakes--agentsession-awakesessions (fillConcurrency = 8at a time) and logsagentsession: turnover fill done.--target-rps): Each second, the conductor swaps--target-rpsidle awake sessions (cmdPark) with parked sessions (cmdWake). Each woken session immediately runs its next scripted step while markedbusyand returns toidlewhen the step finishes.SuspendActor_Register/SuspendActorAsync_RegistersoSuspendActorandSuspendActorAsynchold only turnover parks.runner.pyparsesagentsession: turnover fill done(locust/turnover_window.py), startsserver_telemetry's steady-state window there, and appends aturnover_windowrow tostats.jsonlwith per-request counts, failures, and rates fromstats_history.csvover the turnover phase.agentsession_turnover_sessions{state="awake|parked|waking|parking"}.Testing