Skip to content

benchmarking: what to set and watch before a dense run - #2499

Merged
Max Smythe (maxsmythe) merged 3 commits into
agent-substrate:mainfrom
dims:benchmarking-readme-scale
Oct 28, 2026
Merged

Max Smythe (maxsmythe) merged 3 commits into
agent-substrate:mainfrom
dims:benchmarking-readme-scale

Conversation

@dims

Copy link
Copy Markdown
Collaborator

A dense run meets limits the demo path never reaches. One new section lists them with the setting for each: reaching the API and router without kubectl port-forward, PostgreSQL slots and probe timeouts, the parking budget, synchronized bursts against the ext_proc breaker, the kernel neighbor table, restores under full load, kubelet CPU reservation, and what to watch during the run.

Docs only.

@dims
Davanum Srinivas (dims) marked this pull request as ready for review October 12, 2026 15:15
@dims
Davanum Srinivas (dims) force-pushed the benchmarking-readme-scale branch 2 times, most recently from 0cc3c75 to 067bce6 Compare October 14, 2026 00:42
@maxsmythe

Copy link
Copy Markdown
Collaborator

Thank you for this! Have a couple of questions for clarity, although the general idea of a scaling guide is a good one.

haiyanmeng you may find this interesting.

@maxsmythe Max Smythe (maxsmythe) added kind/docs Improvements or additions to documentation area/benchmarking labels Oct 14, 2026
@haiyanmeng
haiyanmeng self-requested a review October 14, 2026 12:46

@haiyanmeng haiyanmeng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution.

May I ask how did you identify these best practices? Did you encounter these bottlenecks while running the benchmark tests in a dense mode?

Comment thread benchmarking/README.md Resolved
Comment thread benchmarking/README.md Resolved
Comment thread benchmarking/README.md Resolved
@dims

Copy link
Copy Markdown
Collaborator Author

May I ask how did you identify these best practices? Did you encounter these bottlenecks while running the benchmark tests in a dense mode?

Yes, exactly what i did haiyanmeng some of what i did i am slowly pushing to https://github.com/dims/gym-substrate FYI

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments... forgot that they were in draft mode.

Comment thread benchmarking/README.md Resolved
Comment thread benchmarking/README.md Resolved
Comment thread benchmarking/README.md Resolved
Comment thread benchmarking/README.md Resolved
@dims

Copy link
Copy Markdown
Collaborator Author

closing and reopening to re-trigger a flaky ci test

@dims
Davanum Srinivas (dims) force-pushed the benchmarking-readme-scale branch 8 times, most recently from d134a37 to 8b20fa5 Compare October 22, 2026 12:35
@dims
Davanum Srinivas (dims) force-pushed the benchmarking-readme-scale branch 3 times, most recently from 808afb7 to 572a5d5 Compare October 23, 2026 15:48

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you for this!

Giving haiyanmeng a chance for any follow up comments before merging.

@dims
Davanum Srinivas (dims) force-pushed the benchmarking-readme-scale branch 2 times, most recently from bee26e9 to b78308d Compare October 26, 2026 15:20
Limits a dense run meets that the demo path never reaches: port-forward
under load, PostgreSQL slots and probes, the parking budget, synchronized
bursts, the neighbor table, restores under full load, the kubelet.

Signed-off-by: Davanum Srinivas <[email protected]>
The pool flag has no default of 20: unset, pgx sizes the pool at the
node CPU count. The port-forward point applies only to a load generator
outside the cluster. Only the liveness probe restarts postgres.

Signed-off-by: Davanum Srinivas <[email protected]>
Name the oversubscription behind "2x", say headroom rather than steering
restores (Substrate has no such control), and spell out that the postgres
probe fix includes the container resource requests.

Signed-off-by: Davanum Srinivas <[email protected]>
@maxsmythe

Copy link
Copy Markdown
Collaborator

Merging... doc only change.

@maxsmythe
Max Smythe (maxsmythe) added this pull request to the merge queue Oct 28, 2026
Merged via the queue into agent-substrate:main with commit f20af35 Oct 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/benchmarking area/network kind/docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants