For #304: GH comment hook by Oseltamivir · Pull Request #306 · SemiAnalysisAI/InferenceX

Oseltamivir · 2025-12-08T07:46:16Z

Summary

Addresses #304, Adds PR comment–triggered sweeps via /sweep ..., removing reliance on labels. Keeps logic DRY by reusing existing benchmark templates and a new reusable executor.

How

issue_comment on PRs parses the first /sweep line and forwards args to a reusable workflow.
Executor checks out PR head, runs utils/matrix_logic/generate_sweep_configs.py, detects single vs multi, dispatches to benchmark-tmpl.yml or benchmark-multinode-tmpl.yml, then collects results.

What

Added: .github/workflows/pr-comment-sweep.yml: small parser; auth gate; per‑PR concurrency.
Added: .github/workflows/sweep-executor.yml: generates matrix, runs benchmarks, aggregates results.

Examples

/sweep full-sweep --single-node --runner-type h200 --model-prefix dsr1 --seq-lens 1k1k
/sweep full-sweep --multi-node --model-prefix gptoss --seq-lens 1k1k
/sweep runner-model-sweep --runner-type h200 --single-node
No need to pass config paths; executor appends --config-files and --runner-config.

Misc & Security

Only runs on PR comments from OWNER/MEMBER/COLLABORATOR.
Workflows execute with secrets: inherit; generator runs on PR head.

Notes

issue_comment workflows must exist on the default branch to trigger. Currently tries a workflow_dispatch workaround
Benchmarks currently checkout default branch (matching existing templates). If desired, we can add a ref input to run jobs on PR head.
Label-based path remains unchanged.

Oseltamivir · 2025-12-08T15:21:10Z

Can't be actually tested until pushed to main I believe, but the there is a default: full-sweep --single-node --runner-type h200 --model-prefix dsr1 --seq-lens 1k1k --max-conc 4 that is ran here on push

cquil11 · 2025-12-10T16:03:45Z

.github/workflows/pr-comment-sweep.yml

+          pr_from_branch='${{ steps.find.outputs.pr-number }}'
+          pr_number="${DISPATCH_PR:-}"; if [[ -z "$pr_number" ]]; then pr_number="$pr_from_branch"; fi
+          gen_args="${DISPATCH_ARGS:-}"
+          if [[ -z "$gen_args" ]]; then


no default
should just fail in this case

The default(gen_args='full-sweep --sin...) is for testing, will remove when ready to merge

.github/workflows/pr-comment-sweep.yml

github-actions · 2025-12-15T15:38:25Z

@Oseltamivir pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20238048007
(Push on refs/heads/slash-commands)

github-actions · 2025-12-15T15:41:58Z

@Oseltamivir pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20238135307
(Push on refs/heads/slash-commands)

github-actions · 2025-12-15T15:45:26Z

@Oseltamivir pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20238250232
(Push on refs/heads/slash-commands)

cquil11 · 2025-12-17T00:15:47Z

@Oseltamivir can the callback message / comment include the command used to run the sweep?

@Oseltamivir pushed changes and triggered a sweep. With the command <Code formatted command>.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20238250232
(Push on refs/heads/slash-commands)

) [skip-sweep] * add logic for event driven runs new single workflow that runs on merge to main, new perg-changelog.yaml to track performance changes, new logic to parse changelog, removed cron job in full sweep schedulers * testing pt 1 * raise error if yaml diff in perf changelog is not valid * remove unused imports in process_changelog.py * config data key fix * raise error if test-config subprocess fails to run * backfill changelog * backfill changelog pt 2 * backfill changelog pt 3 * backfill changelog pt 4 * backfill changelog pt 5 * backfill changelog pt 6 * add always() condition to upload changelog metadata * backfill changelog pt 7 (test) * backfill changelog pt 8 (revert test) * backfill changelog pt 9 * backfill changelog pt 11 * change if condition for jobs in run sweep workflow * debugging run sweep workflow * debugging run sweep workflow pt 2 * debugging run sweep workflow pt 3 (revert) * debugging run sweep workflow pt 4 * debugging run sweep workflow pt 5 * debugging run sweep workflow pt 6 * debugging run sweep workflow pt 7 * add always() condition to upload changelog metadata (add back, this got removed) * add bmk prefix to results * backfill changelog official * for concurrency group, use more unique sha

…updates (#331) Bumps the github-actions group with 3 updates in the / directory: [actions/checkout](https://github.com/actions/checkout), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/download-artifact](https://github.com/actions/download-artifact). Updates `actions/checkout` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...8e8c483) Updates `actions/upload-artifact` from 5.0.0 to 6.0.0 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@330a01c...b7c566a) Updates `actions/download-artifact` from 6.0.0 to 7.0.0 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@018cc2c...37930b1) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: actions/upload-artifact dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/download-artifact dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…wont be erroneous negative diff [skip-sweep] (#333)

* Update amd-master.yaml * Update perf-changelog.yaml * Update dsr1_fp4_mi355x_docker.sh * Update dsr1_fp4_mi355x_docker.sh --------- Co-authored-by: Cameron Quilici <[email protected]>

github-actions · 2025-12-17T01:59:00Z

@Oseltamivir pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20289011735
(Push on refs/heads/slash-commands)

github-actions · 2025-12-17T02:02:43Z

@cquil11 pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20289080590
(Push on refs/heads/slash-commands)

github-actions · 2025-12-17T03:06:15Z

@Oseltamivir pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20290213946
Command: full-sweep --single-node --runner-type h200 --model-prefix dsr1 --seq-lens 1k1k --max-conc 4
Pinned ref: c13b7f7
(Push on refs/heads/slash-commands)

github-actions · 2025-12-17T16:11:12Z

@Oseltamivir pushed changes and triggered a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20309360272
Command: full-sweep --single-node --runner-type h200 --model-prefix dsr1 --seq-lens 1k1k --max-conc 4
Pinned ref: 80e40d4
(Push on refs/heads/slash-commands)

* Add benchmark script for GPTOSS FP4 B200 TRT-LLM * make changes to perf changelog --------- Co-authored-by: Cameron Quilici <[email protected]>

cquil11

lgtm

Oseltamivir requested a review from a team as a code owner December 8, 2025 07:46

github-project-automation bot added this to InferenceMAX Board Dec 8, 2025

Oseltamivir changed the title ~~Initial commit, for #304~~ For #304: GH comment hook Dec 8, 2025

Oseltamivir force-pushed the slash-commands branch from b8c1a9c to 7ae656f Compare December 8, 2025 15:07

Oseltamivir force-pushed the slash-commands branch from c2ce131 to 66ac63d Compare December 9, 2025 15:04

Oseltamivir added 4 commits December 9, 2025 07:04

Initial commit, for #304

a81882f

Allow testing on own PR

d27616a

condense workflow

dce0b32

Rename Workflow

503ed3b

Oseltamivir force-pushed the slash-commands branch from 66ac63d to 503ed3b Compare December 9, 2025 15:06

Use environments

f1feee4

Oseltamivir temporarily deployed to bryan-test December 10, 2025 14:40 — with GitHub Actions Inactive

cquil11 reviewed Dec 10, 2025

View reviewed changes

.github/workflows/pr-comment-sweep.yml Outdated Show resolved Hide resolved

Oseltamivir force-pushed the slash-commands branch from 05bc10d to f1feee4 Compare December 10, 2025 16:35

Changed environment location

9a5fe6e

Oseltamivir temporarily deployed to bryan-test December 10, 2025 16:35 — with GitHub Actions Inactive

Stricter activation

4524540

Oseltamivir temporarily deployed to bryan-test December 10, 2025 16:46 — with GitHub Actions Inactive

Test replies

91e6d6c

Oseltamivir temporarily deployed to bryan-test December 15, 2025 02:27 — with GitHub Actions Inactive

Oseltamivir added 2 commits December 15, 2025 07:27

Test replies

ac35382

Use token for comment perm

c09922a

Oseltamivir force-pushed the slash-commands branch from fce8ccf to c09922a Compare December 15, 2025 15:38

Forgot validation

4f9eb0c

Oseltamivir force-pushed the slash-commands branch from b013cac to 4f9eb0c Compare December 15, 2025 15:44

Oseltamivir temporarily deployed to bryan-test December 15, 2025 15:45 — with GitHub Actions Inactive

functionstackx moved this to In Progress in InferenceMAX Board Dec 15, 2025

cquil11 and others added 4 commits December 16, 2025 17:58

fix: add final newline to original perf-changelog.yaml so that there …

2a85d1d

…wont be erroneous negative diff [skip-sweep] (#333)

Update MI355x Deepseek-R1 FP4 SGLang Image to v0.5.6.post1 (#330)

1a3e65c

* Update amd-master.yaml * Update perf-changelog.yaml * Update dsr1_fp4_mi355x_docker.sh * Update dsr1_fp4_mi355x_docker.sh --------- Co-authored-by: Cameron Quilici <[email protected]>

Oseltamivir temporarily deployed to bryan-test December 17, 2025 01:59 — with GitHub Actions Inactive

Merge branch 'main' into slash-commands

a0e1b7e

cquil11 temporarily deployed to bryan-test December 17, 2025 02:02 — with GitHub Actions Inactive

TOCTOU

c13b7f7

Oseltamivir temporarily deployed to bryan-test December 17, 2025 03:06 — with GitHub Actions Inactive

Test new env

80e40d4

Oseltamivir temporarily deployed to Outside Collaborator E2E Test December 17, 2025 16:11 — with GitHub Actions Inactive

Oseltamivir and others added 3 commits December 17, 2025 08:53

Ready for merge

724c370

Add benchmark script for GPTOSS FP4 B200 TRT-LLM (#256)

a397f9a

* Add benchmark script for GPTOSS FP4 B200 TRT-LLM * make changes to perf changelog --------- Co-authored-by: Cameron Quilici <[email protected]>

Merge branch 'main' into slash-commands

b9faa9d

cquil11 approved these changes Dec 17, 2025

View reviewed changes

cquil11 merged commit c97d4fa into main Dec 17, 2025

cquil11 deleted the slash-commands branch December 17, 2025 17:48

github-project-automation bot moved this from In Progress to Done in InferenceMAX Board Dec 17, 2025

cquil11 mentioned this pull request Dec 17, 2025

chore: remove auto validation workflow via label #344

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

For #304: GH comment hook#306

For #304: GH comment hook#306
cquil11 merged 21 commits intomainfrom
slash-commands

Oseltamivir commented Dec 8, 2025

Uh oh!

Oseltamivir commented Dec 8, 2025 •

edited

Loading

Uh oh!

cquil11 Dec 10, 2025

Uh oh!

Oseltamivir Dec 10, 2025

Uh oh!

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

cquil11 commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

cquil11 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Oseltamivir commented Dec 8, 2025

Summary

How

What

Examples

Misc & Security

Notes

Uh oh!

Oseltamivir commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cquil11 Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Oseltamivir Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

cquil11 commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

cquil11 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Oseltamivir commented Dec 8, 2025 •

edited

Loading