Change dsr1 fp8 image to lmsysorg/sglang 0.5.5.post3 and fp4 image to 0.5.5.post2 for AMD MI355 by rkarhila-amd · Pull Request #247 · SemiAnalysisAI/InferenceX

rkarhila-amd · 2025-11-19T14:27:17Z

A small but significantly performance increasing change. MI300 and MI325 to follow after testing. Slurm changes untested.

…-mi35x for dsr1-fp8

functionstackx · 2025-11-23T01:06:51Z

@rkarhila-amd thanks for the PR. did u happen to change the wrong thing in amd-master.yaml, i see you editted only fp8 bash scripts and ur title says 'dsr1 fp8 image' but then u only editted dsr1 fp4`

* Adjust preview for dark mode and light mode picture element for better display based on color scheme. * Rounded * Update image alt text in README.md

Co-authored-by: Jatin Gangani <[email protected]>

…#227) * initial poc * remove -d flag when launching docker container * syntax error * compatibility fixes * add correct endpoint prefix * remove reference env var * run vllm serve in background * unescape sequences * stop vllm to stdout after it stops * stop vllm to stdout after it stops pt 2 * get rid of docker stop as no longer in detatched * clone bench serving to tmp dir * clone bench serving to tmp dir pt 2 * add explanatory comment * cleaning up * cleaning up * adding mi355x refactor * adding h200 initial refactor * different way to see server logs * cleanup * now fail if server fails * starting on b200 * doign b200 * reverting erroneous change * fixing b200 * fixing b200 pt 2 * updating mi300 * updating mi300 pt 2 * updating mi300 pt 3 -- remove detached mode * cleaning up mi355x * fixing mi300x and updating 325x * reverting max conc to 512 on gptoss fp4 b200 docker * fixing mi300x and updating 325x * cleanng up * add wait for h200 slurm dsr1 * max num seqs back to 512 for gptoss fpr b200 docker * fix port issue for dsr1 mi300x docker * fix mi355x docker NUM_PROMPTS * adding prop of failure for server logs * add utils function for benchmark * add utils function for benchmark * function-ize the waiting for server to start * dont show arg parsing set -x * dont show arg parsing set +x oops * dont show arg parsing set +x oops * capture server pid * nebdius dont scancel * changes to comments in benchmark lib . sh * Update benchmarks/dsr1_fp4_mi355x_docker.sh Co-authored-by: Copilot <[email protected]> * Update .github/workflows/benchmark-tmpl.yml Co-authored-by: Copilot <[email protected]> * adding back whitespace * adding back whitespace * adding back whitespace * remove tg launch script * Update benchmarks/gptoss_fp4_h100_docker.sh Co-authored-by: Copilot <[email protected]> * Update benchmarks/dsr1_fp8_mi325x_docker.sh Co-authored-by: Copilot <[email protected]> * Update benchmarks/dsr1_fp8_mi355x_docker.sh Co-authored-by: Copilot <[email protected]> * Update benchmarks/gptoss_fp4_b200_trt_slurm.sh Co-authored-by: Copilot <[email protected]> * Audit and correct required environment variables documentation in all benchmark scripts (SemiAnalysisAI#252) * Initial plan * Update required env vars documentation in all benchmark scripts Co-authored-by: cquil11 <[email protected]> * Fix required env vars - remove NF, PREFILL_SIZE, and correct PORT/PORT_OFFSET Co-authored-by: cquil11 <[email protected]> * Remove internally-calculated vars from required env vars (EXTRA_CONFIG_FILE, MAX_NUM_TOKENS, MOE_BACKEND) Co-authored-by: cquil11 <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: cquil11 <[email protected]> * removing oci node rebase with main --------- Co-authored-by: Copilot <[email protected]> Co-authored-by: Copilot <[email protected]>

…SemiAnalysisAI#253) Bumps the github-actions group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5.0.0 to 6.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@08c6903...1af3b93) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#259) Bumps the github-actions group with 1 update: [actions/setup-python](https://github.com/actions/setup-python). Updates `actions/setup-python` from 6.0.0 to 6.1.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@e797f83...83679a8) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bumps DSR1 SGLang code * update how we get the resulting log files --------- Co-authored-by: Elnifio <[email protected]> Co-authored-by: Cameron Quilici <[email protected]>

* Update GPTOSS B200 AGG * set dp attention env vars * Add DP attn comment --------- Co-authored-by: Jatin Gangani <[email protected]>

rkarhila-amd · 2025-12-03T14:41:06Z

did u happen to change the wrong thing in

Yeah, sorry, my bad!

We're now updating this PR to include some changes to runners to enable FP4 on the same lmsys container.

cquil11 · 2025-12-03T16:47:41Z

benchmarks/dsr1_fp8_mi355x_docker.sh

+export ROCM_QUICK_REDUCE_QUANTIZATION=INT4
+
+
+SERVER_LOG=$(mktemp /tmp/server-XXXXXX.log)


cquil11 · 2025-12-03T16:48:07Z

benchmarks/dsr1_fp8_mi355x_slurm.sh

+export RCCL_MSCCL_ENABLE=0
+export ROCM_QUICK_REDUCE_QUANTIZATION=INT4
+
+


nit whitespace

.github/configs/amd-master.yaml

cquil11 · 2025-12-03T20:24:56Z

old perf

Model	Hardware	Framework	Precision	ISL	OSL	TP	EP	DP Attention	Conc	TTFT (ms)	TPOT (ms)	Interactivity (tok/s/user)	E2EL (s)	TPUT per GPU	Output TPUT per GPU	Input TPUT per GPU
deepseek-ai/DeepSeek-R1-0528	MI355X	SGLANG	FP8	1024	1024	8	1	false	4	99.1861	18.5055	54.0381	17.1707	53.4397	26.7035	26.7362

https://github.com/InferenceMAX/InferenceMAX/actions/runs/19776095504#:~:text=206.2476-,deepseek%2Dai/DeepSeek%2DR1%2D0528,26.7362,-deepseek%2Dai/DeepSeek

new perf

Model	Hardware	Framework	Precision	ISL	OSL	TP	EP	DP Attention	Conc	TTFT (ms)	TPOT (ms)	Interactivity (tok/s/user)	E2EL (s)	TPUT per GPU	Output TPUT per GPU	Input TPUT per GPU
deepseek-ai/DeepSeek-R1-0528	MI355X	SGLANG	FP8	1024	1024	8	1	false	4	91.5310	18.4025	54.3404	17.0457	53.7069	26.8370	26.8698

https://github.com/InferenceMAX/InferenceMAX/actions/runs/19902871108?pr=247#:~:text=deepseek%2Dai/DeepSeek,26.8698

looks good to me, if all good with you? @rkarhila-amd
please just address comments and we should be good to go

cquil11 · 2025-12-03T20:28:27Z

pls also correct PR description

Update amd dsr1fp4 images

rkarhila-amd · 2025-12-04T17:30:02Z

Thanks @cquil11 , good to go from my side!

cquil11

lgtm, thank you!

Change AMD MI355 docker image to lmsysorg/sglang:v0.5.5.post2-rocm700…

952b218

…-mi35x for dsr1-fp8

rkarhila-amd requested a review from a team as a code owner November 19, 2025 14:27

Merge branch 'main' into update_AMD_dsr1fp8_images

15452f7

functionstackx added the mi355x_dsr1 label Nov 21, 2025

This was referenced Dec 2, 2025

integrate upstream ROCm SGLang images into InferenceMAX #263

Closed

going forward use community builds upstream ROCm vllm/sglang images instead of AMD's fork images #122

Open

TP4 FP4 for MI355X SGLang #265

Closed

austenstone and others added 9 commits December 3, 2025 16:33

Adjust preview for dark mode and light mode (SemiAnalysisAI#250)

2590adf

* Adjust preview for dark mode and light mode picture element for better display based on color scheme. * Rounded * Update image alt text in README.md

adding ISL/OSL to collect results table summary (SemiAnalysisAI#249)

f527687

Co-authored-by: Jatin Gangani <[email protected]>

Add b200 DGXC node to b200 runners list (SemiAnalysisAI#245)

ba61398

feat: refresh GB200 SGLang DSR1 submission (SemiAnalysisAI#257)

1520664

* Bumps DSR1 SGLang code * update how we get the resulting log files --------- Co-authored-by: Elnifio <[email protected]> Co-authored-by: Cameron Quilici <[email protected]>

Update GPTOSS B200 TRTLLM (SemiAnalysisAI#266)

de4245a

* Update GPTOSS B200 AGG * set dp attention env vars * Add DP attn comment --------- Co-authored-by: Jatin Gangani <[email protected]>

Merge branch 'main' into update_AMD_dsr1fp8_images

3335bec

Fixed community container for MI35x dsr1 for fp8 for real

fdedafb

cquil11 reviewed Dec 3, 2025

View reviewed changes

.github/configs/amd-master.yaml Show resolved Hide resolved

cquil11 added 2 commits December 3, 2025 11:17

Merge branch 'main' into update_AMD_dsr1fp8_images

a3f6a6a

Merge branch 'main' into update_AMD_dsr1fp8_images

a470d47

cquil11 added mi355x_dsr1 and removed mi355x_dsr1 labels Dec 3, 2025

cquil11 temporarily deployed to fork-pr-validation December 3, 2025 17:22 — with GitHub Actions Inactive

cquil11 added mi355x_dsr1 and removed mi355x_dsr1 labels Dec 3, 2025

cquil11 temporarily deployed to fork-pr-validation December 3, 2025 17:27 — with GitHub Actions Inactive

functionstackx mentioned this pull request Dec 3, 2025

chore(config): use a more up to date container for rocm/sgl #200

Closed

ppalanga and others added 3 commits December 4, 2025 03:17

Update dsr1_fp4_mi355x_docker.sh with env flags

07d3877

Update dsr1_fp4_mi355x_slurm.sh

5bdbd31

Merge pull request #1 from rkarhila-amd/update_AMD_dsr1fp4_image

e0706e6

Update amd dsr1fp4 images

rkarhila-amd temporarily deployed to fork-pr-validation December 4, 2025 16:35 — with GitHub Actions Inactive

from post2 to post3 for fp8

cbd0608

rkarhila-amd temporarily deployed to fork-pr-validation December 4, 2025 16:56 — with GitHub Actions Inactive

rkarhila-amd temporarily deployed to fork-pr-validation December 4, 2025 17:06 — with GitHub Actions Inactive

tidy formatting

e55d426

rkarhila-amd changed the title ~~Change dsr1 fp8 image to lmsysorg/sglang 0.5.5.post2 for AMD MI355~~ Change dsr1 fp8 image to lmsysorg/sglang 0.5.5.post3 and fp4 image to 0.5.5.post2 for AMD MI355 Dec 4, 2025

cquil11 removed the mi355x_dsr1 label Dec 4, 2025

Merge branch 'main' into update_AMD_dsr1fp8_images

1a491ac

cquil11 added the mi355x_dsr1 label Dec 4, 2025

cquil11 temporarily deployed to fork-pr-validation December 4, 2025 23:39 — with GitHub Actions Inactive

cquil11 mentioned this pull request Dec 4, 2025

ROCm SGLang: make env vars default #290

Open

cquil11 enabled auto-merge (squash) December 5, 2025 00:02

cquil11 approved these changes Dec 5, 2025

View reviewed changes

cquil11 merged commit 0fe9dcc into SemiAnalysisAI:main Dec 5, 2025
3 checks passed

cquil11 mentioned this pull request Dec 5, 2025

Revert "Change dsr1 fp8 image to lmsysorg/sglang 0.5.5.post3 and fp4 image to 0.5.5.post2 for AMD MI355" #295

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change dsr1 fp8 image to lmsysorg/sglang 0.5.5.post3 and fp4 image to 0.5.5.post2 for AMD MI355#247

Change dsr1 fp8 image to lmsysorg/sglang 0.5.5.post3 and fp4 image to 0.5.5.post2 for AMD MI355#247
cquil11 merged 20 commits intoSemiAnalysisAI:mainfrom
rkarhila-amd:update_AMD_dsr1fp8_images

rkarhila-amd commented Nov 19, 2025

Uh oh!

functionstackx commented Nov 23, 2025

Uh oh!

rkarhila-amd commented Dec 3, 2025

Uh oh!

cquil11 Dec 3, 2025

Uh oh!

cquil11 Dec 3, 2025

Uh oh!

Uh oh!

cquil11 commented Dec 3, 2025

Uh oh!

cquil11 commented Dec 3, 2025

Uh oh!

rkarhila-amd commented Dec 4, 2025

Uh oh!

cquil11 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

		export ROCM_QUICK_REDUCE_QUANTIZATION=INT4


		SERVER_LOG=$(mktemp /tmp/server-XXXXXX.log)

		export RCCL_MSCCL_ENABLE=0
		export ROCM_QUICK_REDUCE_QUANTIZATION=INT4

Conversation

rkarhila-amd commented Nov 19, 2025

Uh oh!

functionstackx commented Nov 23, 2025

Uh oh!

rkarhila-amd commented Dec 3, 2025

Uh oh!

cquil11 Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

cquil11 Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cquil11 commented Dec 3, 2025

Uh oh!

cquil11 commented Dec 3, 2025

Uh oh!

rkarhila-amd commented Dec 4, 2025

Uh oh!

cquil11 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants