Skip to content

About

Qwen3.8-Flash-Next-FP8 on AMD MI325X (gfx942): measured tuning results and drop-in vLLM v0.32.0 settings

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

20 Commits

Folders and files

Repository files navigation

Qwen3.8-Flash-Next-FP8 on AMD MI325X

Serving settings and a two-file vLLM overlay that roughly double prefill and halve agent wall time for Qwen3.8-Flash-Next-FP8 on MI325X (gfx942), behind a LiteLLM gateway. Tuned AITER MoE tables are included; their only isolated end-to-end measurement moved 8 × 32K prefill by under 1%, on TP1.

Results

Plain vLLM v0.32.0 (same flags; no overlay, no tuned tables, no QuickReduce) against this repo, TP2 + expert parallel, run d02 (October). Run-to-run noise is about 3-5%, more on cold TTFT: stock measured 3.48 s and 2.92 s in two runs with the same flags.

TP2, one engine on 2 GPUs stock this repo
Prefill, 8 x 32K concurrent 11,469 tok/s 24,266 tok/s
32K cold TTFT, single request 3.48 s 1.51 s
Decode, 64 concurrent 2,167 tok/s 3,208 tok/s
Agent 32K, 16 sessions x 3 turns, wall 63.1 s 31.9 s
Agent 96K, 8 sessions x 3 turns, wall 84.9 s 41.7 s
GSM8K, 1319 questions zero-shot 0.9629

Accuracy gate for the same run: 238K needle 3/3, functional 9/9.

At 100 simulated users on 2 x TP2 through LiteLLM, with the first overlay revision: 0 errors in 2,312 requests, warm TTFT p50 0.58 s, p99 49.8 s, 94% warm-turn cache hit.

Re-measured 2026-11-03 with the refreshed overlay. Here QuickReduce INT6 and the tuned MoE table are on in both columns, so the difference is the overlay (and, in the last two columns, the newer vLLM). Medians of 3 passes.

TP2 + EP2, MTP 3 v0.32.0 stock v0.32.0 + overlay v0.33.0 (ROCm 10.0) + overlay v0.33.0-rocm72 + overlay
Prefill, 8 x 32K concurrent (tok/s) 12,966 25,917 27,271 26,540
32K cold TTFT (s) 2.94 1.54 1.48 1.48
Decode, 64 concurrent (tok/s) 2,262 3,292 3,552 3,417
Agent 32K, 16 sessions, wall (s) 56.5 32.1 29.1 30.8
Agent 96K, 8 sessions, wall (s) 77.4 42.4 39.1 39.4
GSM8K-200 0.975 0.980 0.965 0.975

Caveats:

  • The newest numbers are one boot per arm, 3 passes, medians; ranges are in results/overlay-refresh-2026-11/.
  • vLLM v0.33.0 gains 2-7% over v0.32.0 on the same ROCm 7.2 (v0.33.0-rocm72) and 5-9% on the ROCm 10.0 build (v0.33.0 tag). Both were tested with the overlay.
  • GSM8K-200 has a standard error of about 1.2 points; the 1319-question run is the accuracy gate (v0.33.0 below).

2026-11-16, 512K context and the v0.33.0 gate (results/v033-512k-2026-11/), one boot per arm, 512K unless noted:

TP2 + EP2, MTP 3 v0.32.0 + d02 overlay, 262K v0.32.0 + d02 overlay v0.33.0 (ROCm 10.0) + overlay
GSM8K-1319 0.9568 0.9644 0.9644
500K retrieval (4 facts), TTFT n/a 4/4, 24.9 s 4/4, 22.8 s
Decode, 1 stream, short prompt (tok/s) 166 ¹ 241
50 simulated users, 15 min: cold / warm TTFT p50 14.2 / 0.53 s 14.8 / 0.56 s 14.0 / 0.49 s

512K comes from YaRN factor 2 in a copy of config.json; 1M (factor 4) also passed but cost 28% cold TTFT. With MTP, YaRN must not go through --hf-overrides: the MTP draft ignores dict overrides and the engine faults past position 262,144. The v0.33.0 column also uses the newer overlay, so its decode gain mixes image and overlay revision (the overlay's decode change is inactive with MTP). These are the deploy/ tp2x2 defaults now. ¹ measured on the b32 arm (same, plus QuickReduce max 256 MB and the deploy/ model-dir layout).

Upgrading an existing deploy/.env: add VIDEO_GID / RENDER_GID (compose now requires them), and note that tp2x2 reads TP2_MAX_MODEL_LEN (default 524288), not MAX_MODEL_LEN. VLLM_IMAGE is shared: tp1x4 users should keep the v0.32.0 line.

Quick start

git clone https://github.com/tc3oliver/qwen3.8-flash-next-mi325x && cd qwen3.8-flash-next-mi325x/deploy
cp .env.example .env && chmod 600 .env   # set MODEL_DIR, VIDEO_GID/RENDER_GID, HIP ids, LITELLM_MASTER_KEY, LAYOUT (tp2x2 or tp1x4)
./scripts/up.sh                          # docker compose; overlay/ and tuned/ are mounted by compose.yaml; tp2x2 builds config/model-yarn512 first
./scripts/status.sh

Pick the HIP device ids for your host first; the mapping in .env.example is from our machine.

What is included

  • overlay/: qsa.py and indexer_qsa.py for vLLM v0.32.0 / v0.33.0 (tiled QSA indexer, sparse-GQA BLOCK_N=32), kernel tests, stock hashes.
  • tuned/aiter/: tuned AITER fused-MoE tables for TP1 and TP2 + EP2 (304 CUs).
  • deploy/: compose for 2 x TP2 or 4 x TP1 plus LiteLLM with session affinity, and operating scripts.
  • results/: raw data for every number above.
  • upstream/: the upstream PRs with their evidence.
  • tools/: benchmark and test scripts.

Caveat: per-row decode (part of #62589) is inactive with MTP 3; the shipped deployment uses the tiled kernel for decode.

Validated on

vllm/vllm-openai-rocm:v0.32.0 @ sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905 (ROCm 7.2.3, torch 2.13, aiter 0.1.24.post1) and, for tp2x2 at 512K, v0.33.0 @ sha256:4fcd39b758b660c4e9c345c021e2ea8f0ccdb31e3c32507d676cc3005ae83fba (ROCm 10.0, aiter 0.1.26), MI325X (gfx942, 304 CUs, 256 GiB HBM), model Qwen/Qwen3.8-Flash-Next-FP8 @ 236dfdf285828023ca3bcd3f37366c58a3469b13.

Upstream status (2026-11-16)

PR content status
vllm#62589 tiled QSA indexer open, no maintainer review
vllm#62590 sparse-GQA launch profile on gfx942 and gfx950 open, no maintainer review
vllm#64662 MI325X Triton FP8 MoE configs open, no maintainer review
aiter#6692 MoE tuner fallback merged 2026-11-03
aiter#6693 gfx942 MoE configs merged 2026-10-22; not yet in the vLLM v0.32.0 / v0.33.0 images, so tuned/aiter/ is still needed

CI on the vLLM PRs waits for a maintainer ready label. The #64662 configs do not apply to the shipped AITER-MoE deployment.

Details

License

Apache-2.0 for this repository. overlay/*.py and the upstream/*/qsa.py files are modified vLLM source (Apache-2.0). The model weights are under the Qwen Community License and are not included.

About

Qwen3.8-Flash-Next-FP8 on AMD MI325X (gfx942): measured tuning results and drop-in vLLM v0.32.0 settings

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages