Repository navigation
[Config] Qwen3.8-Flash-Next FP8 fused-MoE tuned configs for gfx942 (MI325X, 304 CUs) - #6693
Conversation
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
One backend per PR: PR title tags & labels: |
…I325X, 304 CUs) Fused MoE FP8 per_1x128 rows for Qwen3.8-Flash-Next-FP8 (model_dim 2560, inter_dim 640, top-10, SiLU, g1u1) on MI325X (gfx942, cu_num 304): E=512 (TP1, 3 rows) and E=256 (TP2 + EP, 5 rows). Only shapes at least 3% faster than the current default are written (gemm_moe_tune.py --compare --update_improved). Rows are unchanged tuner output. Signed-off-by: Oliver Yu <[email protected]>
f806d41 to
c72c2fd
Compare
zufayu
left a comment
There was a problem hiding this comment.
Config-only gfx942/304-CU tuning rows with scoped keys and measured improvements. LGTM.
Motivation
No fused-MoE rows exist on main for Qwen3.8-Flash-Next-FP8 on gfx942 with
cu_num=304(MI325X). Routed experts: FP8per_1x128block scale, model_dim 2560, inter_dim 640, top-10, SiLU, g1u1. Naming follows the open gfx950 PR #5870. #5379 (MI308X,cu_num=80) has no overlapping keys.Changes
a8w8_blockscale_tuned_fmoe_qwen3_8_flash_next_gfx942.csv: 8 rows, E=512 (TP1: M=4/16384/32768) and E=256 (TP2 + EP: M=2/4/1024/8192/16384). Unmodified tuner output. All are asm 1-stagefmoe_bf16_blockscaleFp8_g1u1_{vs,novs}_silu_1tg_32x128, err 0.0%.a8w8_blockscale_untuned_fmoe_qwen3_8_flash_next_gfx942.csv: M=1..32768 for both E.Only shapes at least 3% faster than the default were written (
--compare --update_improved). Every other shape was within 3% of the default, except M=1, 8, 16, 32, where the best candidate was 12-22% slower.op_tests/tuning_tests/test_config_shape_collision.pypasses with the new file.Performance
MI325X, aiter 0.1.24.post1 (vLLM v0.32.0 ROCm image). Times are from the tuner's
--comparereport, measured on the fused_moe path, in us. The CSVuscolumn is kernel time from the sweep, so it does not match this table.End-to-end (vLLM 0.32.0, TP1, MTP 3, 2 runs): 8x32K cold prefill +0.8% / +0.7%. Decode at c16 and c64 (64 and 256 tokens/step) hits none of the new rows. c1 (4 tokens/step) hits the M=4 row: TPOT 4.0 ms without, 4.2 / 4.0 ms with. GSM8K-200: 0.98 without, 0.965 / 0.97 with (SE ~1.2 pt).
Reproduce
TUNE_ONLY=is needed until #6692 lands. Some gfx942 asm FLAT candidates fault; the tuner restarts its pool (#6013).