Pinned
Muon makes the first MoE layer harder to balance.
DeepSeek trains the router with Muon. Qwen keeps it on AdamW because Muon made it unstable.
We found the router isn't the problem. Muon on the backbone is.
Right after warmup, the busiest expert in the first MoE layer gets 4-7x its fair share. Under AdamW it gets 1.1x.
arxiv.org/abs/2610.37914
