Hi @PKUWZP , I want to confirm this change with you. I saw comments saying put the momentum buffer on CPU to save device memory. So I guess the intention is to allow train larger model with Muon optimizer. But put momentum buffer on CPU also makes Muon optimizer run slower. Maybe allow Muon optimizer in ZeRO offload should work for large model.

Hi @PKUWZP , do you have comments for this PR? Thanks!

@delock Do you have any benchmarking results?

@delock Do you have any benchmarking results?

I tested with finetune Qwen2.5-3B on 2xA100 cards. global batch size is 8.

On master branch the finetune iteration time is 1430ms. With this PR the finetune iteration time is 918ms.

Profiling data shows before this change, a lot of time spent on H2D and D2H copy. After this change, there is no H2D and D2H copy in top profiled items.

@delock Do you have any benchmarking results?

I tested with finetune Qwen2.5-3B on 2xA100 cards. global batch size is 8.

On master branch the finetune iteration time is 1430ms. With this PR the finetune iteration time is 918ms.

Profiling data shows before this change, a lot of time spent on H2D and D2H copy. After this change, there is no H2D and D2H copy in top profiled items.

Hi @PKUWZP , any other comments? I'll merge this PR if you can approve it. Thanks!