Why are you proposing to do that?

If you need seq_dp_world_size on resume and for some reason it's not there then probably store it additionally while leaving dp_world_size alone, no?

Why are you proposing to do that?

If you need seq_dp_world_size on resume and for some reason it's not there then probably store it additionally while leaving dp_world_size alone, no?

cause no moe save checkpoint use this

Thank you for explaining, then yes, it checks out.

@sfc-gh-truwase, I know you weren't involved in DS-MoE - but shouldn't _save_checkpoint and _save_moe_checkpoint follow the exact same recipe wrt saving state_dict (the

only model_state_dict is what's different - perhaps abstracting this last bit into a helper util and having both functions use it, with the only difference passing the model_state_dict that differs? the moe saving one also saves num_experts but it's not using it on load. I think it's diverging in other aspects since nobody bothered to keep in in sync.