Accelerate nd-parallel #3006
Merged
Accelerate nd-parallel #3006
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
accelerate-nd-parallel.md
Outdated
| # Accelerate ND-Parallel: A guide to Efficient Multi-GPU Training | ||
|
|
||
| Training large models on multiple GPUs can be challenging due to the complexities of different parallelism strategies. | ||
| In Accelerate, together with [Axolotl](https://huggingface.co/axolotl-ai-co), we have integrated a quick and easy way |
Contributor
There was a problem hiding this comment.
Link to axolotl's github instead?
accelerate-nd-parallel.md
Outdated
| model = accelerator.prepare(model) | ||
| ``` | ||
|
|
||
| This feature is also integrated into [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), allowing you to compose |
Contributor
There was a problem hiding this comment.
Link to axolotl's ND-parallel docs?
SunMarc
approved these changes
Aug 6, 2025
Member
SunMarc
left a comment
SunMarc
left a comment
There was a problem hiding this comment.
Nice technical blog. Can we plug into some figures so that it is easier to understand ? Also it might be nice for users who wants to learn more to redirect them to https://huggingface.co/spaces/nanotron/ultrascale-playbook?section=first_steps:_training_on_one_gpu.
Lastly, for each strategy you explained, can you also add a cmd to launch the respective training in nd_parallel file ?
accelerate-nd-parallel.md
Outdated
| tensor_parallel_size: 2 | ||
| ``` | ||
|
|
||
| To get up and running quickly, you can check the examples in the [accelerate repository](https://github.com/huggingface/accelerate/blob/main/examples/fsdp2/nd_parallel.py) or their counterpart in [Axolotl](TODO) |
Contributor
There was a problem hiding this comment.
Let's add the examples from axolotl-ai-cloud/axolotl#3019 and an axolotl train ... command here once the linked PR is ready to go
accelerate-nd-parallel.md
Outdated
|
|
||
| Whilst we can make further memory-compute tradeoffs and offload model parameters and gradients to the CPU to train larger models, this can be prohibitively slow. Instead, let’s consider how we can effectively utilise even more devices to train larger models whilst maintaining high data throughput. | ||
|
|
||
| We use the term node to refer to a single machine which hosts multiple GPUs (often 8), with fast intra-node communication channels using e.g. NVLink between devices. When utilising multiple nodes for training, we rely on relatively slower inter-node communication channels between machines using e.g. Infiniband. We also refer to the total number of devices in the process pool as the world size - e.g. a single node with 8 GPUs represents a world size of 8, and 4 nodes would represent a world size of 32. |
Member
There was a problem hiding this comment.
Suggested change
| We use the term node to refer to a single machine which hosts multiple GPUs (often 8), with fast intra-node communication channels using e.g. NVLink between devices. When utilising multiple nodes for training, we rely on relatively slower inter-node communication channels between machines using e.g. Infiniband. We also refer to the total number of devices in the process pool as the world size - e.g. a single node with 8 GPUs represents a world size of 8, and 4 nodes would represent a world size of 32. | |
| We use the term node to refer to a single machine which hosts multiple GPUs (often 8), with fast intra-node communication channels using e.g. NVLink between devices. When using multiple nodes for training, we rely on relatively slower inter-node communication channels between machines using e.g. Infiniband. We also refer to the total number of devices in the process pool as the world size - e.g. a single node with 8 GPUs represents a world size of 8, and 4 nodes would represent a world size of 32. |
SunMarc
reviewed
Aug 6, 2025
accelerate-nd-parallel.md
Outdated
Comment on lines
135
to
139
| ## Some notes: | ||
|
|
||
| While we may talk about the remaining parallelism combinations, we feel like it's going to have very diminishing returns. You can combine any of the above parallelism strategies, in any way. We encourage you to experiment with | ||
| this, gain some intuition, because the future is distributed! | ||
|
|
Member
There was a problem hiding this comment.
maybe add a final section for coming items that are being worked on, e,.g trainer and trl integration
SunMarc
reviewed
Aug 7, 2025
accelerate-nd-parallel.md
Outdated
Comment on lines
39
to
40
| model = AutoModelForCausalLM.from_pretrained("your-model-name", tp_size=pc.tp_size, device_mesh=accelerator.torch_device_mesh) | ||
| model = accelerator.prepare(model) |
Member
There was a problem hiding this comment.
we don't need to specify the tp size normally as this information is in the device_mesh
Suggested change
| model = AutoModelForCausalLM.from_pretrained("your-model-name", tp_size=pc.tp_size, device_mesh=accelerator.torch_device_mesh) | |
| model = accelerator.prepare(model) | |
| model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh) | |
| model = accelerator.prepare(model) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.