Repository navigation
feat: add use_pinned_memory option for block swap in multiple models - #700
Conversation
|
@kohya-ss thank you i just tested and uses more RAM and shared VRAM but sadly same speed you can also add temporary debug to print timings if you want i can provide output after testing |
…Offloader, improve event handling
|
Speed has been improved when using pinned memory. Debug output has also been added. Debug output can be enabled by setting the environment variable MUSUBI_TUNER_OFFLOADER_DEBUG to 1. In my environment, block swap completes in about 170ms when using the staging buffer, and in about 100ms when using pinned memory for Qwen-Image with bf16. |
|
@kohya-ss thanks i tested - 35 block swap dreambooth deleted older message to make it clear here so here i tested 10 steps without pinned memory and with pinned memory full logs sadly pinned memory avg speed is even slower i hope this debug logs can be helpful the top ones are without pinned memory starting from line 3902 is with pinned memory also here as a gist : https://gist.github.com/FurkanGozukara/be71c679157c05e8788f9cd5ac0913ab |
|
ok i just noticed that INFO:musubi_tuner.hv_train_network:enable swap 35 blocks to CPU from device: cuda, use pinned memory: True this message only appears during lora not fine tuning currently not implemented for fine tuning? i got to sleep now so i can test tomorrow more hopefully |
|
@kohya-ss sorry for delay i just made a more proper test LoRA training, 128 rank, 1328, 40 block swap, FP8 scaled is not enabled, 10 steps with pinned memory : https://gist.github.com/FurkanGozukara/f47dc4d3da86679415b1437454f839c5 10_steps_with_pinned_memory.txt without pinned memory older way : https://gist.github.com/FurkanGozukara/fbfbb1895541c5f35cf6c22b6f20827b 10_steps_without_pinned_memory.txt without pinned memory it is faster somehow the analysis of Gemini is belowExcellent question. Based on a detailed analysis of the two log files, the conclusion is clear but counter-intuitive. The VerdictThe training run WITHOUT pinned memory was faster. Here is the key evidence from the end of both logs after 10 steps:
The training process averaged 15.91 seconds per iteration without pinned memory, compared to 17.39 seconds per iteration with it. This means the run without pinned memory was approximately 9.3% faster. Why Was It Faster? (The Technical Explanation)This result seems backward at first glance. Pinned (or page-locked) memory is supposed to speed up data transfers between the CPU and GPU. So why was it slower here? The answer lies not in the raw speed of the memory itself, but in how the data transfers were implemented and synchronized in the two different modes. The detailed timing logs for the "Weight swap timing" reveal the story. 1. Different Transfer MechanismsThe logs show two distinct methods for swapping model blocks between the GPU (CUDA) and CPU.
2. Synchronous vs. Asynchronous OperationsThe key difference is in the Let's compare the "Weight swap timing" for a forward pass (e.g., at step
3. Interpretation and ConclusionThe The In summary: The performance difference is not about the physical speed of the memory but about the software's strategy. The asynchronous, pipelined approach used for standard memory transfers was more efficient at hiding latency than the synchronous approach used for pinned memory transfers in this specific training script. The overhead of waiting for synchronous copies to finish was far greater than the overhead of making an extra memory copy in an asynchronous pipeline. |
|
Thank you for the detailed report. When tested with RTX PRO 6000 TCC mode driver, the fluctuations were still there but much less noticeable, and using pinned memory is significantly faster. Typical values without pinned memory: with pinned memory: Therefore, I think this is probably due to the WDDM mode driver. I think you can reduce the fluctuations in transfer speed, the transfer speed itself, and the time it takes to sync by not using other apps while training, turning off Hardware-Accelerated GPU Scheduling, etc. |
|
@kohya-ss thank you so much it was as you said i turned of hardware acceleration restarted pc and in 10 steps it is now like 50% faster :D 17.39s to 11.73s/it, in 10 steps |
That's great😀 |
thank you so much are there any other optimization we can make other than this? "turning off Hardware-Accelerated GPU Scheduling," |
|
also can you merge it if possible? @kohya-ss ty so much |
|
It appears you have two GPUs, but both the screen display and training appear to be running on GPU 0. Connecting a display to GPU 1 may be slightly faster. Gen 4 x8 has a bandwidth of approximately 15GB/s. Since 16/60 blocks are transferred in forward and backward pass, and also in both CPU->GPU and GPU->CPU directions, if the model weights are 20GB (fp8 scaled), a transfer of 20 * (16/60) * 2 * 2, approximately 20GB is required. Transferring 20GB will take about 2 seconds in Gen 4 x8, including overhead. Since one step takes 7-8 seconds, if the transfer is completely asynchronous, Gen 4 x8 should be sufficient. |
|
@kohya-ss i just tested this moved display cables to second gpu restarted pc did 40 steps without pinned memory speed is 6.76 second / it with pinned memory it was 6.47 second / it on linux I am getting like 4 second / it :( on runpod so indeed it brought some speed increase but this linux speed, the speed on runpod is so extremely more compared to windows what we get
this one below from linux - fp8 scaled is 4 second / it probably i can get some more faster with lesser block swap when gpu is empty lets see how faster i can push but 4 second / it seems impossible
|
Awesome! |
The option is added to Wan/HunyuanVideo/FramePack/Flux.1 Kontext too. However, unfortunately, for long videos and high-resolution videos, the calculation time is longer than the transfer time, so I think the improvement is limited. |
|
In my environment, without fp8 scaled and 40 block swap is 4sec/it, so the WDDM mode driver is likely to be the cause. To improve GPU performance in a Windows environment, it is generally a good idea to check the following:
|
|
Speed improvement of about 5% when not using pin memory by 797c38a. |
|
I tried to turn off WDDM but sadly shameless nvidia blocked it for even RTX 5090 so only way is using native Linux i presume? yes Linux speed is 4 second / it for rtx 5090 i tested on runpod I will check Resizable BAR (ReBAR) in BIOS settings first time hearing |
|
I enabled ReBAR but it didnt make big difference I was expecting huge. By the way turning off hardware accelerated GPU scheduling indeed makes big difference for pinned memory Maybe we can get some ideas from here : Orbmu2k/nvidiaProfileInspector#357
|
…se two streams for transfer
|
The latest implementation reuses pinned memory and use two CUDA streams (GPU->CPU and CPU->GPU). The latest implementation is faster on the RTX A6000, but the previous implementation seems to be faster on the RTX 6000 Blackwell... |
|
One stream seems to be faster than two streams. |
|
On RTX 6000 Blackwell Max-Q (Windows, TCC mode), training of Qwen-Image LoRA (128 rank, 1328x1328, batch size 1, fp8 scaled) can be achieved with almost no performance drop using both using pinned memory with block swap 40, and non-pinned with block swap 10. Power consumption also remains consistently above 290W. Edit: On RTX A6000 (Windows, WDDM mode), non-pinned is faster than pinned. |
There was a problem hiding this comment.
Pull Request Overview
This PR adds a new use_pinned_memory parameter to the block swapping functionality to provide users with an option to optimize data transfer performance between CPU and GPU during block swapping operations.
- Added
use_pinned_memoryparameter toenable_block_swap()methods across all model classes - Implemented two distinct swapping strategies in the offloading utility based on the flag
- Added command-line argument
--use_pinned_memory_for_block_swapto training and generation scripts
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/musubi_tuner/modules/custom_offloading_utils.py | Core implementation of pinned memory block swapping strategy and refactored CUDA offloading logic |
| src/musubi_tuner/wan/modules/model.py | Added use_pinned_memory parameter to WAN model's enable_block_swap() method |
| src/musubi_tuner/qwen_image/qwen_image_model.py | Added use_pinned_memory parameter to Qwen image model's enable_block_swap() method |
| src/musubi_tuner/hunyuan_model/models.py | Added use_pinned_memory parameter to Hunyuan model's enable_block_swap() method |
| src/musubi_tuner/flux/flux_models.py | Added use_pinned_memory parameter to FLUX model's enable_block_swap() method |
| src/musubi_tuner/frame_pack/hunyuan_video_packed.py | Added use_pinned_memory parameter to packed Hunyuan video model's enable_block_swap() method |
| src/musubi_tuner/qwen_image_train.py | Updated to pass use_pinned_memory argument from command-line args |
| src/musubi_tuner/qwen_image_generate_image.py | Added command-line argument and updated function call |
| src/musubi_tuner/hv_train_network.py | Added command-line argument and updated function call with logging |
| docs/hunyuan_video.md | Added documentation for the new --use_pinned_memory_for_block_swap option |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <[email protected]>
is the key. shameless NVIDIA blocked it for RTX 5090 :( thank you so much for trying more I will test latest config on Windows sadly in WDDM mode |
…ansformer3DModelPacked
…b.com/kohya-ss/musubi-tuner into feat-use-pinned-memory-for-block-swap
|
looks like enabling TCC mode is not that hard on Windows i just requires a driver hack i wonder if we could simplify that |
|
LoRA training for Qwen-Image, FramePack, Wan, FLUX.1 Kontext and HunyuanVideo has been tested. |
|
@kohya-ss can we add this for fine tuning as well please ty |
|
comparison on 7900xtx. seems faster, but total time is slightly longer . pinned memory disabled: https://gist.github.com/xzuyn/bcc2ba07673923cc3be047f203ed8bd8 |
|
@xzuyn thanks for test. so If AMD was selling GPUs from 96 GB VRAM instead of same as NVIDIA they would make sense to purchase I can't understand how incompetent they are :( |
…in offloader helper source: kohya-ss/musubi-tuner#700







Add
--use_pinned_memory_for_block_swapfor each training script to enable pinned memory. Will work with Windows and Linux, but tested with Windows only.Qwen-Image fine tuning is tested.