Skip to content

feat: add use_pinned_memory option for block swap in multiple models - #700

Merged
kohya-ss merged 11 commits into
mainfrom
feat-use-pinned-memory-for-block-swap
Nov 2, 2025
Merged

kohya-ss merged 11 commits into
mainfrom
feat-use-pinned-memory-for-block-swap

Conversation

@kohya-ss

@kohya-ss kohya-ss commented Oct 30, 2025 •

Copy link
Copy Markdown
Owner

Add --use_pinned_memory_for_block_swap for each training script to enable pinned memory. Will work with Windows and Linux, but tested with Windows only.

Qwen-Image fine tuning is tested.

@FurkanGozukara

FurkanGozukara commented Oct 30, 2025 •

Copy link
Copy Markdown
Contributor

@kohya-ss thank you i just tested and uses more RAM and shared VRAM but sadly same speed

you can also add temporary debug to print timings if you want i can provide output after testing

@kohya-ss

kohya-ss commented Oct 30, 2025 •

Copy link
Copy Markdown
Owner Author

Speed ​​has been improved when using pinned memory.

Debug output has also been added. Debug output can be enabled by setting the environment variable MUSUBI_TUNER_OFFLOADER_DEBUG to 1.

In my environment, block swap completes in about 170ms when using the staging buffer, and in about 100ms when using pinned memory for Qwen-Image with bf16.

@FurkanGozukara

Copy link
Copy Markdown
Contributor

@kohya-ss thanks i tested - 35 block swap dreambooth

deleted older message to make it clear here

so here i tested 10 steps without pinned memory

and with pinned memory

full logs

sadly pinned memory avg speed is even slower

i hope this debug logs can be helpful

the top ones are without pinned memory

starting from line 3902 is with pinned memory

debug_logs.txt

also here as a gist : https://gist.github.com/FurkanGozukara/be71c679157c05e8788f9cd5ac0913ab

@FurkanGozukara

Copy link
Copy Markdown
Contributor

ok i just noticed that

INFO:musubi_tuner.hv_train_network:enable swap 35 blocks to CPU from device: cuda, use pinned memory: True

this message only appears during lora not fine tuning

currently not implemented for fine tuning? i got to sleep now so i can test tomorrow more hopefully

@FurkanGozukara

FurkanGozukara commented Oct 31, 2025 •

Copy link
Copy Markdown
Contributor

@kohya-ss sorry for delay i just made a more proper test

LoRA training, 128 rank, 1328, 40 block swap, FP8 scaled is not enabled, 10 steps

with pinned memory : https://gist.github.com/FurkanGozukara/f47dc4d3da86679415b1437454f839c5

10_steps_with_pinned_memory.txt

without pinned memory older way : https://gist.github.com/FurkanGozukara/fbfbb1895541c5f35cf6c22b6f20827b

10_steps_without_pinned_memory.txt

without pinned memory it is faster somehow

the analysis of Gemini is below

Excellent question. Based on a detailed analysis of the two log files, the conclusion is clear but counter-intuitive.

The Verdict

The training run WITHOUT pinned memory was faster.

Here is the key evidence from the end of both logs after 10 steps:

  • With Pinned Memory: 10/600 [02:53<2:50:59, **17.39s/it**, avr_loss=0.0977]
  • Without Pinned Memory: 10/600 [02:39<2:36:27, **15.91s/it**, avr_loss=0.0978]

The training process averaged 15.91 seconds per iteration without pinned memory, compared to 17.39 seconds per iteration with it. This means the run without pinned memory was approximately 9.3% faster.


Why Was It Faster? (The Technical Explanation)

This result seems backward at first glance. Pinned (or page-locked) memory is supposed to speed up data transfers between the CPU and GPU. So why was it slower here?

The answer lies not in the raw speed of the memory itself, but in how the data transfers were implemented and synchronized in the two different modes. The detailed timing logs for the "Weight swap timing" reveal the story.

1. Different Transfer Mechanisms

The logs show two distinct methods for swapping model blocks between the GPU (CUDA) and CPU.

  • With Pinned Memory: The transfer is direct.

    • cuda to cpu: Data moves directly from GPU memory to pinned CPU memory.
    • cpu to cuda: Data moves directly from pinned CPU memory to GPU memory.
    • This is followed by a very long synchronize step.
  • Without Pinned Memory: The transfer uses an intermediate "staging" buffer. This is a classic method to handle transfers from regular (pageable) CPU memory.

    • cuda to staging A -> staging A to CPU: A two-step process to get data from GPU to CPU.
    • cpu to staging B -> staging B to CUDA: A two-step process to get data from CPU to GPU.
    • This allows for asynchronous operations.

2. Synchronous vs. Asynchronous Operations

The key difference is in the synchronize time. This step represents the time the program waits for all the initiated copy operations to fully complete before proceeding.

Let's compare the "Weight swap timing" for a forward pass (e.g., at step 80) and a backward pass (e.g., at step 120).

Operation With Pinned Memory Without Pinned Memory Analysis
Forward Pass Swap (at 80) Overall: 124.95 ms Overall: 144.15 ms The individual forward pass swaps are slightly faster with pinned memory.
cuda to cpu / staging A to CPU 0.42 ms 37.82 ms Direct copy is much faster than the staged copy.
cpu to cuda / staging B to CUDA 0.20 ms 0.31 ms Direct copy is slightly faster.
synchronize 115.93 ms 29.39 ms This is the critical difference.
Backward Pass Swap (at 120) Overall: 187.54 ms Overall: 274.39 ms The individual backward pass swaps are also faster with pinned memory.
cuda to cpu / staging A to CPU 0.45 ms 40.21 ms Direct copy is much faster.
synchronize 115.60 ms 93.45 ms The synchronization wait is still significantly longer with pinned memory.

3. Interpretation and Conclusion

The without pinned memory implementation, despite using a slower two-step copy process, is structured to be more asynchronous. It can initiate a data transfer and then immediately continue with other work (like CPU-side computations) before it has to wait for the transfer to finish. This "hides" the data transfer latency by overlapping it with useful work. The final synchronize time is much shorter because much of the waiting has already been absorbed during other operations.

The with pinned memory implementation, although it uses a faster direct copy mechanism, appears to be implemented in a more synchronous manner. The code initiates a copy and then immediately enters a long wait (synchronize) for it to complete, creating a bottleneck. The CPU/GPU is stalled for over 115ms just waiting, which nullifies the benefit of the faster direct copy.

In summary:

The performance difference is not about the physical speed of the memory but about the software's strategy. The asynchronous, pipelined approach used for standard memory transfers was more efficient at hiding latency than the synchronous approach used for pinned memory transfers in this specific training script. The overhead of waiting for synchronous copies to finish was far greater than the overhead of making an extra memory copy in an asynchronous pipeline.

@kohya-ss

Copy link
Copy Markdown
Owner Author

Thank you for the detailed report.
Looking at the logs you shared, we can see that the time taken for each block swap is very fluctuated. This may be due to the WDDM mode of the Windows driver.

When tested with RTX PRO 6000 TCC mode driver, the fluctuations were still there but much less noticeable, and using pinned memory is significantly faster.

Typical values without pinned memory:

[qwen-image-block] Weight swap timing at 100:
  find modules: 0.18ms
  synchronize before swap: 0.04ms
  cuda to staging A: 0.36ms
  cpu to staging B: 73.24ms
  wait staging A: 0.09ms
  staging B to CUDA: 0.27ms
  staging A to CPU: 72.32ms
  synchronize: 0.05ms
Overall time: 147.69ms, total time in sections: 146.56ms

with pinned memory:

[qwen-image-block] Weight swap timing at 100:
  find modules: 0.15ms
  synchronize before swap: 0.03ms
  cuda to cpu: 1.68ms
  wait cpu: 0.05ms
  cpu to cuda: 0.30ms
  synchronize: 32.17ms
Overall time: 35.19ms, total time in sections: 34.37ms

Therefore, I think this is probably due to the WDDM mode driver.

I think you can reduce the fluctuations in transfer speed, the transfer speed itself, and the time it takes to sync by not using other apps while training, turning off Hardware-Accelerated GPU Scheduling, etc.

@FurkanGozukara

Copy link
Copy Markdown
Contributor

@kohya-ss thank you so much it was as you said

i turned of hardware acceleration restarted pc and in 10 steps it is now like 50% faster :D

17.39s to 11.73s/it, in 10 steps

new_logs.txt

@kohya-ss

Copy link
Copy Markdown
Owner Author

i turned of hardware acceleration restarted pc and in 10 steps it is now like 50% faster :D

That's great😀

@FurkanGozukara

Copy link
Copy Markdown
Contributor

turning off Hardware-Accelerated GPU Scheduling,

thank you so much

are there any other optimization we can make other than this? "turning off Hardware-Accelerated GPU Scheduling,"

@FurkanGozukara

Copy link
Copy Markdown
Contributor

also can you merge it if possible? @kohya-ss ty so much

@FurkanGozukara

FurkanGozukara commented Oct 31, 2025 •

Copy link
Copy Markdown
Contributor

Ok I guess i talked too early

i am doing a longer test with FP8 Scaled and sadly we are still way way slower than Linux - I tested RTX 5090 on linux

what is the real culprit i really wonder

image

my BUS interface is 8x

maybe that is another culprit i need to check this :D

image

when i train my second gpu rtx 3090 ti, its speed like almost half of rtx 5090 and it has good amount of watt usage

so bus interface is not the culprit - this one uses 4x only atm

image

@kohya-ss

Copy link
Copy Markdown
Owner Author

It appears you have two GPUs, but both the screen display and training appear to be running on GPU 0. Connecting a display to GPU 1 may be slightly faster.

Gen 4 x8 has a bandwidth of approximately 15GB/s. Since 16/60 blocks are transferred in forward and backward pass, and also in both CPU->GPU and GPU->CPU directions, if the model weights are 20GB (fp8 scaled), a transfer of 20 * (16/60) * 2 * 2, approximately 20GB is required. Transferring 20GB will take about 2 seconds in Gen 4 x8, including overhead. Since one step takes 7-8 seconds, if the transfer is completely asynchronous, Gen 4 x8 should be sufficient.

@FurkanGozukara

Copy link
Copy Markdown
Contributor

@kohya-ss i just tested this

moved display cables to second gpu

restarted pc

did 40 steps without pinned memory

speed is 6.76 second / it

with pinned memory it was 6.47 second / it

on linux I am getting like 4 second / it :( on runpod

so indeed it brought some speed increase but this linux speed, the speed on runpod is so extremely more compared to windows what we get

image

this one below from linux - fp8 scaled is 4 second / it

probably i can get some more faster with lesser block swap when gpu is empty lets see how faster i can push but 4 second / it seems impossible

image

@FurkanGozukara

Copy link
Copy Markdown
Contributor

ok with only 7 block swap i am able to push up to 6.10 second / it

with overlocking it gets even better since there is still space but can't reach linux :(

image

@JonkoXL

JonkoXL commented Oct 31, 2025

Copy link
Copy Markdown

Add --use_pinned_memory_for_block_swap for each training script to enable pinned memory. Will work with Windows and Linux, but tested with Windows only.

Qwen-Image fine tuning is tested.

Awesome!
Will you add it for Wan as well? May be beneficial for block swapping with longer / higher res video training.

@kohya-ss

kohya-ss commented Nov 1, 2025

Copy link
Copy Markdown
Owner Author

Awesome!
Will you add it for Wan as well? May be beneficial for block swapping with longer / higher res video training.

The option is added to Wan/HunyuanVideo/FramePack/Flux.1 Kontext too. However, unfortunately, for long videos and high-resolution videos, the calculation time is longer than the transfer time, so I think the improvement is limited.

@kohya-ss

kohya-ss commented Nov 1, 2025 •

Copy link
Copy Markdown
Owner Author

In my environment, without fp8 scaled and 40 block swap is 4sec/it, so the WDDM mode driver is likely to be the cause.

To improve GPU performance in a Windows environment, it is generally a good idea to check the following:

@kohya-ss

kohya-ss commented Nov 1, 2025

Copy link
Copy Markdown
Owner Author

Speed ​​improvement of about 5% when not using pin memory by 797c38a.

@FurkanGozukara

FurkanGozukara commented Nov 1, 2025 •

Copy link
Copy Markdown
Contributor

I tried to turn off WDDM but sadly shameless nvidia blocked it for even RTX 5090

so only way is using native Linux i presume? yes Linux speed is 4 second / it for rtx 5090 i tested on runpod

I will check Resizable BAR (ReBAR) in BIOS settings first time hearing

@FurkanGozukara

FurkanGozukara commented Nov 1, 2025 •

Copy link
Copy Markdown
Contributor

I enabled ReBAR but it didnt make big difference

I was expecting huge.

By the way turning off hardware accelerated GPU scheduling indeed makes big difference for pinned memory

Maybe we can get some ideas from here : Orbmu2k/nvidiaProfileInspector#357

image

@kohya-ss

kohya-ss commented Nov 2, 2025

Copy link
Copy Markdown
Owner Author

The latest implementation reuses pinned memory and use two CUDA streams (GPU->CPU and CPU->GPU). The latest implementation is faster on the RTX A6000, but the previous implementation seems to be faster on the RTX 6000 Blackwell...

@kohya-ss

kohya-ss commented Nov 2, 2025

Copy link
Copy Markdown
Owner Author

One stream seems to be faster than two streams.

@kohya-ss

kohya-ss commented Nov 2, 2025 •

Copy link
Copy Markdown
Owner Author

On RTX 6000 Blackwell Max-Q (Windows, TCC mode), training of Qwen-Image LoRA (128 rank, 1328x1328, batch size 1, fp8 scaled) can be achieved with almost no performance drop using both using pinned memory with block swap 40, and non-pinned with block swap 10. Power consumption also remains consistently above 290W.

Edit: On RTX A6000 (Windows, WDDM mode), non-pinned is faster than pinned.

@kohya-ss
kohya-ss requested a review from Copilot November 2, 2025 07:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new use_pinned_memory parameter to the block swapping functionality to provide users with an option to optimize data transfer performance between CPU and GPU during block swapping operations.

  • Added use_pinned_memory parameter to enable_block_swap() methods across all model classes
  • Implemented two distinct swapping strategies in the offloading utility based on the flag
  • Added command-line argument --use_pinned_memory_for_block_swap to training and generation scripts

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/musubi_tuner/modules/custom_offloading_utils.py Core implementation of pinned memory block swapping strategy and refactored CUDA offloading logic
src/musubi_tuner/wan/modules/model.py Added use_pinned_memory parameter to WAN model's enable_block_swap() method
src/musubi_tuner/qwen_image/qwen_image_model.py Added use_pinned_memory parameter to Qwen image model's enable_block_swap() method
src/musubi_tuner/hunyuan_model/models.py Added use_pinned_memory parameter to Hunyuan model's enable_block_swap() method
src/musubi_tuner/flux/flux_models.py Added use_pinned_memory parameter to FLUX model's enable_block_swap() method
src/musubi_tuner/frame_pack/hunyuan_video_packed.py Added use_pinned_memory parameter to packed Hunyuan video model's enable_block_swap() method
src/musubi_tuner/qwen_image_train.py Updated to pass use_pinned_memory argument from command-line args
src/musubi_tuner/qwen_image_generate_image.py Added command-line argument and updated function call
src/musubi_tuner/hv_train_network.py Added command-line argument and updated function call with logging
docs/hunyuan_video.md Added documentation for the new --use_pinned_memory_for_block_swap option

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/hunyuan_video.md Resolved
@FurkanGozukara

FurkanGozukara commented Nov 2, 2025 •

Copy link
Copy Markdown
Contributor

TCC mode

is the key. shameless NVIDIA blocked it for RTX 5090 :(

thank you so much for trying more

I will test latest config on Windows sadly in WDDM mode

@FurkanGozukara

Copy link
Copy Markdown
Contributor

looks like enabling TCC mode is not that hard on Windows

https://www.bilibili.com/opus/891652532297793543

i just requires a driver hack

i wonder if we could simplify that

@kohya-ss

kohya-ss commented Nov 2, 2025

Copy link
Copy Markdown
Owner Author

LoRA training for Qwen-Image, FramePack, Wan, FLUX.1 Kontext and HunyuanVideo has been tested.

@kohya-ss
kohya-ss merged commit dd61304 into main Nov 2, 2025
@kohya-ss
kohya-ss deleted the feat-use-pinned-memory-for-block-swap branch November 2, 2025 12:45
@FurkanGozukara

Copy link
Copy Markdown
Contributor

@kohya-ss can we add this for fine tuning as well please

ty

@xzuyn

xzuyn commented Nov 3, 2025

Copy link
Copy Markdown
Contributor

comparison on 7900xtx. seems faster, but total time is slightly longer .

pinned memory disabled: https://gist.github.com/xzuyn/bcc2ba07673923cc3be047f203ed8bd8
pinned memory enabled: https://gist.github.com/xzuyn/9a3320f0917e75807377a5b77b9565ba

@FurkanGozukara

FurkanGozukara commented Nov 3, 2025 •

Copy link
Copy Markdown
Contributor

@xzuyn thanks for test. so If AMD was selling GPUs from 96 GB VRAM instead of same as NVIDIA they would make sense to purchase

I can't understand how incompetent they are :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants