Repository navigation
Dynamic FBcache parameter with nunchaku - #104
Conversation
|
I have a question: in the dependencies part, you wrote that it is necessary to ensure this PR is merged. However, I see that this PR has not yet been merged into the nunchaku repository. Will this have any impact? |
Yes. this pr requires the code. That PR has not been merged yet because it still has a small amount of memory allocated to the workflow. It does not affect the functionality, but we are working on a fix. |
|
@ToTheBeginning |
|
Could you please merge this pr? |
|
@Bluear7878 Thanks for your contribution, merged now. (Sorry for the late reply, I was busy on other projects recent days. |
Closes #103
This PR introduces support for the dynamic modification of
FBCacheandDoubleFBCachewhen utilizing the nunchaku feature. This allows for on-the-fly adjustments of cache thresholds, enabling a dynamic trade-off between inference speed and output quality without needing to restart the application.🚨 Important Dependency
For this implementation to function correctly, the updates from PR #452 must be applied first. Please ensure that the changes from nunchux-ai/nunchaku#452 are merged before this PR.
Performance was measured on an NVIDIA A5000 GPU.
Additionally, it's worth noting that the efficiency of both
FBCacheandDoubleFBCacheincreases with larger step sizes.FBCache Performance
For
FBCache, I evaluated performance based on bothresidual_diff_thresholdandresidual_diff_threshold_single.Inference Time (seconds)
The table below summarizes the inference time for different threshold combinations.