👷 Added Kernels on the Hub x TRL guide #3969
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
docs/source/_toctree.yml
Outdated
| title: Command Line Interface (CLI) | ||
| - local: jobs_training | ||
| title: Training using Jobs | ||
| - local: kernels_hub |
There was a problem hiding this comment.
I'd move this section under "Integration"
- local: deepspeed_integration
title: DeepSpeed
local: kernel_hub_integration
title: Kernel Hub
- local: liger_kernel_integration
title: Liger Kernel
There was a problem hiding this comment.
Updated!
I considered Integrations to be the section for external toolkits so I was unsure about adding it there.
|
|
||
| [PLOT] | ||
|
|
||
| ## Combining FlashAttention Kernels with Liger Kernels |
There was a problem hiding this comment.
do we pull the kernel from the hub here as well? - if yes, then let's mention it.
There was a problem hiding this comment.
Not pulled from the Hub at the moment
| </Tip> | ||
|
|
||
|
|
||
| ## Comparing Attention Implementations |
There was a problem hiding this comment.
not sure if this is really needed tbh (in the spirit to ship fast)
Co-authored-by: vb <[email protected]>
Co-authored-by: vb <[email protected]>
Co-authored-by: vb <[email protected]>
Co-authored-by: vb <[email protected]>
|
Should I add the script used for benchmarking somewhere? It's a modification of sft.py with a callback (gist) |
lewtun
left a comment
lewtun
left a comment
There was a problem hiding this comment.
LGTM with a turbo nit :) Great stuff @sergiopaniego !
|
|
||
| Building Flash Attention from source can be time-consuming, often taking anywhere from several minutes to hours, depending on your hardware, CUDA/PyTorch configuration, and whether precompiled wheels are available. | ||
|
|
||
| In contrast, **Hugging Face Kernels** provide a much faster and more reliable workflow. Developers don’t need to worry about complex setups—everything is handled automatically. In our benchmarks, kernels were ready to use in about **2.5 seconds**, with no compilation required. This allows you to start training almost instantly, significantly accelerating development. Simply specify the desired version, and `kernels` takes care of the rest. |
There was a problem hiding this comment.
This is such a cool feature of kernels! I've lost a cumulative few days of my life waiting for FA2 to compile :D
There was a problem hiding this comment.
I know the feeling 😭
Co-authored-by: lewtun <[email protected]>
Co-authored-by: vb <[email protected]> Co-authored-by: Kashif Rasul <[email protected]> Co-authored-by: lewtun <[email protected]>
What does this PR do?
Before submitting
Pull Request section?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.