Feature request
Microsoft has introduced their microsoft/LongRoPE implementation. Unlike plug-and-play solutions, LongRoPE requires hyperparameter tuning via a genetic algorithm. This implementation is likely the same as described in the Su on Phi-3. Are there any plans to incorporate LongRoPE into LLaMA?
Motivation
In my research on long content, I have managed to integrate LongRoPE into LLaMA with some minor code adjustments.
I am curious if Huggingface is also working on integrating this feature.
Your contribution
If necessary.
Feature request
Microsoft has introduced their microsoft/LongRoPE implementation. Unlike plug-and-play solutions, LongRoPE requires hyperparameter tuning via a genetic algorithm. This implementation is likely the same as described in the
Suon Phi-3. Are there any plans to incorporate LongRoPE into LLaMA?Motivation
In my research on long content, I have managed to integrate LongRoPE into LLaMA with some minor code adjustments.
I am curious if Huggingface is also working on integrating this feature.
Your contribution
If necessary.