Adding FastSpeech2#11135
Conversation
|
|
||
| class FastSpeech2Config(PretrainedConfig): | ||
| model_type = "fastspeech2" | ||
|
|
There was a problem hiding this comment.
We would need some docs and an example here I think :-)
| def __init__(self, | ||
| # Audio and mel | ||
| ### for LJSpeech ### | ||
| sampling_rate = 22050, |
There was a problem hiding this comment.
Could we remove the space around = here to comply with our style in other config files? :-)
| return last_hidden | ||
|
|
||
| class G2PDecoder(nn.Module): | ||
| global g2idx, idx2g, p2idx, idx2p |
There was a problem hiding this comment.
Can we remove those global variables? I think they are not used no?
| # graphenes. Tie the embeddings to the fastspeech2 embeddings to | ||
| # increase generalizations. | ||
|
|
||
| class G2PEncoder(nn.Module): |
There was a problem hiding this comment.
I'm a bit surprised to find the nn.Module here.
I think FastSpeech2 does not contain a nn.GRU layer no?
Neither the paper, nor the other implementation mention anything of a gated recurrent unit. Could you elaborate a bit here? :-)
There was a problem hiding this comment.
Ah sorry, just read your PR description. I think it will be difficult to merge user-specific code to src/transformers for us -> do you think there could be a possibility to merge the official FastSpeech2 implementation and then maybe just add a script under examples/research_projects/fast_speech_2 that includes costum modules?
There was a problem hiding this comment.
No worries @patrickvonplaten. I'm happy to help you guys merge the officeial FastSpeech2 if someone can take a crack at it first, I can contribute :)
|
Hey @ontocord, Thanks a lot for opening this pull request :-) We are very much open to adding FastSpeech2 to IMO, the easiest way to approach this would be to translate [this](TF code) to PyTorch: https://github.com/TensorSpeech/TensorFlowTTS/blob/master/tensorflow_tts/models/fastspeech2.py and add it to our lib. Let me know if it would also be ok/interesting for you to stay closer to the official code -> I'm very happy to help you get this PR merged then :-) |
|
Happy for you to close this PR in favor of an official implementation. Just as an aside, the G2P (GRU->GRU) code is actually based on the original impementation from the Fastspeech2 paper. But it uses https://github.com/Kyubyong/g2p which is slower than pytorh and based on Numpy. I re-wrote the G2P in pytorch based on the G2P author's notes, and retrained it so it's faster. From the paper: "To alleviate the mispronunciation problem, we convert the I think this module is really one of the things that keeps the Fastspeech2 model (and tacotron 2 and similar models) from generalizing to more languages. In theory you could just train on character level, but it's harder. DM if you want to discuss work arounds... |
|
Hey @ontocord @patrickvonplaten, I was wondering if there has been a followup to this PR. I'd love to see transformer TTS models like FastSpeech2 in this library and would be more than happy to help contribute if possible! |
|
I also think we should eventually add models like FastSpeech2 to the library. Gently ping to @anton-l here who was interested in this addition as well. |
|
@patrickvonplaten @anton-l Do we only add models with official weights from the paper authors? AFAIK FastSpeech2 has plenty of unofficial implementations with weights, but there is no official repository (PwC). I think we should reach out to the author (Yi Ren is on GitHub), and if that doesn't work out, consider which implementation/weights we want to port. What do you think? Also if you'd prefer, I'll open a new issue dedicated to this discussion instead of hijacking this PR. |
|
I think we should definitely reach out to the original authors! Feel free to contact them :-) |
|
Just emailed the first author and cc'd both you and Anton! I'll keep you posted. |
|
I would be interested in working on something more generic than fastspeech2 which needs a g2p module. It’s not truly end to end.
… On Jan 3, 2022, at 7:55 AM, Jake Tae ***@***.***> wrote:
Just emailed the first author and cc'd both you and Anton! I'll keep you posted.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.
|
4 participants
What does this PR do?
This is a draft PR for Fastspeech2 which includes melgan and a custom g2p pythorch module. See https://huggingface.co/ontocord/fastspeech2-en
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@patil-suraj
Models: