Add Seamless M4T model by ylacombe · Pull Request #25693 · huggingface/transformers

ylacombe · 2023-08-23T17:32:03Z

What does this PR do?

Meta recently introduced Seamless M4T, a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text.

SeamlessM4T supports multiple audio and/or translation tasks, namely S2TT, S2ST, T2TT, T2ST, where the last T stands for translation.
In other words, this model seamlessly supports audio|text to translated audio|text.

SeamlessM4T weights are already available on the hub (large and medium) and the code is available on the seamless_communication git repo.

In terms of architecture, and after having discussed with @sanchit-gandhi, I've came up with 4 differents models for the 4 tasks and one model that can do each task.

I've been working on the integration for a couple of days already. At the moment, the converting script is more or less ready and the different models can generate.

Here is a TODO of what's left to be done:

Agree on the current architecture and some modeling details (for example what outputs)
integrate feature extraction (fbank) and tokenizer (similar to NLLB)
Integrate their vocoder
Write and format docstrings
Do integration tests - there's probably some modeling discrepancy at the moment (except for the speech encoder with a one-to-one correspondance)
Finish regular tests
There's probably some work to be done for the optimal generation config

cc @sanchit-gandhi and @ArthurZucker !

ylacombe

Hey @ArthurZucker, I've made some refactoring on the tokenizer, which I believe will content you 🤗

I still have to add copied from statements in the tests, but I think you can already finish your review here, if that's ok with you!
Thanks!

ylacombe · 2023-10-09T18:39:03Z

 <!--This tip is automatically generated by `make fix-copies`, do not fill manually!-->

-[BART](../model_doc/bart), [BigBird-Pegasus](../model_doc/bigbird_pegasus), [Blenderbot](../model_doc/blenderbot), [BlenderbotSmall](../model_doc/blenderbot-small), [Encoder decoder](../model_doc/encoder-decoder), [FairSeq Machine-Translation](../model_doc/fsmt), [GPTSAN-japanese](../model_doc/gptsan-japanese), [LED](../model_doc/led), [LongT5](../model_doc/longt5), [M2M100](../model_doc/m2m_100), [Marian](../model_doc/marian), [mBART](../model_doc/mbart), [MT5](../model_doc/mt5), [MVP](../model_doc/mvp), [NLLB](../model_doc/nllb), [NLLB-MOE](../model_doc/nllb-moe), [Pegasus](../model_doc/pegasus), [PEGASUS-X](../model_doc/pegasus_x), [PLBart](../model_doc/plbart), [ProphetNet](../model_doc/prophetnet), [SwitchTransformers](../model_doc/switch_transformers), [T5](../model_doc/t5), [UMT5](../model_doc/umt5), [XLM-ProphetNet](../model_doc/xlm-prophetnet)
+[BART](../model_doc/bart), [BigBird-Pegasus](../model_doc/bigbird_pegasus), [Blenderbot](../model_doc/blenderbot), [BlenderbotSmall](../model_doc/blenderbot-small), [Encoder decoder](../model_doc/encoder-decoder), [FairSeq Machine-Translation](../model_doc/fsmt), [GPTSAN-japanese](../model_doc/gptsan-japanese), [LED](../model_doc/led), [LongT5](../model_doc/longt5), [M2M100](../model_doc/m2m_100), [Marian](../model_doc/marian), [mBART](../model_doc/mbart), [MT5](../model_doc/mt5), [MVP](../model_doc/mvp), [NLLB](../model_doc/nllb), [NLLB-MOE](../model_doc/nllb-moe), [Pegasus](../model_doc/pegasus), [PEGASUS-X](../model_doc/pegasus_x), [PLBart](../model_doc/plbart), [ProphetNet](../model_doc/prophetnet), [SeamlessM4T](../model_doc/seamless_m4t), [SwitchTransformers](../model_doc/switch_transformers), [T5](../model_doc/t5), [UMT5](../model_doc/umt5), [XLM-ProphetNet](../model_doc/xlm-prophetnet)


I actually don't know why it happens, do you have any ideas?

ylacombe · 2023-10-09T18:40:26Z

+            represented by the `inputs_ids` passed when calling the Text-To-Units sub-model of [`~SeamlessM4TModel`],
+            [`~SeamlessM4TForSpeechToSpeech`] or [`~SeamlessM4TForTextToSpeech`].
+
+        > Parameters shared across sub-models


This seems to break check_repository_consistency, any ideas why @ydshieh ? I think PretrainedConfig uses something similar

Sorry for being late. I will check this afternoon!

It's conv_depthwise_kernel_size (int, defaults to 31, *optional*, defaults to 31): causing problem

change it to conv_depthwise_kernel_size (int, *optional*, defaults to 31):

then run

python utils/check_docstrings.py --fix_and_overwrite

For lines like > Text-To-Unit (t2u) model specific parameters, I think utils/check_docstrings.py could not handle it. Is this a standard way to add comment while we are in an Args block? Could this be rendered correctly?

For lines like > Text-To-Unit (t2u) model specific parameters, I think utils/check_docstrings.py could not handle it. Is this a standard way to add comment while we are in an Args block? Could this be rendered correctly?

Hi @ydshieh , thanks for your feedback! this is something that I've mimicked from PretrainedConfig

transformers/src/transformers/configuration_utils.py

Line 190 in 41c42f8

> Parameters for fine-tuning tasks

How should I handle this ? I'd like to keep these lines!

Added to OBJECT_TO_IGNORE in utils/check_docstrings.py after having check it passes our requirements without the weird > markdown.
Thanks for your offline support on this @ydshieh

ylacombe · 2023-10-09T18:42:43Z

As requested, I've made a big cleaning, I believe it's much better now, thanks for your guidance @ArthurZucker !

ylacombe · 2023-10-09T18:43:36Z

+        if src_lang is not None:
+            self.src_lang = src_lang
+        if tgt_lang is not None:
+            self.tgt_lang = tgt_lang


I actually think they should be in the docstrings, WDYT?

ylacombe · 2023-10-09T18:45:20Z

+    @classmethod
+    def _from_pretrained(
+        cls,
+        resolved_vocab_files,
+        pretrained_model_name_or_path,
+        init_configuration,
+        *init_inputs,
+        token=None,
+        cache_dir=None,
+        local_files_only=False,
+        _commit_hash=None,
+        _is_local=False,
+        **kwargs,
+    ):
+        tokenizer = super()._from_pretrained(
+            resolved_vocab_files,
+            pretrained_model_name_or_path,
+            init_configuration,
+            *init_inputs,
+            token=token,
+            cache_dir=cache_dir,
+            local_files_only=local_files_only,
+            _commit_hash=_commit_hash,
+            _is_local=_is_local,
+            **kwargs,
+        )
+
+        # ensure also set after from pretrained
+        tokenizer.set_src_lang_special_tokens(tokenizer._src_lang)
+        tokenizer.set_tgt_lang_special_tokens(tokenizer._tgt_lang)
+
+        return tokenizer


I unfortunately had to add this to pass test, otherwise, it won't use set_src_lang_special_tokens and set_tgt_lang_special_tokens

but it should be enough in the init no ?

Well, it should but from_pretrained first call __init__, and then add special tokens, so the lang tokens are set to unk_token_id

If the tokenizer fast is properly saved this should not happen (meaning the additional special tokens are added after only if you don't have a fast tokenizer file save. Otherwise it's directly passed. This is something I gotta take care in the intiialization (meaning yes you should be able to set the target language normally without having to call this

I'm quite sure this happens during the regular test suite and not the integration tests though!

ylacombe · 2023-10-10T08:48:18Z

+        pass
+
+    @unittest.skip(
+        reason="Expected missing keys serve when using SeamlessM4TForXXX.from_pretrained from a checkpoint saved by SeamlessM4TModel.save_pretrained."


Well, for example, I've set SeamlessM4TForSpeechToSpeech keys to ignore like this:
_keys_to_ignore_on_load_missing = ["text_encoder"]

But the model doesn't have a text_encoder, it is used to make it compatible with loading weights from a SeamlessM4TModel checkpoint without warnings.

ylacombe · 2023-10-10T08:49:47Z

+        pass
+
+    @unittest.skip(
+        reason="SeamlessM4TModel has actually a bigger architecture than seamlessM4T models for specific tasks."


The base model is SeamlessM4TModel, but it has actually more components than the non-base models, a.k.a the task-specific models

ylacombe · 2023-10-10T08:53:04Z

 docs/source/en/model_doc/roformer.md
 docs/source/en/model_doc/rwkv.md
 docs/source/en/model_doc/sam.md
+docs/source/en/model_doc/seamless_m4t.md


Just slow I believe, I'll move it to the right file than

ArthurZucker

Looks a lot better indeed! Left a few small nits

ArthurZucker · 2023-10-10T09:11:20Z

+        # fairseq  | '<pad>'   | '<unk>' | '<s>' | '</s>' | 'an' | 'en' | '▁d' | 'er' | 'in' | '▁s'
+
+        # Mimic fairseq token-to-id alignment for the first 4 token
+        self.fairseq_tokens_to_ids = {pad_token: 0, unk_token: 1, bos_token: 2, eos_token: 3}


Suggested change

self.fairseq_tokens_to_ids = {pad_token: 0, unk_token: 1, bos_token: 2, eos_token: 3}

self._added_tokens_decoder = {AddedToken(pad_token): 0, AddedToken(unk_token): 1, AddedToken(bos_token): 2, AddedToken(eos_token): 3}

ArthurZucker · 2023-10-10T09:12:29Z

+        if token in self.fairseq_tokens_to_ids:
+            return self.fairseq_tokens_to_ids[token]
+        spm_id = self.sp_model.PieceToId(token)


this should not be needed. The tokens that are part of the added tokens decoder do not reach this part

ArthurZucker · 2023-10-10T09:14:03Z

+        inputs = self(raw_inputs, add_special_tokens=True, return_tensors=return_tensors, **extra_kwargs)
+        if "__" not in tgt_lang:
+            tgt_lang = f"__{tgt_lang}__"
+        tgt_lang_id = self.convert_tokens_to_ids(tgt_lang)


convert_tokens_to_ids calls super. convert_tokens_to_ids_with_added_voc so no worries if you don't have the fairseq token id

ArthurZucker · 2023-10-10T09:14:11Z

+            self._convert_id_to_token(i): i for i in range(self.fairseq_offset, self.vocab_size + self.fairseq_offset)
+        }
+        # need to ensure that fairseq_tokens_to_id are placed at the beginning of the vocabulary
+        vocab.update(self.fairseq_tokens_to_ids)


Suggested change

vocab.update(self.fairseq_tokens_to_ids)

ArthurZucker · 2023-10-10T09:14:54Z

+
+    def get_vocab(self):
+        vocab = {
+            self._convert_id_to_token(i): i for i in range(self.fairseq_offset, self.vocab_size + self.fairseq_offset)


Suggested change

self._convert_id_to_token(i): i for i in range(self.fairseq_offset, self.vocab_size + self.fairseq_offset)

self.convert_ids_to_tokens(i): i for i in range(self.fairseq_offset, self.vocab_size + self.fairseq_offset)

ArthurZucker · 2023-10-10T09:15:13Z

+    def _tokenize(self, text: str) -> List[str]:
+        return self.sp_model.encode(text, out_type=str)


(by that I mean if the sentencepiece model allways adds an extra space after the special tokens!

ArthurZucker · 2023-10-10T09:16:41Z

+    @classmethod
+    def _from_pretrained(
+        cls,
+        resolved_vocab_files,
+        pretrained_model_name_or_path,
+        init_configuration,
+        *init_inputs,
+        token=None,
+        cache_dir=None,
+        local_files_only=False,
+        _commit_hash=None,
+        _is_local=False,
+        **kwargs,
+    ):
+        tokenizer = super()._from_pretrained(
+            resolved_vocab_files,
+            pretrained_model_name_or_path,
+            init_configuration,
+            *init_inputs,
+            token=token,
+            cache_dir=cache_dir,
+            local_files_only=local_files_only,
+            _commit_hash=_commit_hash,
+            _is_local=_is_local,
+            **kwargs,
+        )
+
+        # ensure also set after from pretrained
+        tokenizer.set_src_lang_special_tokens(tokenizer._src_lang)
+        tokenizer.set_tgt_lang_special_tokens(tokenizer._tgt_lang)
+
+        return tokenizer


but it should be enough in the init no ?

ylacombe · 2023-10-10T10:07:53Z

Thanks for the quick review, I'll address most your comments this morning.

I still have this pending questions if you have time:

Add Seamless M4T model #25693 (comment)
That one's for @ydshieh : Add Seamless M4T model #25693 (comment)

ylacombe · 2023-10-10T11:05:47Z

Hey @ArthurZucker, your suggestions made it even cleaner! I've also added some copied from statements! Thanks for that!

Do you think you can review a bit more thoroughly the modeling code soon ?

ArthurZucker · 2023-10-10T18:38:23Z

on it tomorrow!

ArthurZucker

Overall looks good to me. THanks for the huge effort!

ArthurZucker · 2023-10-13T10:37:07Z

+}
+
+
+class SeamlessM4TConfig(PretrainedConfig):


No problem thanks for explaining! 🤗

ArthurZucker · 2023-10-13T10:37:59Z

+
+
+def _load_hf_config(model_type="medium"):
+    if model_type == "medium":


ok, wondering for potential futur models but we'll see then!

ArthurZucker · 2023-10-13T10:38:14Z

+    Meta SeamlessM4T is made of 8 main components:
+    - speech_encoder (#1) and speech_encoder_frontend (#2)
+    - t2u_model (#3)
+    - text_encoder (#4) and text_encoder_frontend (#5)
+    - text_decoder (#6) [and text_decoder_frontend (#5) = equals to text_encoder_frontend]
+    - final_proj (#7)
+    - vocoder (#8)


nice thanks!

ArthurZucker · 2023-10-13T10:39:29Z

+            frame_length=400,
+            hop_length=160,
+            fft_length=512,
+            power=2.0,
+            center=False,
+            preemphasis=0.97,
+            mel_filters=self.mel_filters,
+            log_mel="log",
+            mel_floor=1.192092955078125e-07,
+            remove_dc_offset=True,


no worries!

ArthurZucker · 2023-10-13T11:06:26Z

+            raise ValueError(
+                "`input_ids`,`input_features`, `inputs_embeds` and `encoder_outputs` are all empty. Make sure at least one of them is not."
+            )
+        elif input_features is not None:


so many warningsss

ylacombe · 2023-10-18T13:28:01Z

Thanks for the last review @ArthurZucker!
Two last things to address before merging:
1.~~SeamlessM4TConfig formatting: #25693 (comment) - cc @ydshieh , could you take a look at it?~~
2. Assisted generation test: I can't seem to pass SeamlessM4TModelWithTextInputTest::test_assisted_decoding_sample and this might be out of my scope. Could you take a look at it @gante ? It's basically testing with SeamlessM4TForTextToText which is a basic seq2seq model! [UPDATE: ignore test after offline discusions]

…down

ylacombe added 30 commits August 16, 2023 14:56

first raw commit

fb7d0ab

still POC

48da0bf

tentative convert script

2c493a5

almost working speech encoder conversion scripts

ef5106d

intermediate code for encoder/decoders

d83ea6b

add modeling code

f0bc513

first version of speech encoder

70661ae

make style

3874353

add new adapter layer architecture

c37b7bd

add adapter block

3ded19b

add first tentative config

0bf81fd

add working speech encoder conversion

4bbf681

base model convert works now

e54bdd5

make style

0de52f7

remove unnecessary classes

ca7a980

remove unecessary functions

aac2a34

add modeling code speech encoder

3735b07

rework logics

66225db

forward pass of sub components work

41a826f

add modeling codes

ae3a7e0

some config modifs and modeling code modifs

4f29e2e

save WIP

4692f59

new edits

451d11e

same output speech encoder

319333e

correct attention mask

5928c72

correct attention mask

8bd3a17

fix generation

0342b69

new generation logics

09331ac

erase comments

b20f23b

make style

74d06c1

ylacombe commented Oct 10, 2023

View reviewed changes

ArthurZucker reviewed Oct 10, 2023

View reviewed changes

ylacombe added 2 commits October 10, 2023 10:40

modify tokenizer according to last comments

48b3488

add copied from statements in tests

ebee245

correct convert script

87a5886

ArthurZucker approved these changes Oct 13, 2023

View reviewed changes

ylacombe and others added 9 commits October 17, 2023 16:01

Merge branch 'huggingface:main' into add-S2S-model

e4685cb

correct parameter docstring style

cad5136

correct tokenization

a4f437d

correct multi gpus

c367cb9

make style

b137431

clean modeling code

8c7f5a4

make style

22aca15

add copied from statements

b0e2626

add copied statements

bec7235

ylacombe and others added 8 commits October 18, 2023 13:46

add support with ASR pipeline

14c4d4a

remove file added inadvertently

121187a

fix docstrings seamlessM4TModel

0563778

add seamlessM4TConfig to OBJECTS_TO_IGNORE due of unconventional mark…

7620fd6

…down

Merge branch 'huggingface:main' into add-S2S-model

79dda0a

add seamlessm4t to assisted generation ignored models

e65cf14

Merge branch 'huggingface:main' into add-S2S-model

8682fc1

Merge branch 'huggingface:main' into add-S2S-model

6369fd6

ylacombe merged commit cb45f71 into huggingface:main Oct 23, 2023

	self.fairseq_tokens_to_ids = {pad_token: 0, unk_token: 1, bos_token: 2, eos_token: 3}
	self._added_tokens_decoder = {AddedToken(pad_token): 0, AddedToken(unk_token): 1, AddedToken(bos_token): 2, AddedToken(eos_token): 3}

	self._convert_id_to_token(i): i for i in range(self.fairseq_offset, self.vocab_size + self.fairseq_offset)
	self.convert_ids_to_tokens(i): i for i in range(self.fairseq_offset, self.vocab_size + self.fairseq_offset)

		def _tokenize(self, text: str) -> List[str]:
		return self.sp_model.encode(text, out_type=str)



		def _load_hf_config(model_type="medium"):
		if model_type == "medium":

		}


		class SeamlessM4TConfig(PretrainedConfig):

Uh oh!

Conversation

ylacombe commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

ylacombe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ydshieh Oct 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ylacombe commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ylacombe commented Oct 10, 2023

Uh oh!

ArthurZucker commented Oct 10, 2023

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ylacombe commented Aug 23, 2023 •

edited

Loading

ydshieh Oct 13, 2023 •

edited

Loading

ylacombe commented Oct 10, 2023 •

edited

Loading

ylacombe commented Oct 18, 2023 •

edited

Loading