Added support for Tapas Model by JuheonChu · Pull Request #520 · huggingface/optimum

JuheonChu · 2022-11-27T09:57:30Z

What does this PR do?

This PR adds BetterTransformerAPI for Tapas model.

Fixes #20372

Before submitting

This PR adds new support for BetterTransformer integration for the Tapas model.
This PR adds documentation that indicates BetterTransofrmer integration for Tapas is added.

Questions

Can I ask you how I can test to add Bettertransformer feature for Tapas Model?

To: @younesbelkada, @sgugger

JuheonChu · 2022-11-27T09:58:37Z

Do you mind if I ask you whether I am on the right track?
Any comment or feedback will be very appreciated.

…r-Tapas

…hub.com/JuheonChu/optimum into Add-Bettertransformer-support-for-Tapas

michaelbenayoun

For me it looks like you did everything well!
Left a few nits and comments.
@younesbelkada will be back on Wednesday, so we should wait for his approval, but great work!

michaelbenayoun · 2022-11-28T10:57:27Z

    Wav2Vec2EncoderLayerBetterTransformer,
    WhisperEncoderLayerBetterTransformer,
+    TapasLayerBetterTransformer,
+


Suggested change

HuggingFaceDocBuilderDev · 2022-11-28T11:18:58Z

The documentation is not available anymore as the PR was closed or merged.

test_better_encoder Co-authored-by: Michael Benayoun <[email protected]>

Co-authored-by: Michael Benayoun <[email protected]>

…r-Tapas

JuheonChu · 2022-11-28T13:55:57Z

Thanks for your suggestion and feedback! I reformatted with black, and can I proceed with the CircleCI test one more time to check if my reformatting file works?

sgugger

Thanks for your contribution! @younesbelkada good to merge if you are satisfied.

Co-authored-by: Sylvain Gugger <[email protected]>

fxmarty

Waiting for the tests to run but it looks good to me!

fxmarty · 2022-11-28T15:28:32Z

Apparently a test is not passing: https://github.com/huggingface/optimum/actions/runs/3566116329/jobs/5992236636

You can run them with pytest tests/bettertransformer/test_XXX.py -k "name_of_test" --exitfirst -s locally to see if a specific test passes.

Can you run as well make style for the code quality?

…hub.com/JuheonChu/optimum into Add-Bettertransformer-support-for-Tapas

JuheonChu · 2022-11-28T16:33:19Z

@fxmarty I applied make style command and made commits with it!
Thanks for your suggestion and I will try to test it out based on your suggestion.
I sincerely appreciate with your guidance.

JuheonChu · 2022-11-28T17:16:35Z

It seems to me that module packages in tests/bettertransformer/test_bettertransformer_encoder.py are importing all fail. However, I would not like to change the structure of a file system based on the suggestion from here. May I ask you for any suggestions?

Thank you for your time.

Co-authored-by: fxmarty <[email protected]>

Call super() in the _init() to inherit from BetterTransformerBaseLayer Co-authored-by: fxmarty <[email protected]>

…r-Tapas

younesbelkada

Thanks so much for your efforts @JuheonChu !
I left few comments, I think that the conversion is currently failing because the keys that you are trying to retrieve (for eg tapas_layer.attention.query.weight does not match, here you should use tapas_layer.attention.self.query.weight instead. Check the detailed modeling file here. )
One way to make sure that your conversion worked is to constantly run this snippet and debug further 💪 :

from transformers import AutoModel
from optimum.bettertransformer import BetterTransformer

model_id = "hf-internal-testing/tiny-random-TapasModel"
model = AutoModel.from_pretrained(model_id)
bt_model = BetterTransformer.transform(model)

Thanks and let us know if you face into any issue!

younesbelkada · 2022-11-29T09:30:42Z

 ALL_ENCODER_DECODER_MODELS_TO_TEST = [
    "hf-internal-testing/tiny-random-FSMTModel",
    "hf-internal-testing/tiny-random-BartModel",
+    "hf-internal-testing/tiny-random-TapasModel",


Could you move this to ALL_ENCODER_MODELS_TO_TEST instead? Tapas is an encoder-only model ;)

Reading https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/tapas#transformers.TapasModel

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.

Oh I see! Thanks for an insightful explanation! I just moved it! :)

fxmarty · 2022-11-29T09:36:30Z

+        self.in_proj_weight = nn.Parameter(
+            torch.cat(
+                [
+                    tapas_layer.attention.query.weight,


looking at the Tapas implementation, https://github.com/huggingface/transformers/blob/28247e78819ab9756b81f8df39611c333d099400/src/transformers/models/tapas/modeling_tapas.py#L442 , I think we need here and in the rest tapas_layer.attention.self.query.weight, that is why the test is failing!

Oh........ thanks you for providing me with your insights!

fxmarty · 2022-11-29T09:42:30Z

Actually, looking at https://github.com/huggingface/transformers/blob/28247e78819ab9756b81f8df39611c333d099400/src/transformers/models/tapas/modeling_tapas.py#L539

I think the TapasLayerBetterTransformer is not needed. I think we simply need to use the mapping "TapasLayer": BetLayerBetterTransformer,

What do you think @younesbelkada ?

Co-authored-by: Younes Belkada <[email protected]>

younesbelkada · 2022-11-29T11:00:50Z

yes probably, thanks for the heads up @fxmarty !
@JuheonChu , you can also modify the mapping with "TapasLayer": BertLayerBetterTransformer and let us know if the conversion works fine!

JuheonChu · 2022-11-29T11:22:06Z

Just made changes accordingly! @younesbelkada Thank you so much! I will try the conversions and let you know!
Do you mind if you can check the commit above is the one you intended to say?

fxmarty · 2022-11-29T13:17:21Z

    BertLayerBetterTransformer,
    DistilBertLayerBetterTransformer,
    FSMTEncoderLayerBetterTransformer,
+    TapasLayerBetterTransformer,


To be removed

fxmarty · 2022-11-29T13:18:48Z

Seems all good now @JuheonChu ! Thank you for this PR! So in the end using BertLayerBetterTransformer was enough to work with Tapas. If you can just address the points above, and we are ready to merge :)

JuheonChu · 2022-11-29T13:26:43Z

Thank you so much, everyone!! @fxmarty @younesbelkada @sgugger @michaelbenayoun.
I just made changes accordingly! @fxmarty.
Do you mind if you can double check if my commits are correct as you intended?

fxmarty · 2022-11-29T13:49:48Z



-class TapasLayerBetterTransformer(BetterTransformerBaseLayer):
    def __init__(self, tapas_layer, config):


I meant to remove the class TapasLayerBetterTransformer in full 😅 It is not needed in the end since BertLayerBetterTransformer can handle Tapas.

I think I removed class TapasLayerBetterTransformer. Is there anything to be adjusted in this file?

On your branch, with optimum dev installed (pip install -e . in the optimum top folder), can you try to run the script @younesbelkada provided you and see what happens?

from transformers import AutoModel from optimum.bettertransformer import BetterTransformer model_id = "hf-internal-testing/tiny-random-TapasModel" model = AutoModel.from_pretrained(model_id) bt_model = BetterTransformer.transform(model)

younesbelkada · 2022-11-30T10:27:19Z

Hey @JuheonChu !
I have pushed directly on your branch ;) The tests should now pass 💪

younesbelkada

Thanks a lot @JuheonChu for adding Tapas support for BetterTransformer! Great job!
Looking forward to your next contributions 💪

michaelbenayoun

LGTM!

JuheonChu · 2022-12-10T03:58:49Z

@younesbelkada May I ask you where I can check the merged BetterTransformer feature of Tapas model?

Added support for Tapas Model

7939e13

JuheonChu and others added 3 commits November 27, 2022 06:07

Merge branch 'huggingface:main' into Add-Bettertransformer-support-fo…

2db0d45

…r-Tapas

Added support for Tapas Model

283186a

Merge branch 'Add-Bettertransformer-support-for-Tapas' of https://git…

b874831

…hub.com/JuheonChu/optimum into Add-Bettertransformer-support-for-Tapas

michaelbenayoun reviewed Nov 28, 2022

View reviewed changes

This was referenced Nov 28, 2022

Community contribution - BetterTransformer integration for more models! huggingface/transformers#20372

Closed

Community contribution - BetterTransformer integration for more models! #488

Open

Juheon Chu and others added 4 commits November 28, 2022 08:50

reformatted files with black

7c24a42

Update tests/bettertransformer/test_bettertransformer_encoder.py

12d6154

test_better_encoder Co-authored-by: Michael Benayoun <[email protected]>

Update optimum/bettertransformer/models/encoder_models.py

14b35ca

Co-authored-by: Michael Benayoun <[email protected]>

Merge branch 'huggingface:main' into Add-Bettertransformer-support-fo…

9632f98

…r-Tapas

sgugger approved these changes Nov 28, 2022

View reviewed changes

Comment thread tests/bettertransformer/test_bettertransformer_encoder.py Outdated

Update tests/bettertransformer/test_bettertransformer_encoder.py

d2e6ebb

Co-authored-by: Sylvain Gugger <[email protected]>

fxmarty approved these changes Nov 28, 2022

View reviewed changes

Juheon Chu added 2 commits November 28, 2022 11:25

Styled optimum files

762a804

Merge branch 'Add-Bettertransformer-support-for-Tapas' of https://git…

ba1079e

…hub.com/JuheonChu/optimum into Add-Bettertransformer-support-for-Tapas

fxmarty reviewed Nov 28, 2022

View reviewed changes

Comment thread optimum/bettertransformer/models/encoder_models.py Outdated

Update optimum/bettertransformer/models/encoder_models.py

4047267

Co-authored-by: fxmarty <[email protected]>

fxmarty reviewed Nov 28, 2022

View reviewed changes

Comment thread optimum/bettertransformer/models/encoder_models.py Outdated

JuheonChu and others added 2 commits November 28, 2022 16:39

Update optimum/bettertransformer/models/encoder_models.py

ac716cc

Call super() in the _init() to inherit from BetterTransformerBaseLayer Co-authored-by: fxmarty <[email protected]>

Merge branch 'huggingface:main' into Add-Bettertransformer-support-fo…

add72dd

…r-Tapas

younesbelkada reviewed Nov 29, 2022

View reviewed changes