Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

WIP Conversion from/to huggingface's downstream models#196

Merged
tholor merged 20 commits intomasterfrom
conversion_huggingface
Jan 21, 2020
Merged

WIP Conversion from/to huggingface's downstream models#196
tholor merged 20 commits intomasterfrom
conversion_huggingface

Conversation

@tholor
Copy link
Copy Markdown
Member

@tholor tholor commented Jan 16, 2020

Let's improve the compatibility with huggingface's transformers. So far we could only load the LanguageModel and Tokenizer from their format.

I really like the idea of their new model hub (https://github.com/huggingface/transformers).
Let's add support to use those downstream models in FARM (huggingface -> FARM) and vice-versa upload FARM models there (FARM -> huggingface).

With this, there's no "lock-in" for users and they can choose whatever framework works best for them in their phase of development :)

@tholor tholor changed the title Conversion from/to huggingface's downstream models WIP Conversion from/to huggingface's downstream models Jan 16, 2020
tanaysoni
tanaysoni reviewed Jan 21, 2020
View reviewed changes
@tholor
Copy link
Copy Markdown
Member Author

tholor commented Jan 21, 2020

Ok let's merge this basic version & increase functionality in subsequent PRs (especially for NER etc.)

@tholor tholor merged commit 2b9873e into master Jan 21, 2020
@tholor tholor deleted the conversion_huggingface branch April 28, 2020 07:31
@tnhaider
Copy link
Copy Markdown

tnhaider commented Oct 5, 2020

Hi everyone,

I tried converting some models. For multi-label text classification head it tells me that only 1 head is allowed. I can work around that. However, for a model with TokenClassificationHead it tells me the following:

  File "conversion_huggingface_models.py", line 43, in convert_to_transformers
    transformer_model = model.convert_to_transformers()
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/adaptive_model.py", line 543, in convert_to_transformers
    self.language_model.model.config.id2label = {id: label for id, label in enumerate(self.prediction_heads[0].label_list)}
  File "/hpc/users/thomas.haider/Documents/workspace/python37-venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'TokenClassificationHead' object has no attribute 'label_list'

Any idea what's going on there?

Thanks.

@tnhaider
Copy link
Copy Markdown

tnhaider commented Oct 5, 2020

With a single prediction head I get the following error. It doesn't matter which script (w/ or w/o classification).

  File "conversion_huggingface_models_classification.py", line 64, in <module>
    convert_to_transformers(sys.argv[1], sys.argv[2])
  File "conversion_huggingface_models_classification.py", line 42, in convert_to_transformers
    model = AdaptiveModel.load(farm_input_dir, device="cpu")
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/adaptive_model.py", line 339, in load
    head = PredictionHead.load(config_file, strict=strict)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/prediction_head.py", line 117, in load
    prediction_head.load_state_dict(torch.load(model_file, map_location=torch.device("cpu")), strict=strict)
  File "/home/thomas.haider/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1045, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for TextClassificationHead:
	Unexpected key(s) in state_dict: "loss_fct.weight".

@Timoeller
Copy link
Copy Markdown
Contributor

Timoeller commented Oct 5, 2020

About `Unexpected key(s) in state_dict: "loss_fct.weight"

We currently do not support class weights when converting models. So you would need to train a FARM model without class weights or somehow exclude the weights from conversion.

About `AttributeError: 'TokenClassificationHead' object has no attribute 'label_list'

Could you please create a separate issue with a minimal example script, so we can reproduce your error? Similar to how we managed #553

@Timoeller Timoeller mentioned this pull request Oct 6, 2020
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@tanaysoni tanaysoni tanaysoni approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tholor @tnhaider @Timoeller @tanaysoni