Environment info
transformers version: 3.4.0- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?): Yes
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
Who can help
@sgugger
Information
Model I am using (Bert, XLNet ...):BERT
The problem arises when using:
The tasks I am working on is:
Not sure if it's considered official, but it's just finetuning for language modeling on a custom dataset.
To reproduce
Steps to reproduce the behavior:
from transformers import DataCollatorForWholeWordMask
from transformers import Trainer, TrainingArguments
from transformers import TextDataset
from transformers import BertTokenizer
from transformers import BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking')
tokenizer.add_tokens(["[new]"])
model = BertForMaskedLM.from_pretrained('bert-large-uncased-whole-word-masking')
model.resize_token_embeddings(len(tokenizer))
model.train()
dataset = TextDataset(
tokenizer=tokenizer,
file_path="./bert_train_set.txt",
block_size=512
)
data_collator = DataCollatorForWholeWordMask(
tokenizer
)
training_args = TrainingArguments(
output_dir="./BERT",
overwrite_output_dir=True,
num_train_epochs=10,
per_gpu_train_batch_size=16,
save_steps=500,
save_total_limit=2,
learning_rate = 2e-5
)
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=dataset,
prediction_loss_only=True,
)
trainer.train()/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in __call__(self, examples)
316 examples = [{"input_ids": e} for e in examples]
317
--> 318 batch_input = self._tensorize_batch(input_ids)
319
320 mask_labels = []
AttributeError: 'DataCollatorForWholeWordMask' object has no attribute '_tensorize_batch'
Expected behavior
Trainer should proceed to feed tensors to the model. I think someone just forgot to copypaste _tensorize_batch from DataCollatorForPermutationLanguageModeling, either that or inheritance is off.
Environment info
transformersversion: 3.4.0Who can help
@sgugger
Information
Model I am using (Bert, XLNet ...):BERT
The problem arises when using:
The tasks I am working on is:
Not sure if it's considered official, but it's just finetuning for language modeling on a custom dataset.
To reproduce
Steps to reproduce the behavior:
Expected behavior
Trainer should proceed to feed tensors to the model. I think someone just forgot to copypaste _tensorize_batch from DataCollatorForPermutationLanguageModeling, either that or inheritance is off.