System Info

• transformers version: 4.35.2
• Platform: Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.8.18
• Huggingface_hub version: 0.19.4
• Safetensors version: 0.4.1
• Accelerate version: 0.25.0
• Accelerate config: not found
• PyTorch version (GPU?): 1.13.1+cu116 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@muellerzr @pacman100

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import os
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer
from transformers import EarlyStoppingCallback, IntervalStrategy
import numpy as np
import evaluate

os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu_id)
from datasets import Dataset, DatasetDict
train_k = pd.read_csv('train.csv', usecols=["text", "k"])
train_k.rename(columns={"text":"text", "k":"label"}, inplace=True)
val_k = pd.read_csv('val.csv', usecols=["text", "k"])
val_k.rename(columns={"text":"text", "k":"label"}, inplace=True)
test_k = pd.read_csv('test.csv', usecols=["text", "k"])
test_k.rename(columns={"text":"text", "k":"label"}, inplace=True)

train_k = Dataset.from_pandas(train_k)
val_k = Dataset.from_pandas(val_k)
test_k = Dataset.from_pandas(test_k)

ds = DatasetDict()
ds['train'] = train_k
ds['val'] = val_k
ds['test'] = test_k

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
def tokenize_function(examples):
    return tokenizer(str(examples['text']), padding="max_length", truncation=True)

tokenized_datasets = ds.map(tokenize_function)
tokenized_train_k = tokenized_datasets["train"]
tokenized_val_k = tokenized_datasets["val"]

model_k = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=6)
training_args = TrainingArguments(output_dir="trained_k_predictors", evaluation_strategy="steps", eval_steps=100, metric_for_best_model = 'f1', learning_rate=1e-3, num_train_epochs=5, weight_decay=0.01, load_best_model_at_end=True, per_device_train_batch_size = 16, per_device_eval_batch_size = 32, save_total_limit = 3, optim="adafactor", label_names=['label'], remove_unused_columns=False,)
metric = evaluate.load("f1")
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {'f1': metric.compute(predictions=predictions, references=labels)}
trainer = Trainer(model=model_k, args=training_args, train_dataset=tokenized_train_k, eval_dataset=tokenized_val_k, compute_metrics=compute_metrics, callbacks = [EarlyStoppingCallback(early_stopping_patience=3)])
trainer.train()

Error message:

{'eval_runtime': 21.6631, 'eval_samples_per_second': 208.926, 'eval_steps_per_second': 3.277, 'epoch': 0.47}
9%|████████████████ | 5 [26/1960]
00/5305 [06:15<41:00, 1.95it/s]
100%|████████████████████████████████████████████████████████████████████� $ [24/1960]
�█████████████████████████████████████████████�early stopping required metric_for_best_model, but did not find eval_f1 so [23/1960]
early stopping is disabled██████████████████████████| 71/71 [00:21<00:00, 3.42it/s]
Traceback (most recent call last):
File "/scratch/manish/apl/apl_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/scratch/manish/apl/apl_env/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/scratch/manish/apl/src/apl.py", line 386, in
main()
File "/scratch/manish/apl/src/apl.py", line 139, in main
trainer.train()
File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 1922, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 2282, in _maybe_log_save_evaluate
self._save_checkpoint(model, trial, metrics=metrics)
File "/scratch/manish/apl/apl_env/lib/python3.8/site-packages/transformers/trainer.py", line 2407, in _save_checkpoint
metric_value = metrics[metric_to_check]
KeyError: 'eval_f1'
9%|███████████████▉ | 500 [3/1960]
/5305 [06:18<1:00:34, 1.32it/s]

Expected behavior

Train the model with early stopping enabled.