System Info
None
Who can help?
@ArthurZucker
Information
Tasks
Reproduction
SequenceClassification Model finds the last token that is not a padding token in each row by cal the first position of pad token when a pad_token_id is defined in the configuration.
the code in https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L951
sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1
When pad_token is eos token, what is calculated using this method will be the position of the first eos token. However, in some LLM templates, eos is often added at the end of the prompt. Also, when a model such llama does not have a default pad token, using eos as a pad token is a common practice.
For example, in llama, eos token is <|eot_id|>
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", num_labels=1)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
tokenizer.eos_token, model.config.pad_token_id, tokenizer.pad_token
# ('<|eot_id|>', None, None)
Using chat template to encode QA pairs we can get
prompt = "what is 1+1?"
response1 = "1+1=2"
response2 = "1+1=3"
conv1 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response1}]
conv2 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response2}]
conv1_tokenized = tokenizer.apply_chat_template(conv1, tokenize=True, return_tensors="pt")
conv2_tokenized = tokenizer.apply_chat_template(conv2, tokenize=True, return_tensors="pt")
# conv1_tokenized(input_ids)
# (tensor([[128000, 128006, 882, 128007, 271, 12840, 374, 220, 16,
# 10, 16, 30, 128009, 128006, 78191, 128007, 271, 16,
# 10, 16, 28, 17, 128009, 128006, 78191, 128007, 271]]),
# conv1_tokenized(tokens)
# '<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nwhat is 1+1?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n1+1=2<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n')
In Llama chat template, eos_token is added on the end of prompt. If we do not pad the input_ids and don't set pad_token_id, the score is correct.
with torch.no_grad():
score1 = model(conv1_tokenized).logits[0][0].item()
score2 = model(conv2_tokenized).logits[0][0].item()
print(f"Score for response 1: {score1}")
print(f"Score for response 2: {score2}")
# Score for response 1: 1.7297050952911377
# Score for response 2: 1.43972647190094
If we set pad_token_id = eos_token_id, we get the same score for different QA pairs with the same prompt
model.config.pad_token_id = tokenizer.eos_token_id
with torch.no_grad():
score1 = model(conv1_tokenized).logits[0][0].item()
score2 = model(conv2_tokenized).logits[0][0].item()
print(f"Score for response 1: {score1}")
print(f"Score for response 2: {score2}")
# Score for response 1: -1.857212781906128
# Score for response 2: -1.857212781906128
This is because the score of SequenceClassification Model is the last non-pad token's logits，and the last non-pad token is the token before the first eos token. This is incorrect especially when training reward models with preference pairs that have the same prompt.
The complete recurrence script is as follows
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", num_labels=1)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
prompt = "what is 1+1?"
response1 = "1+1=2"
response2 = "1+1=3"
conv1 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response1}]
conv2 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response2}]
conv1_tokenized1 = tokenizer.apply_chat_template(conv1, tokenize=True, return_tensors="pt")
conv2_tokenized1 = tokenizer.apply_chat_template(conv2, tokenize=True, return_tensors="pt")
model.config.pad_token_id = tokenizer.eos_token_id
with torch.no_grad():
score1 = model(conv1_tokenized1).logits[0][0].item()
score2 = model(conv2_tokenized1).logits[0][0].item()
print(f"Score for response 1: {score1}")
print(f"Score for response 2: {score2}")
Expected behavior
Maybe we should prioritize using attention mask to calculate the position of the last non-pad token.
System Info
None
Who can help?
@ArthurZucker
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
SequenceClassification Model finds the last token that is not a padding token in each row by cal the first position of pad token when a
pad_token_idis defined in the configuration.the code in https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L951
When pad_token is eos token, what is calculated using this method will be the position of the first eos token. However, in some LLM templates, eos is often added at the end of the prompt. Also, when a model such llama does not have a default pad token, using eos as a pad token is a common practice.
For example, in llama, eos token is <|eot_id|>
Using chat template to encode QA pairs we can get
In Llama chat template, eos_token is added on the end of prompt. If we do not pad the input_ids and don't set pad_token_id, the score is correct.
If we set pad_token_id = eos_token_id, we get the same score for different QA pairs with the same prompt
This is because the score of SequenceClassification Model is the last non-pad token's logits，and the last non-pad token is the token before the first eos token. This is incorrect especially when training reward models with preference pairs that have the same prompt.
The complete recurrence script is as follows
Expected behavior
Maybe we should prioritize using attention mask to calculate the position of the last non-pad token.