System Info
transformers version: 4.28.1- Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.0
- Safetensors version: not installed
- PyTorch version (GPU?): 1.10.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
Who can help?
@ArthurZucker @younesbelkada
Information
Tasks
Reproduction
In BlipForQuestionAnswering.forward(), if the labels parameter is provided, then lines 1218-1222 set decoder_input_ids to the right-shifted version of labels, then passes both those variables into self.text_decoder, which is an instance of BlipTextLMHeadModel:
if labels is not None and decoder_input_ids is None:
# get decoder inputs from shifting lm labels to the right - this is used in training mode
decoder_input_ids = self._shift_right(labels)
# replace possible -100 values in labels by `pad_token_id`
labels = labels.masked_fill(labels == self.decoder_pad_token_id, -100)
answer_output = self.text_decoder(
input_ids=decoder_input_ids,
attention_mask=decoder_attention_mask,
encoder_hidden_states=question_embeds,
encoder_attention_mask=attention_mask,
labels=labels,
return_dict=return_dict,
reduction="mean",
)However, in the code for BlipTextLMHeadModel.forward(), it seems like it's already doing that shift for you:
if labels is not None:
# we are doing next-token prediction; shift prediction scores and input ids by one
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
labels = labels[:, 1:].contiguous().to(shifted_prediction_scores.device)
loss_fct = CrossEntropyLoss(reduction=reduction, label_smoothing=0.1)
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
if reduction == "none":
lm_loss = lm_loss.view(prediction_scores.size(0), -1).sum(1)Am I just misinterpreting this, or is the shift done twice, i.e., the loss is for next-next token prediction??
EDIT: As another point, the official Jupyter notebook for BLIP creates and instance of and trains BlipForConditionalGeneration, which also uses BlipTextLMHeadModel as the decoder. In this case, the input_ids and labels are the same (not shifted):
for idx, batch in enumerate(train_dataloader):
input_ids = batch.pop("input_ids").to(device)
pixel_values = batch.pop("pixel_values").to(device)
outputs = model(input_ids=input_ids,
pixel_values=pixel_values,
labels=input_ids)Inside BlipForConditionalGeneration.forward(), it also doesn't shift the tokens:
outputs = self.text_decoder(
input_ids=input_ids,
attention_mask=attention_mask,
encoder_hidden_states=image_embeds,
labels=labels,
return_dict=return_dict,
reduction="mean",
)EDIT 2: Seems like the original BLIP code similarly only shifts once. In BLIP_VQA.forward(), located here, there is no shift:
answer = self.tokenizer(answer, padding='longest', return_tensors="pt").to(image.device)
answer.input_ids[:,0] = self.tokenizer.bos_token_id
answer_targets = answer.input_ids.masked_fill(answer.input_ids == self.tokenizer.pad_token_id, -100)
question_output = self.text_encoder(question.input_ids,
attention_mask = question.attention_mask,
encoder_hidden_states = image_embeds,
encoder_attention_mask = image_atts,
return_dict = True)
question_states = []
question_atts = []
for b, n in enumerate(n):
question_states += [question_output.last_hidden_state[b]]*n
question_atts += [question.attention_mask[b]]*n
question_states = torch.stack(question_states,0)
question_atts = torch.stack(question_atts,0)
answer_output = self.text_decoder(answer.input_ids,
attention_mask = answer.attention_mask,
encoder_hidden_states = question_states,
encoder_attention_mask = question_atts,
labels = answer_targets,
return_dict = True,
reduction = 'none',
) and there is a shift in self.text_decoder.forward(), as seen here:
prediction_scores = self.cls(sequence_output)
if return_logits:
return prediction_scores[:, :-1, :].contiguous()
lm_loss = None
if labels is not None:
# we are doing next-token prediction; shift prediction scores and input ids by one
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
labels = labels[:, 1:].contiguous()
loss_fct = CrossEntropyLoss(reduction=reduction, label_smoothing=0.1)
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
if reduction=='none':
lm_loss = lm_loss.view(prediction_scores.size(0),-1).sum(1) Only the text_decoder itself shifts the text (again, in the forward function).
Expected behavior
N/A
System Info
transformersversion: 4.28.1Who can help?
@ArthurZucker @younesbelkada
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
In
BlipForQuestionAnswering.forward(), if thelabelsparameter is provided, then lines 1218-1222 setdecoder_input_idsto the right-shifted version oflabels, then passes both those variables intoself.text_decoder, which is an instance ofBlipTextLMHeadModel:However, in the code for
BlipTextLMHeadModel.forward(), it seems like it's already doing that shift for you:Am I just misinterpreting this, or is the shift done twice, i.e., the loss is for next-next token prediction??
EDIT: As another point, the official Jupyter notebook for BLIP creates and instance of and trains
BlipForConditionalGeneration, which also usesBlipTextLMHeadModelas the decoder. In this case, theinput_idsandlabelsare the same (not shifted):Inside
BlipForConditionalGeneration.forward(), it also doesn't shift the tokens:EDIT 2: Seems like the original BLIP code similarly only shifts once. In
BLIP_VQA.forward(), located here, there is no shift:and there is a shift in
self.text_decoder.forward(), as seen here:Only the
text_decoderitself shifts the text (again, in the forward function).Expected behavior
N/A