System Info
Transformers 4.35.2
Who can help?
No response
Information
Tasks
Reproduction
Not really a bug, more a misleading feature:
Computing the negative log-likelihood (NLL) is useful for understanding the probability of a caption for a given image, using BLIP generative text decoder. However, if one uses BLIP for ConditionalGeneration as explained here
https://huggingface.co/docs/transformers/model_doc/blip#transformers.BlipForConditionalGeneration
adapted for computation of the NLL, one would naturally do:
from PIL import Image
import requests
from transformers import AutoProcessor, BlipForConditionalGeneration
processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "A image of two cats"
inputs = processor(images=image, text=text, return_tensors="pt")
outputs = model(**inputs, labels=inputs['input_ids'])
nll=outputs.loss.item()
However, the loss is computed with label smoothing as in training, because it is hard-coded in BLIPLLM head (just like in the original Salesforce code)
|
loss_fct = CrossEntropyLoss(reduction=reduction, label_smoothing=0.1) |
Therefore it isn't the true NLL that the call to .loss returns, and I believe the documentation should be clearer on this.
I propose to:
- change the doc to make this clearer
- or add a parameter label_smoothing when initializing the BLIP model
- or add a function to compute NLL explicitely, separated from .loss, e.g.:
def return_nll(scores, target):
loss_fct = CrossEntropyLoss(reduction='mean', label_smoothing=0.0) # we're setting it to 0
loss = loss_fct(scores, target)
return loss
def compute_generative_probability(model, processor, image, text):
inputs = processor(images=image, text=text,
return_tensors="pt", padding=True)
outputs = model(**inputs, labels=inputs['input_ids'])
shifted_predictions_scores = outputs.logits[0 , :-1, :].contiguous()
shifted_labels = inputs["input_ids"][0, 1:].contiguous().to(shifted_predictions_scores.device)
nll = return_nll(shifted_predictions_scores, target=shifted_labels)
return nll
Writing this so that others researchers are aware :) Thanks a lot for the amazing library
Expected behavior
See code above
System Info
Transformers 4.35.2
Who can help?
No response
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
Not really a bug, more a misleading feature:
Computing the negative log-likelihood (NLL) is useful for understanding the probability of a caption for a given image, using BLIP generative text decoder. However, if one uses BLIP for ConditionalGeneration as explained here
https://huggingface.co/docs/transformers/model_doc/blip#transformers.BlipForConditionalGeneration
adapted for computation of the NLL, one would naturally do:
However, the loss is computed with label smoothing as in training, because it is hard-coded in BLIPLLM head (just like in the original Salesforce code)
transformers/src/transformers/models/blip/modeling_blip_text.py
Line 892 in c48787f
Therefore it isn't the true NLL that the call to .loss returns, and I believe the documentation should be clearer on this.
I propose to:
Writing this so that others researchers are aware :) Thanks a lot for the amazing library
Expected behavior
See code above