System Info
transformers version: 4.43.0.dev0- Platform: Linux-6.5.0-44-generic-x86_64-with-glibc2.39
- Python version: 3.10.14
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.3
- Accelerate version: 0.25.0
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:
- Using GPU in script?:
- GPU type: NVIDIA RTX A4500
Who can help?
@ArthurZucker
Information
Tasks
Reproduction
- change the
output_attentions=output_attentions to be output_attentions=True at https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L617
- run the official example script:
from PIL import Image
import requests
from transformers import AutoProcessor, LlavaForConditionalGeneration
model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf")
processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")
prompt = "USER: <image>\nWhat's the content of the image? ASSISTANT:"
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=prompt, images=image, return_tensors="pt")
generate_ids = model.generate(**inputs, max_new_tokens=15)
print(
processor.batch_decode(
generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
)
The generation will be cut down like this:
USER: \nWhat's the content of the image? ASSISTANT: the
This is due to the fact that the implementations of LlamaAttention and LlamaSdpaAttention are not equivalent.
It can be fixed by align the implementations of LlamaAttention.forward with the execution logic of torch.nn.functional.scaled_dot_product_attention like this:
causal_mask = attention_mask
is_causal = True if causal_mask is None and q_len > 1 else False
if attention_mask is not None: # no matter the length, we just slice it
causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
attn_weights = attn_weights + causal_mask
elif is_causal:
assert causal_mask is None
attn_bias = torch.zeros(query_states.shape[-2], key_states.shape[-2], dtype=query_states.dtype)
temp_mask = torch.ones(query_states.shape[-2], key_states.shape[-2], dtype=torch.bool).tril(diagonal=0)
attn_bias.masked_fill_(temp_mask.logical_not(), float("-inf"))
attn_bias.to(query_states.dtype)
attn_weights += attn_bias
change the implementation from
|
if attention_mask is not None: # no matter the length, we just slice it |
to the code above works for me(
transformers version: 4.43.0.dev0).
Expected behavior
The output should be like this:
What's the content of the image? ASSISTANT: The image features a street scene with a stop sign, a red building,
System Info
transformersversion: 4.43.0.dev0Who can help?
@ArthurZucker
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
output_attentions=output_attentionsto beoutput_attentions=Trueat https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L617The generation will be cut down like this:
This is due to the fact that the implementations of
LlamaAttentionandLlamaSdpaAttentionare not equivalent.It can be fixed by align the implementations of
LlamaAttention.forwardwith the execution logic oftorch.nn.functional.scaled_dot_product_attentionlike this:change the implementation from
transformers/src/transformers/models/llama/modeling_llama.py
Line 330 in e316c52
transformersversion: 4.43.0.dev0).Expected behavior
The output should be like this: