System Info

• transformers version: 4.34.0
• Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
• Python version: 3.10.13
• Huggingface_hub version: 0.17.3
• Safetensors version: 0.4.0
• Accelerate version: 0.20.3
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed

Who can help?

@younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

If you run a batch through mistral with flash attention 2 with right padding, you get

ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the 
input.

I am not doing generation, just calling forward. Is the error message incorrect and you actually meant to prevent all usage of right padding here? Or is the implementation wrong and this was meant to only prevent generate usage of right padding? Or perhaps I am missing something else. Thanks!

Expected behavior

Either right padding is ok for calling forward, or the error message correctly states the problem.