System Info

• transformers version: 4.36.0.dev0
• Platform: macOS-13.4.1-arm64-arm-64bit
• Python version: 3.10.13
• Huggingface_hub version: 0.19.3
• Safetensors version: 0.3.3
• Accelerate version: 0.23.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

def main():
    model_id = "gpt2"
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token
    model = AutoModelForCausalLM.from_pretrained(model_id)
    prefix = "Hello"

    input_ids = tokenizer([prefix], add_special_tokens=False, return_tensors="pt", padding=True)["input_ids"]

    def empty_prefix_allowed_tokens_fn(batch_id, sent):
        return []

    try:
        output = model.generate(
            input_ids,
            do_sample=False,
            max_length=10,
            num_beams=1,
            prefix_allowed_tokens_fn=empty_prefix_allowed_tokens_fn,
            num_return_sequences=1
        )
        generations = tokenizer.batch_decode(output, skip_special_tokens=True)
        print(generations)
    except RuntimeError as e:
        print("RuntimeError encountered:", e)

if __name__ == '__main__':
    main()

While the example above may seems to be very artificial, but actually this is a very common problem. Suppose you want to use LLM to generate a json object, c.f. #27557, and the constraints will return an empty set of allowed tokens once the json object is complete such as {"ip": "127.0.0.1"}, then this issue will occur.

Expected behavior

I expect the sampling should behave like in greedy search, i.e. when the allowed token list is empty, the model will return PAD instead of throwing an error.

Related issues

• model.generate with prefix_allowed_tokens_fn throws RuntimeError: probability tensor contains either inf, nan or element < 0 #15169
• RunTimeError when using prefix_allowed_tokens_fn and top-k/top-p sampling in model.generate #13707

The above two issues are about the same problem as ours, but no fix has been done so far.