System Info

• transformers version: 4.39.0.dev0 (commit hash 923733c)
• Platform: Linux-6.5.0-10022-tuxedo-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.20.3
• Safetensors version: 0.4.2
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.1+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

# any llama (1) based tokenizer
llama_tokenizer = AutoTokenizer.from_pretrained('hf-internal-testing/llama-tokenizer')

sentence = 'Sampel to demonstrate the issue'

input_encoding = llama_tokenizer(sentence, add_special_tokens=True)
tokens = input_encoding.tokens()
word_ids = input_encoding.word_ids()

print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')

Prints:
Tokens produced: ['<s>', '▁S', 'amp', 'el', '▁to', '▁demonstrate', '▁the', '▁issue']
Referenced word ids: [None, 0, 0, 0, 0, 0, 0, 0]

Expected behavior

Given a fast tokenizer, batch encodings should be able to return the tokenized tokens as well as the corresponding word_ids. The first None is to be expected though (see this example).