CodeLlama tokenizer drops leading whitespace on decode round-trip #47487

System Info

• transformers version: 5.15.0.dev0 (main)
• tokenizers version: 0.22.2
• Python: 3.11
• PyTorch: not relevant (tokenizer-only repro)

This continues #46491, which was auto-closed by the stale bot without a fix. The bug is still present on current main, so I am refiling with an up-to-date reproduction and one extra detail (tabs/newlines) that the earlier thread did not cover. I have scoped this in detail and a PR will follow.

Who can help?

@ArthurZucker @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf")
for s in ["   leading spaces", "    indented_line"]:
    decoded = tok.decode(tok.encode(s, add_special_tokens=False))
    print(f"{'OK' if decoded == s else 'BUG'}: {s!r} -> {decoded!r}")

Output on transformers 5.15.0.dev0 (main), tokenizers 0.22.2:

BUG: '   leading spaces' -> '  leading spaces'
BUG: '    indented_line' -> '   indented_line'

Exactly one leading space is lost on round-trip. Trailing, internal, and non-space characters round-trip correctly.

Root cause: src/transformers/models/code_llama/tokenization_code_llama.py sets the decoder to:

self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse(), decoders.Strip(content=" ", left=1)]
)

The trailing Strip(content=" ", left=1) unconditionally removes one leading space. It is meant to drop the synthetic ▁ that the Metaspace pre-tokenizer prepends to the first token (prepend_scheme="first"), but by decode time the synthetic prefix and a real user-provided leading space are indistinguishable in the string, so it eats a real space whenever the input has two or more leading spaces. The token structure does distinguish them (the synthetic prefix merges into the first content token, e.g. ▁hello=22172, while a real leading space is a separate pure-▁ token such as ▁=29871 or ▁▁=259), and any fix also has to handle inputs starting with a tab, newline, or non-ASCII character, which carry the synthetic ▁ but are not spaces.

Expected behavior

decode(encode(s)) should preserve the user's leading whitespace, so " indented_line" round-trips to " indented_line" rather than losing a space. This matters for code, where leading indentation is significant.

The one exception is a single leading space: " hello" and "hello" encode to identical ids ([22172]), so that case is genuinely unrecoverable and not expected to round-trip. Two-or-more leading spaces, and leading tabs/newlines, should all be preserved.