Fix CodeLlama tokenizer dropping leading whitespace on decode by SuryanshSS1011 · Pull Request #47488 · huggingface/transformers

SuryanshSS1011 · 2026-07-22T21:37:50Z

What does this PR do?

From the tagged issue, CodeLlamaTokenizer builds its plain-path pipeline with a Metaspace(prepend_scheme="first") pre-tokenizer. Metaspace runs replace(" ", "▁") before its guarded prepend, so a real leading space becomes ▁ and the prepend is then skipped, which fuses a user-provided leading space into the synthetic prefix at encode time. " hello" and "hello" encode to the same ids, so the leading space is already gone before decode runs, and the ids also diverge from the pipeline the checkpoint ships in tokenizer.json.

The earlier version of this PR recovered the space in _decode, but the loss happens at encode, so this PR fixes it directly in the pipeline instead. It builds the pipeline from the Prepend("▁") + Replace(" ", "▁") normalizer with no pre-tokenizer, which is what the checkpoint ships in tokenizer.json. Leading whitespace now round-trips exactly, including a single leading space, tabs, newlines, and real code with nested indentation.

The same fusion also affected the most common round-trip, encode(text) followed by decode(ids, skip_special_tokens=True). The BOS carried the synthetic prefix, so decoding dropped it and left a spurious leading space on ordinary text ("def foo():" decoded to " def foo():"). This is fixed too.

This PR also updates test_spm_edge_cases to the reference tokenization (<s> is followed by ▁[), un-skips test_simple_encode_decode and test_no_differences_showcase which the fix restores, points the two <s> assertions at codellama/CodeLlama-7b-hf since the hf-internal-testing/llama-code-tokenizer fixture marks <s> as normalized=True, and corrects the whitespace note in code_llama.md. The existing LlamaIntegrationTest tokenizer tests still pass.

Code Agent Policy

(First-time contributors only): I confirm that this PR description and code is not written by an LLM or code agent

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline and the
Pull Request checks?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes according to the guidelines?
Did you write any new necessary tests?

Who can review?

@ArthurZucker and @itazap

Yashash-Yallapragada · 2026-07-26T13:20:06Z

Tested this branch against various leading-whitespace cases — plain encode/decode round-trips correctly now for 0/1/2/3+ leading spaces, tabs, and newlines. Nice fix, and the re-encode check is a clean way to disambiguate the synthetic prefix.

One gap: FIM/infilling mode (tokenize(prefix, suffix=...)) still has a leading-space bug, separate from the one this PR fixes.

Repro:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf")
prefix = "   def foo():\n    "
suffix = "\n    return x"
tokens = tok.tokenize(prefix, suffix=suffix)
ids = tok.convert_tokens_to_ids(tokens)
print(repr(tok.decode(ids)))
# '    def foo():\n     \n    return x'  <- 4 leading spaces, input had 3

Looks like CodeLlamaTokenizer.tokenize() prepends its own " " to the prefix for FIM formatting (separate from the Metaspace synthetic prefix), and _decode's re-encode check doesn't account for that since it only checks against plain self.encode(). Given real code almost always has leading indentation, this seems worth covering too — happy to help dig into it if useful.

SuryanshSS1011 · 2026-07-27T03:58:53Z

@ArthurZucker @itazap, can you take a look at this?

There's a separate leading-space bug in FIM infilling (raised here), which I confirmed is independent of this PR since raw _tokenizer.decode shows it too, meaning it's encode-side from the " " + text prepend in tokenize(), and not something this PR's decode check can reach.

I'd prefer to keep this PR to the plain-decode fix and open a separate issue for FIM since that prepend looks load-bearing and riskier to touch, but I can look into handling both here if that's the direction you want. How would you like this handled?

Yashash-Yallapragada · 2026-07-28T08:45:43Z

Sounds good — happy to open a separate issue for the FIM bug with the repro, so it's tracked. I can also take a shot at fixing it if that's welcome, since I've already been digging through the tokenize()/_decode interaction.

Yashash-Yallapragada · 2026-07-28T09:22:02Z

Opened #47602 to track the FIM/infilling leading-space bug separately, as discussed.

itazap

we sorry we should not have to overwrite _decode here, the normalizer should be properly constructed in the _tokenzier above

itazap · 2026-07-26T20:53:03Z

    def eot_token(self):
        return self._eot_token

+    def _decode(


sorry we should not have to overwrite _decode here, the normalizer should be properly constructed in the _tokenzier above

Thanks for the direction! Looking into the architecture more deeply, I think I have found the core fix: swapping the plain path's Metaspace pre-tokenizer for the Prepend("▁") + Replace(" ","▁") normalizer (with pre_tokenizer = None), and delete the _decode override.

Initial results support this hypothesis (leading whitespace round-trips on real CodeLlama-7b-hf, matching the shipped tokenizer.json). I will verify it completely and update the body to reflect the new approach when done!

Done, I removed _decode override and the pipeline is now built from the Prepend("▁") + Replace(" ", "▁") normalizer with pre_tokenizer = None, matching the shipped tokenizer.json. I also updated the PR body to reflect this change of approach.

Two things I would flag for you to take a look at since they may not be perfectly clear/obvious:

The pipeline has to be assigned after super().__init__(), otherwise the added-token trie is built against the Prepend normalizer and normalized=True special tokens get split mid-text (<s>).

One narrow behavior change: "<s>hello" decoded with skip_special_tokens=False now shows a space after the special token ("<s> hello"). The shipped tokenizer.json decodes identically so it is reference-faithful, and it does not affect the normal BOS path, but flagging in case you would rather special-case it.

Hey, I did some digging and code llama was based on llama2 so pre meatspace era. Additionally, CodeLlama-7b-hf is now in tests as opposed to the internal checkpoint which didnt have matching params. the init now initialized the same tokenizer as in tokenizer.json on this original checkpoint!

I made a push to support this, let me know what you think and if it covers all the edge cases you had. Thanks!

This makes sense and everything looks great! I verified the push against the edge cases I had (especially leading whitespace down to a single space, the <s> handling, and also the FIM prefix path) and they all round-trip exactly. Nothing left on my end. Thanks for taking this the rest of the way!

…is llama2

github-actions · 2026-07-31T12:10:35Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: code_llama

github-actions · 2026-07-31T12:16:19Z

CI recap

Dashboard: View test results in Grafana
Latest run: 30629584773:2
Result: success | Jobs: 2 | Tests: 7 | Failures: 0 | Duration: 18s

HuggingFaceDocBuilderDev · 2026-07-31T12:23:46Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

itazap

Ready to merge! 🤝

…gface#47488) * Build CodeLlama pipeline from the reference SentencePiece normalizer * match tokenizer init to real checkpoint's tokenizer.json, code llama is llama2 --------- Co-authored-by: itazap <[email protected]>

…#47488)" (#47861) This reverts commit 1b20eb1. #47488 replaced the `Metaspace` pre-tokenizer with a `Prepend("▁") + Replace(" ", "▁")` normalizer. That changes tokenization in two ways that break existing checkpoints. 1. Added-token contents are themselves normalized when the added-vocabulary matcher (daachorse) is rebuilt, so `<s>` is stored as `▁<s>` and stops matching `<s>` in the text. Any checkpoint shipping its specials with `normalized=True` loses them: tok.tokenize("Hey how are<s>you?<unk> <unk>") ['▁Hey', '▁how', '▁are', '<', 's', '>', 'you', '?', '<', 'unk', '>', '▁<unk>'] 2. The prepend runs on every chunk produced by the added-token split, so a metaspace is inserted after every special token. `[INST] How are you doing?<s>[/INST]` moves `[` from 29961 to 518, i.e. every multi-turn Llama-2 chat prompt tokenizes differently. #47488 wrote this into its own expected outputs rather than treating it as a bug. The leading-whitespace round-trip reported in #47487 is real, but it is decode-only and does not need a pipeline change.

SuryanshSS1011 force-pushed the fix/codellama-decode-leading-whitespace branch from 6eab0fc to 80de760 Compare July 22, 2026 22:02

Sanjays2402 reviewed Jul 24, 2026

View reviewed changes

Comment thread src/transformers/models/code_llama/tokenization_code_llama.py Resolved

Yashash-Yallapragada mentioned this pull request Jul 28, 2026

CodeLlamaTokenizer FIM/infilling drops correct leading whitespace on decode #47602

Closed

1 of 4 tasks

itazap reviewed Jul 28, 2026

View reviewed changes

SuryanshSS1011 force-pushed the fix/codellama-decode-leading-whitespace branch 2 times, most recently from c14ab06 to d91aef8 Compare July 29, 2026 04:21

Build CodeLlama pipeline from the reference SentencePiece normalizer

d01567f

SuryanshSS1011 force-pushed the fix/codellama-decode-leading-whitespace branch from d91aef8 to d01567f Compare July 29, 2026 04:21

match tokenizer init to real checkpoint's tokenizer.json, code llama …

33a00a2

…is llama2

qgallouedec mentioned this pull request Jul 31, 2026

CodeLlama tokenizer drops leading whitespace on decode round-trip #47487

Closed

2 of 4 tasks

itazap self-requested a review July 31, 2026 14:44

itazap approved these changes Jul 31, 2026

View reviewed changes

itazap added this pull request to the merge queue Jul 31, 2026

Merged via the queue into huggingface:main with commit 1b20eb1 Jul 31, 2026
32 checks passed

SuryanshSS1011 mentioned this pull request Aug 1, 2026

Build Llama pipeline from the reference SentencePiece normalizer #47700

Closed

2 of 4 tasks

ArthurZucker mentioned this pull request Aug 10, 2026

Revert "Fix CodeLlama tokenizer dropping leading whitespace on decode (#47488)" #47861

Merged

ArthurZucker mentioned this pull request Aug 10, 2026

Strip the metaspace prefix space only when it is unambiguous #47862

Open

Conversation

SuryanshSS1011 commented Jul 22, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

Uh oh!

Yashash-Yallapragada commented Jul 26, 2026

Uh oh!

SuryanshSS1011 commented Jul 27, 2026

Uh oh!

Yashash-Yallapragada commented Jul 28, 2026

Uh oh!

Yashash-Yallapragada commented Jul 28, 2026

Uh oh!

itazap left a comment

Choose a reason for hiding this comment

Uh oh!

itazap Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

SuryanshSS1011 Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuryanshSS1011 Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itazap Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuryanshSS1011 Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

CI recap

Uh oh!

HuggingFaceDocBuilderDev commented Jul 31, 2026

Uh oh!

itazap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SuryanshSS1011 commented Jul 22, 2026 •

edited by github-actions Bot

Loading

SuryanshSS1011 Jul 28, 2026 •

edited

Loading

SuryanshSS1011 Jul 29, 2026 •

edited

Loading

itazap Jul 31, 2026 •

edited

Loading

SuryanshSS1011 Jul 31, 2026 •

edited

Loading