Repository navigation
DeepSeek-Coder v1 tokenizer produces incorrect output on transformers v5+ (gap in PR #44801's fix) #46489
Description
System Info
transformersversion: 5.10.2tokenizersversion: 0.22.2- Python version: 3.11.2
- Platform: Linux (Penn State HPC, RHEL 8.10)
- PyTorch version: not relevant (tokenizer-only repro)
Who can help?
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-1.3b-instruct")
print(f"tokenizer class: {type(tok).__name__}")
for s in ["How are you doing?",
"def fib(n):\n if n < 2:\n return n",
"Hello, world! 1234",
" leading spaces"]:
ids = tok.encode(s, add_special_tokens=False)
decoded = tok.decode(ids)
print(f" match={decoded == s!s:<5} input={s!r} decoded={decoded!r}")Output on transformers 5.10.2 + tokenizers 0.22.2:
tokenizer class: LlamaTokenizer
match=False input='How are you doing?' decoded='Howareyoudoing?'
match=False input='def fib(n):\n if n < ... decoded='deffib(n):ifn<2:returnn'
match=False input='Hello, world! 1234' decoded='Hello,world!1234'
match=False input=' leading spaces' decoded='leadingspaces'
All whitespace (spaces, newlines, tabs) is stripped on the encode → decode round-trip.
Root cause
PR #44801 (closes #44779) addressed this for DeepSeek-R1 / V2 / V3 / V4 / VL / OCR by adding them to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS in src/transformers/models/auto/tokenization_auto.py. That set is keyed on model_type.
DeepSeek-Coder v1 (1.3B / 6.7B / 33B Instruct) declares model_type: llama in its config.json and tokenizer_class: LlamaTokenizerFast in tokenizer_config.json, so it falls through to LlamaTokenizer. But its tokenizer.json has the same ByteLevel-BPE pipeline as the other DeepSeek variants:
| Model | normalizer | pre_tokenizer subtypes | decoder | model |
|---|---|---|---|---|
deepseek-coder-1.3b-instruct |
Sequence (empty) |
Split×4, Digits, ByteLevel |
ByteLevel |
BPE |
DeepSeek-Coder-V2-Lite-Instruct |
Sequence (empty) |
Split×5, Digits, ByteLevel |
ByteLevel |
BPE |
DeepSeek-R1 |
Sequence (empty) |
Split×3, ByteLevel |
ByteLevel |
BPE |
LlamaTokenizer is SentencePiece-based and cannot consume a ByteLevel-BPE tokenizer.json correctly, and hence the stripped whitespace.
Why the existing fix doesn't trivially extend
Adding llama to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS would force all llama-typed models onto TokenizersBackend, including genuine Llama models that should use LlamaTokenizer. So the fix needs a different shape.
Two possible directions:
- Per-checkpoint override: explicit list of known-broken
(model_id, expected_backend)tuples. This is narrow and surgical, but the list needs maintenance as new checkpoints land. - Pipeline-shape detection: when loading, if the resolved tokenizer class is SentencePiece-based but
tokenizer.jsonhasmodel.type=BPE+decoder.type=ByteLevel, forceTokenizersBackend. More robust, slightly more code.
Happy to implement either once maintainers confirm which is preferred.
Expected behavior
tok.decode(tok.encode(s)) should round-trip whitespace, as it does on transformers v4 and on the other DeepSeek variants in v5+ (DeepSeek-R1, DeepSeek-Coder-V2-Lite-Instruct — both load as TokenizersBackend and round-trip cleanly).
This is a tokenization-correctness bug, not a UX one — downstream tasks that depend on exact-token round-trip (code-generation evaluation, edit-distance against ground-truth, RL reward signals on tokenized output) silently produce wrong values.
Activity
SuryanshSS1011 commentedon Jun 8, 2026
Coverage update
I ran a broader probe across DeepSeek model families on transformers 5.10.2; same
encode → decoderound-trip test.Results:
Same
model_type: llamamechanism, expanded scope:deepseek-coder-1.3b-{base,instruct}deepseek-coder-6.7b-{base,instruct}deepseek-coder-33b-{base,instruct}deepseek-llm-7b-{base,chat}All 8 declare
model_type: llamaand load asLlamaTokenizer. The original report's scope is "all DeepSeek-Coder v1 sizes + DeepSeek-LLM v1," not justdeepseek-coder-1.3b-instruct.Confirmed working (sanity baseline):
DeepSeek-V3.1,deepseek-vl2-tiny,DeepSeek-Coder-V2-Lite-Instruct,DeepSeek-V3— allTokenizersBackend, 0/3 mismatches.Possibly-related, different mechanism:
DeepSeek-VL-7b-chatis also broken (3/3 mismatches, loads asLlamaTokenizer), but viamodel_type: multi_modalityrather thanllama. Themulti_modalitymodel_type isn't inMODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS. Same end-state, different routing. Happy to file separately if you'd prefer to keep that scoped distinct from this one, let me know.itazap commentedon Jun 8, 2026
will add these here! https://github.com/huggingface/transformers/pull/46091/changes#diff-d804e851851cdebeb8048938f1f8beec1cfa78bf7b1f06af86faa450f9d18defR408
kpal002 commentedon Jun 12, 2026
Opened PR #46580 with a fix.
Root cause: DeepSeek-Coder v1 / DeepSeek-LLM v1 checkpoints declare
model_type: llamaandtokenizer_class: LlamaTokenizerFastin their Hub config, but theirtokenizer.jsonis actually a ByteLevel-BPE tokenizer (same architecture asTokenizersBackend). The existingMODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASSmechanism can't help here: it fires only when the registered class mismatches the hub class — but after stripping"Fast",LlamaTokenizer == LlamaTokenizer, so the check silently passes and the model falls through toLlamaTokenizer(SentencePiece-based), which strips all whitespace.Adding
"llama"to that set would break every genuine Llama checkpoint, so the fix introduces a separateCHECKPOINT_PREFIXES_WITH_INCORRECT_HUB_TOKENIZER_CLASSfrozenset keyed on Hub checkpoint name prefixes (deepseek-ai/deepseek-coder-,deepseek-ai/deepseek-llm-). An early check inAutoTokenizer.from_pretrainedroutes any matching checkpoint directly toTokenizersBackendbefore the model_type lookup runs.Also fixes
deepseek-ai/deepseek-vl-7b-{base,chat}(model_type: multi_modality) by adding"multi_modality"toMODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS.Affected checkpoints:
deepseek-ai/deepseek-coder-{1.3b,6.7b,33b}-{base,instruct}deepseek-ai/deepseek-llm-7b-{base,chat}deepseek-ai/deepseek-vl-7b-{base,chat}itazap commentedon Jun 19, 2026
fix merged
SuryanshSS1011 commentedon Jun 28, 2026
Verified on current main:
deepseek-ai/deepseek-coder-*now routes throughTokenizersBackendvia the checkpoint-id override intokenization_auto.py, so the encode→decode output is correct. Closing as fixed. Thanks!