v1.0.0-rc.4

Release Candidate number 4 is out!
It brings back the required Python APIs to use tokenizers in other python libraries, for example transformers, vLLM, ...

Install it with:

pip install --pre tokenizers==1.0.0rc4

We're making steady progress towards feature parity with 0.23 on encoding and decoding, the main milestone left before a stable 1.0.0!

Breaking changes from v1.0.0-rc.3

If you built something on a previous release candidate, here are the breaking changes from v1.0.0-rc.3:

• Python: The Tokenizer class no longer encodes, you need to call tokenizer.compile() first. A Tokenizer now holds the tokenizer.json config and can be edited. compile() builds a frozen CompiledTokenizer from it. CompiledTokenizer can encode and decode. (#2562)

  tokenizer = Tokenizer.from_pretrained("openai-community/gpt2")
  tokenizer.truncation = Truncation(max_length=512)
  compiled = tokenizer.compile()
  compiled.encode("Hello, World!").ids

Benefits:

• Encoding takes no lock and releases the GIL
• Copying a Tokenizer is cheap: copies share the config until one of them is edited
• Padding and truncation live in the config and become the encode defaults; each encode call can still override them

New

• Python: composition API. Build a tokenizer from components, or swap one out on an existing tokenizer:
  • tokenizer.normalizer, .pre_tokenizer, .model, .post_processor and .decoder can be read and set. Components are shared like in 0.23: editing one updates every tokenizer using it (#2579)
  • Models: BPE, Unigram, WordLevel, WordPiece, and Tokenizer(model) (#2569)
  • Post-processors: TemplateProcessing, RobertaProcessing, BertProcessing, ByteLevel and Sequence (#2573)
  • Sequence for normalizers (#2577)
  • Component classes are now generated from the Rust declarations, so adding the remaining ones is cheap (#2629)
• Python: serialization. Tokenizer.from_str, to_str and save (#2563)
• Python: vocab APIs. get_vocab, get_vocab_size, token_to_id and id_to_token (#2614)
• Python: added tokens. AddedToken, Tokenizer.add_tokens, Tokenizer.add_special_tokens and CompiledTokenizer.added_tokens. Adding tokens one at a time is ~1000x faster than 0.23.1 (5k calls: 4 ms vs 3.85 s) (#2615)
• Python: DecodeStream, compatible with vLLM's usage (#2551)

Performance

• Added tokens use ~1000x less memory: gpt-oss goes from 1.71 MiB to 1.6 KiB. Decode is up to 20% faster (#2612)
• Many threads encoding at once no longer build a throwaway thread pool when the shared one is busy: up to 3x the throughput with 32 concurrent callers (#2633)
• Loading a BPE model builds its vocabulary once instead of three times (#2613)

Fixes

• Fixes a bug causing forked child processes to hang when they loaded or unpickled a tokenizer after the parent had loaded one. This is the pattern behind datasets.map(num_proc=...) and PyTorch DataLoader fork workers (#2581, #2632)
• Fixes a bug causing byte-level BPE tokenizers whose vocabulary lacks some bytes to fail to load. They load again with a warning, and encoding drops the missing bytes, like 0.23 (#2550)
• Fixes a bug causing WordPiece tokenizers with a "\0" token in their vocabulary (e.g. ai4bharat/IndicBERTv2-MLM-only) to fail to load (#2661)
• Fixes a bug causing Split pre-tokenizers read from tokenizer.json to ignore the fancy-regex backend when it is enabled (#2547)

Internal

• Dependency updates, CI and README maintenance (#2575, #2582, #2587, #2606, #2609, #2610, #2621, #2659)

Contributors

• @SBrandeis
• @McPatate
• @ArthurZucker
• @dlowzzxx made their first contribution 🎉 Thank you!

Full Changelog: v1.0.0-rc.3...v1.0.0-rc.4

This changelog has been written with AI assistance

v1.0.0-rc.3

1.0.0-rc.3 is mainly a bindings release: it brings tokenizers to the browser with WebAssembly bindings!

Install them with:

npm install tokenizers-wasm@next

We're making steady progress towards feature parity with 0.23, the main milestone left before a stable 1.0.0.

New

• WebAssembly bindings, published on npm as tokenizers-wasm. Release candidates go out under the next dist-tag (#2450, #2497, #2552)
• Python: encode pairs of text (#2453)

Fixes

• Fixes a bug causing byte-level BPE tokenizers whose vocabulary lacks bytes that can never appear in UTF-8 (e.g. ModernBERT) to fail to load. The error now names the missing byte (#2466)
• Fixes a bug causing WordPiece tokenizers with an empty-string token in their vocabulary to fail to load (#2538)
• Fixes a bug causing the Python, Node and WASM bindings to reject any tokenizer.json with a regex Split pre-tokenizer or Replace normalizer, which covers NLLB and every model transformers converted from SentencePiece (#2536)
• Fixes a bug causing data races when Python's Encoding.*_array getters were used concurrently; they now return an owned copy instead of a view (#2464)
• Fixes a bug causing a 64-bit hash collision between two tokens to silently drop one of them when loading a vocabulary; it is now an error. Duplicate tokens log a warning instead of printing the whole vocab to stdout (#2467)
• Fixes a bug causing ParityBpeTrainer to produce a different output than the regular BPE trainer; it now trains into Vocab + Merges like the others (#2474)

Python

• huggingface_hub 2.0 is allowed (#2486)
• One unnecessary Encoding copy at the Rust/Python boundary is gone (#2452)

Internal

• Umbrella crate features: depend on tokenizers alone and pick components by feature, e.g. cargo add tokenizers -F train or cargo add tokenizers -F bpe --no-default-features. The tk-* crates still work on their own (#2476, #2477, #2479)
  • The bindings now depend on the umbrella crate (#2478, #2552)
  • The crate README and docs.rs front page now explain how to install the crate and pick its features (#2481)
• Dependency updates, CI hardening and other maintenance (#2446, #2448, #2449, #2465, #2471, #2472, #2473, #2475, #2480, #2492, #2495, #2510, #2537, #2548)

Thank you

• @SBrandeis
• @paulinebm
• @musicinmybrain

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

This changelog has been written with AI assistance

Release v0.23.3

Python package release, other packages are unchanged

🚨 Fixs a breaking change introduced in v0.23.1

What's changed

Patch for the python package with three fixes:

• Loosen the huggingface_hub requirements to allow huggingface_hub v2 #2486 (comment) - Unblocks transformers installs
• Fix type hints not resolving for pyright: #2456
• Fix a breaking change from 0.23.1 regarding truncation and overflowing: #2091

Full Changelog: v0.23.2...v0.23.3

Making tokenizers the sota tokenization library, same API, same token IDs, same standards

We are happy to finally bring tokenizers up to speed ⚡ !
tokenizers was released 7 years ago, in a time where running the model was always the bottleneck. The ecosystem has evolved so much that tokenizers became a the bottleneck.

As we saw many different libraries and project pop out, improving the tokenizer, we realized that our library's entire stack was too outdated for external contributors to contribute. This pushed us to refactor the codebase, and make sure the next developers come here to push the performances of tokenization.

The TLDR; is that we're a lot faster than before, crate size 6x smaller, peak memory is reduced.
We worked a lot to reduce the UTF-8 tax, and make sure the improvements span across all languages.

For more details on the release, read this.

Big Kudos to @SBrandeis and @McPatate for this sprint. Special thanks to @sebpop for his involvement as well.

This is the last v0 release, we are moving to v1!!

More details coming soon 👀

What's Changed

• Update daachorse from 1.0.1 to 3.0.0 by @musicinmybrain in #2050
• Ci test fixes by @sebpop in #2079
• Avoid full vocab clone in get_vocab_size() by @eunseo9311 in #2074
• ci: pin benchmark jobs to a larger dedicated runner by @ArthurZucker in #2084
• ci: target benchmark runners by group, not nonexistent label by @ArthurZucker in #2090
• Bindings: acquire model read-lock once per call instead of per pre-token by @sebpop in #2072
• ci: split audit in a separate job by @SBrandeis in #2103
• ci: install libcairo on benchmark runners by @SBrandeis in #2100
• feat(ci): enable npm's trusted publisher for releases by @McPatate in #2111
• chore: bump pyo3 to version 0.29 by @SBrandeis in #2115
• perf: add benchmarks for the decode API by @SBrandeis in #2099
• ci: cache HF test data and authenticate Hub downloads to avoid rate limits by @SBrandeis in #2102
• Implementing Parity-aware BPE by @cimeister in #1974
• Make doc builds faster by @mishig25 in #2158
• chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #2208
• chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #2244
• chore: remove stale examples (kills 50% of dependabot traffic) by @ArthurZucker in #2285
• chore(node): drop 8 unused devDependencies by @ArthurZucker in #2286
• ci: pin every action to one SHA, drop stale audit suppressions by @ArthurZucker in #2291
• chore(node): take node security alerts from 33 to 0 by @ArthurZucker in #2287
• fix(python): pin the ruff rule set so a ruff release can't redden main by @ArthurZucker in #2292
• Add a ParityBpeTrainer example and list it in the trainers API docs by @cimeister in #2217

New Contributors

• @eunseo9311 made their first contribution in #2074
• @cimeister made their first contribution in #1974

Full Changelog: v0.23.1...v0.23.2

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

🚨 Breaking changes

• Drop Python 3.9 (#1952) — requires-python = ">=3.10"; 3.9 users stay on 0.22.x.
• add_tokens normalizes content at insertion (#1995) — re-saved tokenizer.json may differ in the added_tokens block. Existing files load unchanged.
• Type stubs are precise (#1928, #1997) — methods that returned Any now return real types; mypy --strict may surface previously-hidden errors. Stub layout also moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi. This breaks the surface of some of the processors like RobertaProcessign's __init__ .
• 3.14t-only: setters/getters return PyResult<T> because of Arc<RwLock<Tokenizer>>; a poisoned lock surfaces as PyException instead of a panic.

⚡ Performance — measured locally on this Mac, not lifted from PRs

Run with cargo bench --bench <name> -- --save-baseline v0_22_2 on v0.22.2, then --baseline v0_22_2 on v0.23.1. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.

Added-vocabulary deserialize — the headline win (#1995, #1999)

bench: improve added_vocab_deserialize to reflect real-world workloads (#2000) is now representative of how transformers actually loads tokenizer.json files. The combined effect of daachorse for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:

Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".

BPE encode

The BPE per-thread cache PR (#2028) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.

Llama-3 encode

Truncation early exit (#1990)

Right-direction truncation no longer pre-tokenizes past max_length. The new truncation_benchmark doesn't exist on v0.22.2 so there's no apples-to-apples here, but the PR's own measurements on the same machine showed −20–28% across a range of max_length values for right-truncation; left-truncation unchanged.

Other perf improvements (no direct comparable bench)

• BPE::Builder::build no longer formats strings in a hot loop (#2010) — ~45% faster Tokenizer::from_file on Llama-3 in the PR's profile.
• BPE per-thread cache (#2028) — see Vera numbers in PR description for parallel scale-out.

🔄 Serialization / deserialization

The tokenizer.json format is forward-compatible: existing files load on 0.23 unchanged. Two things to know if you re-save:

• added_tokens entries created via add_tokens(..., normalized=True) will have their content normalized at save time — see breaking-change note above.
• tokenizer.train(...) no longer keeps a redundant added_tokens/special_tokens Vec separate from the added_tokens_map_r. Public API surface unchanged; only the internal struct shape moved.

bench: improve added_vocab_deserialize to reflect real-world workloads (#2000) lands a more realistic micro-benchmark for this surface; if you're tracking deserialize perf in your own CI, the new bench is the one to compare against.

🐍 Python: free-threaded 3.14t support

Dedicated wheels for python3.14t (the free-threaded build introduced in PEP 703). The wheel:

• Declares Py_MOD_GIL_NOT_USED, so importing tokenizers does not force the GIL back on.
• Builds without the abi3 cargo feature (free-threaded Python doesn't expose the limited API).
• Goes through Arc<RwLock<Tokenizer>> for the inner state so concurrent setters and encoders don't race PyO3's per-pyclass borrow check.

A new stress-test module tests/test_freethreaded.py exercises N-encoder × M-setter races on a single Tokenizer and asserts no RuntimeError: Already borrowed, no RwLock poisoning, and that sys._is_gil_enabled() is False post-import.

For the regular CPython wheel everything is unchanged.

📦 Node.js bindings: first proper multi-platform release since 2023

The npm package now ships 13 platforms (macOS x64/arm64/universal, Windows x64/i686/arm64, Linux x64/arm64/armv7 in both glibc and musl, Android arm64/armv7) — previous workflows only built 3 of those, leaving Apple Silicon / Linux ARM / Alpine users with package-not-found errors since 2023 (#1365, #1703, #1922). Fixed via #1970 + #2034, which also bumps @napi-rs/cli to v3 and switches cross-builds to cargo-zigbuild.

🧷 Type hints & typing for all classes (#1928, #1997)

Every class in the python bindings now ships proper .pyi stubs — Tokenizer, AddedToken, Encoding, every decoder / model / normalizer / pre-tokenizer / processor / trainer. Editors and type checkers (mypy, pyright, ty) see real signatures with types and docstrings instead of falling back to Any.

The stubs are generated automatically from the compiled extension via tools/stub-gen (Rust binary using pyo3-introspection). Re-running make style regenerates them; CI guards against regenerated-vs-checked-in drift. If the generator ever returns 0 docstrings (e.g. because the [patch.crates-io] pin in .cargo/config.toml falls out of sync with the pyo3 dep version), it now hard-aborts with a precise diagnostic instead of silently emitting bare-bones stubs.

>>> from tokenizers import Tokenizer
>>> # IDEs now resolve every method, every kwarg, every return type
>>> Tokenizer.from_pretrained("bert-base-cased")

⚠️ As called out in breaking changes: stricter type info means previously-hidden type errors in user code may now surface under mypy --strict.

✨ Other features

• Unigram sampling: models.Unigram now exposes alpha and nbest_size for subword regularization (parity with Google's implementation, #1994). Closes long-standing requests #730 and #849.
• Weakref support on Tokenizer (#1958) — useful for long-lived caches that don't want to keep tokenizers alive.
• CI benchmark regression detection on PRs (#2013) — every PR runs ci_benchmark against the stored baseline and posts a comparison chart to the PR.
• Longer-context Llama-3 benchmarks (#1971) for tracking head-room on multi-thousand-token inputs.

🛠 Other fixes

• EncodingVisualizer: unclosed annotation span fixed (#1911), HTML escape applied to output (#1937).
• DecodeStream: __copy__ / __deepcopy__ (#1930).
• Pre-tokenize: removed an unnecessary to_vec() from slice (#1964).
• Replace wget / norvig URL with HF Hub downloads in test data fetch (#2018).
• uv support in the Python Makefile (#1977).
• Several security-pin bumps on workflow SHAs (#2004, #2005, #2006, #2016, #2017).

👥 Contributors

Thanks to everyone who shipped commits between v0.22.2 and v0.23.1:

@ArthurZucker, @finnagin, @gordonmessmer, @jberg5, @kennethsible, @llukito, @MayCXC, @McPatate, @michaelfeil, @mrkm4ntr, @musicinmybrain, @ngoldbaum, @OhashiReon, @paulinebm, @podarok, @rtrompier, @sebpop, @Shivam-Bhardwaj, @threexc, @wheynelau, @xanderlent — plus @dependabot and @hf-security-analysis for keeping pins fresh.

Full Changelog: v0.22.2...v0.23.1

What's Changed

Okay mostly doing the release for these PR:

• Update deserialize of added tokens by @ArthurZucker in #1891
• update stub for typing by @ArthurZucker in #1896
• bump PyO3 to 0.26 by @davidhewitt in #1901

Basically good typing with at least ty, and a lot fast (from 4 to 8x faster) loading vocab with a lot of added tokens and GIL free !?

• ci: add support for building Win-ARM64 wheels by @MugundanMCW in #1869

• Add cargo-semver-checks to Rust CI workflow by @haixuanTao in #1875

• Update indicatif dependency by @gordonmessmer in #1867

• Bump node-forge from 1.3.1 to 1.3.2 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1889

• Bump js-yaml from 3.14.1 to 3.14.2 in /bindings/node by @dependabot[bot] in #1892

• fix: used normalize_str in BaseTokenizer.normalize by @ishitab02 in #1884

• [MINOR:TYPO] Update mod.rs by @cakiki in #1883

• Remove runtime stderr warning from Python bindings by @Copilot in #1898

• Mark immutable pyclasses as frozen by @ngoldbaum in #1861

• DOCS: add add_prefix_space to processors.ByteLevel by @CloseChoice in #1878

• Bump express from 4.21.2 to 4.22.1 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1903

New Contributors

• @MugundanMCW made their first contribution in #1869
• @haixuanTao made their first contribution in #1875
• @gordonmessmer made their first contribution in #1867
• @ishitab02 made their first contribution in #1884
• @Copilot made their first contribution in #1898
• @ngoldbaum made their first contribution in #1861
• @CloseChoice made their first contribution in #1878

Full Changelog: v0.22.1...v0.22.2

Release v0.22.1

Main change:

• Bump huggingface_hub upper version (#1866) from @Wauplin
• chore(trainer): add and improve trainer signature (#1838) from @shenxiangzhuang
• Some doc updates: c91d76a, 7b02178, 57eb8d7

What's Changed

• Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1827
• Implement from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by @sondalex in #1758
• fix: use AHashMap to fix compile error by @b00f in #1840
• New stream by @ArthurZucker in #1856
• [docs] Add more decoders by @pcuenca in #1849
• Fix missing parenthesis in EncodingVisualizer.calculate_label_colors by @Liam-DeVoe in #1853
• Update quicktour.mdx re: Issue #1625 by @WilliamPLaCroix in #1846
• remove stray comment by @sanderland in #1831
• Fix typo in README by @aisk in #1808
• RUSTSEC-2024-0436 - replace paste with pastey by @nystromjd in #1834
• Tokenizer: Add native async bindings, via py03-async-runtimes. by @michaelfeil in #1843

New Contributors

• @b00f made their first contribution in #1840
• @pcuenca made their first contribution in #1849
• @Liam-DeVoe made their first contribution in #1853
• @WilliamPLaCroix made their first contribution in #1846
• @sanderland made their first contribution in #1831
• @aisk made their first contribution in #1808
• @nystromjd made their first contribution in #1834
• @michaelfeil made their first contribution in #1843

Full Changelog: v0.21.3...v0.22.0rc0

Full Changelog: v0.21.3...v0.21.4

No change, the 0.21.3 release failed, this is just a re-release.

https://github.com/huggingface/tokenizers/releases/tag/v0.21.3