Repository navigation
fix(serialize): honor the regex backend when reading Split - #2547
Merged
ArthurZucker merged 4 commits intoOct 21, 2026
Merged
Conversation
Assisted-by: OpenAI Codex
dlowzzxx
marked this pull request as ready for review
October 18, 2026 10:23
SBrandeis
reviewed
Oct 19, 2026
Co-authored-by: Simon Brandeis <[email protected]>
SBrandeis
reviewed
Oct 19, 2026
SBrandeis
left a comment
SBrandeis
left a comment
Collaborator
There was a problem hiding this comment.
Nice catch, thanks!
Let's simplify the tests too
Co-authored-by: Simon Brandeis <[email protected]> Assisted-by: OpenAI Codex
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
SBrandeis
approved these changes
Oct 19, 2026
SBrandeis
left a comment
SBrandeis
left a comment
Collaborator
There was a problem hiding this comment.
lgtm, let's wait on review from @ArthurZucker
SBrandeis
added a commit
that referenced
this pull request
Oct 22, 2026
ByteLevel, Metaspace, Split, Digits and Punctuation, with 0.23.3's argument order, defaults and JSON key order (all positional or keyword, like 0.23.3). Readable attributes: ByteLevel.add_prefix_space, Metaspace.replacement and Metaspace.split. ByteLevel.alphabet() returns the core's 256 byte-to-unicode characters as a list. Split and Punctuation share one conversion of `behavior` to its JSON spelling. An unknown behavior or prepend_scheme raises ValueError like 0.23.3, with our own message. The core refuses ByteLevel(add_prefix_space=True), 0.23's default, when it compiles a tokenizer, so that case is pinned on its JSON only. A regex Split fails at encode on this stack, which predates #2547; its JSON is pinned too. Includes the encode check of a regex Split, which main's #2547 enables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #2511; complements #2536's binding feature changes.
read_splitalways usesSplit::native, leaving regex search unavailable even whenfancy-regexis enabled. A JSON WordLevel tokenizer with vocab{a: 0, b: 1, <unk>: 2}andSplit(Regex(" +"), Removed)errors ona binstead of encoding[0, 1].Use
Split::newwhen the backend is enabled. Keep native construction without it so DeepSeek's individually unrecognized regexes can still form the fused sequence. Pipeline canonicalization remains unchanged.The regression idea follows #2534. Following review, the regression is a single direct Split encode test gated on
wordlevelandfancy-regex, as suggested by SBrandeis. The extra native-pattern test and Sequence/disabled-backend assertions have been removed.Validation on Linux Docker with Rust 1.98.1, rerun after the review changes:
Full Python/Node bindings and workspace matrices were not run. This does not claim BLOOMZ end-to-end compatibility or replace the binding fix.
AI-assisted, disclosed in the commit trailers.