fixing sentencepiece detection for transformers 5.0+ (still backwards compatible)#373
Merged
HenryNdubuaku merged 3 commits intomainfrom Feb 23, 2026
Merged
Conversation
… compatible) Signed-off-by: Noah Cylich <[email protected]>
ncylich
force-pushed
the
sentencepiece-detection-transformers-5.0
branch
from
0314014 to
afcedf3
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the HuggingFace tokenizer conversion logic to more reliably detect and locate SentencePiece .model files across newer Transformers versions, while aiming to stay compatible with older setups.
Changes:
- Added
_find_sp_model()to locate a SentencePiece model from a local directory, the HF cache, or the Hub repo file list. - Switched SentencePiece vs BPE branching in
convert_hf_tokenizer()to rely on the discovered.modelpath rather than tokenizer attributes. - Adjusted Gemma special-token backfill to use the discovered SentencePiece model path.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Noah Cylich <[email protected]>
Signed-off-by: HenryNdubuaku <[email protected]>
ncylich
added a commit
that referenced
this pull request
Feb 24, 2026
… compatible) (#373) * fixing sentencepiece detection for transformers 5.0+ (still backwards compatible) Signed-off-by: Noah Cylich <[email protected]> * cleaned unused import and made file finding more consistent Signed-off-by: Noah Cylich <[email protected]> * cleanup Signed-off-by: HenryNdubuaku <[email protected]> --------- Signed-off-by: Noah Cylich <[email protected]> Signed-off-by: HenryNdubuaku <[email protected]> Co-authored-by: HenryNdubuaku <[email protected]>
cattermelon1234
pushed a commit
to cattermelon1234/cactus
that referenced
this pull request
Feb 28, 2026
… compatible) (cactus-compute#373) * fixing sentencepiece detection for transformers 5.0+ (still backwards compatible) Signed-off-by: Noah Cylich <[email protected]> * cleaned unused import and made file finding more consistent Signed-off-by: Noah Cylich <[email protected]> * cleanup Signed-off-by: HenryNdubuaku <[email protected]> --------- Signed-off-by: Noah Cylich <[email protected]> Signed-off-by: HenryNdubuaku <[email protected]> Co-authored-by: HenryNdubuaku <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tested with both 4.57.7 and 5.2.0
across the following models:
gemma3 270m
qwen3 0.6b
lfm2 1.2b
nomic embed text v2 moe
works by first checking if the model is local and examining its folder for sp model, then checks the hf cache, and if that doesn't exist, checks the remote files list