You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5c19002
Browse filesBrowse the repository at this point in the historyBrowse files
Update Extensions and expose literal tokenizer encoding (#2697)
## Motivation
Applications that combine structured prompt controls with ordinary text
need to encode the text without interpreting marker-looking content as
tokenizer control syntax. Disabling `add_special_tokens` only controls
automatic token insertion; it does not disable recognition of registered
added or special token strings. Changing shared tokenizer options
between requests would also make behavior dependent on concurrent
callers.
## Approach
Update the Extensions dependency to landed commit
`58dfaaed11614e32270acb9d6a50b2a8bc7dd64e`, which includes
microsoft/onnxruntime-extensions#1134, and regenerate the dependency
governance manifest.
Expose the new primitive through the additive
`OgaTokenizerEncodeLiteral` C API and `OgaTokenizer::EncodeLiteral` C++
wrapper. Literal encoding bypasses registered control-token recognition
and automatic control-token insertion while retaining ordinary text
normalization. Successful calls append one sequence, including an empty
sequence for empty input. Invalid or unrepresentable input returns an
error without appending a sequence. Existing `OgaTokenizerEncode`
behavior and tokenizer options remain unchanged. Concurrent
legacy/literal encoding on the same tokenizer is supported with distinct
output sequences, stable options, exclusive other tokenizer operations,
and synchronized lifetimes. This is a narrow encoding-only exception to
the C API's non-thread-safe default.
## Validation
- Windows x64 CPU/MinSizeRel build of `unit_tests` against the declared
landed Extensions commit, verified from the fetched checkout with no
Extensions source override.
- Seven focused C/C++ literal API tests passed with real tokenizer/model
data and no skipped tests. Coverage includes control-marker text,
unchanged legacy behavior/options, empty input, malformed UTF-8, null
arguments, concurrent literal/legacy calls, and bounded CPU generation
from literal tokens.
- Twelve downstream native Prompt API compatibility tests passed with
the rebuilt runtime.
- Regenerated governance manifest verified to change only the Extensions
registration; changed C++ ranges passed repository clang-format checks
and whitespace validation.
- Local validation is Windows CPU only. Other platform/provider
validation remains for PR CI.
---------
Co-authored-by: Copilot <[email protected]>
0 commit comments