Skip to content

Commit 5c19002

Browse files
mustjabCopilot
andauthored
Update Extensions and expose literal tokenizer encoding (#2697)
## Motivation Applications that combine structured prompt controls with ordinary text need to encode the text without interpreting marker-looking content as tokenizer control syntax. Disabling `add_special_tokens` only controls automatic token insertion; it does not disable recognition of registered added or special token strings. Changing shared tokenizer options between requests would also make behavior dependent on concurrent callers. ## Approach Update the Extensions dependency to landed commit `58dfaaed11614e32270acb9d6a50b2a8bc7dd64e`, which includes microsoft/onnxruntime-extensions#1134, and regenerate the dependency governance manifest. Expose the new primitive through the additive `OgaTokenizerEncodeLiteral` C API and `OgaTokenizer::EncodeLiteral` C++ wrapper. Literal encoding bypasses registered control-token recognition and automatic control-token insertion while retaining ordinary text normalization. Successful calls append one sequence, including an empty sequence for empty input. Invalid or unrepresentable input returns an error without appending a sequence. Existing `OgaTokenizerEncode` behavior and tokenizer options remain unchanged. Concurrent legacy/literal encoding on the same tokenizer is supported with distinct output sequences, stable options, exclusive other tokenizer operations, and synchronized lifetimes. This is a narrow encoding-only exception to the C API's non-thread-safe default. ## Validation - Windows x64 CPU/MinSizeRel build of `unit_tests` against the declared landed Extensions commit, verified from the fetched checkout with no Extensions source override. - Seven focused C/C++ literal API tests passed with real tokenizer/model data and no skipped tests. Coverage includes control-marker text, unchanged legacy behavior/options, empty input, malformed UTF-8, null arguments, concurrent literal/legacy calls, and bounded CPU generation from literal tokens. - Twelve downstream native Prompt API compatibility tests passed with the rebuilt runtime. - Regenerated governance manifest verified to change only the Extensions registration; changed C++ ranges passed repository clang-format checks and whitespace validation. - Local validation is Windows CPU only. Other platform/provider validation remains for PR CI. --------- Co-authored-by: Copilot <[email protected]>
1 parent e3e8d72 commit 5c19002

7 files changed

Lines changed: 348 additions & 6 deletions

File tree

‎cmake/deps.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ pybind11;https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.zip;f78029
1414
googletest;https://github.com/google/googletest/archive/530d5c8c84abd2a46f38583ee817743c9b3a42b4.zip;5e3a61db2aa975cfd0f97ba92c818744e7fa7034
1515
microsoft_wil;https://github.com/microsoft/wil/archive/refs/tags/v1.0.230629.1.zip;e4a542a323c070376f7c2d1973d0f7ddbc1d2fa5
1616
directx_headers;https://github.com/microsoft/DirectX-Headers/archive/refs/tags/v1.613.1.zip;47653509a3371eabb156360f42faf582f314bf2e
17-
onnxruntime_extensions;https://github.com/microsoft/onnxruntime-extensions.git;99fec2db1c0a17de60eb0f1c2353c91f8d5e22ca
17+
onnxruntime_extensions;https://github.com/microsoft/onnxruntime-extensions.git;58dfaaed11614e32270acb9d6a50b2a8bc7dd64e
1818

1919
# These two dependencies are for the optional constrained decoding feature (USE_GUIDANCE)
2020
llguidance;https://github.com/microsoft/llguidance.git;94fa39128ef184ffeda33845f6d333f332a34b4d

‎src/models/preprocessing/genai_tokenizer.cpp‎

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
#include "tensor.h"
99

1010
#include <algorithm>
11+
#include <stdexcept>
12+
#include <string_view>
1113

1214
namespace Generators {
1315

@@ -99,17 +101,43 @@ std::unique_ptr<TokenizerStream> Tokenizer::CreateStream() const {
99101
}
100102

101103
void Tokenizer::UpdateOptions(const char* const* keys, const char* const* values, size_t num_options) {
102-
// Tap into ORT Extensions API
103-
CheckResult(OrtxUpdateTokenizerOptions(tokenizer_, const_cast<const char**>(keys), const_cast<const char**>(values), num_options));
104+
if (num_options > 0 && (!keys || !values))
105+
throw std::invalid_argument("Tokenizer option keys and values must not be null.");
106+
107+
bool literal = literal_;
108+
std::vector<const char*> extension_keys;
109+
std::vector<const char*> extension_values;
110+
for (size_t i = 0; i < num_options; ++i) {
111+
if (!keys[i] || !values[i])
112+
throw std::invalid_argument("Tokenizer option keys and values must not be null.");
113+
if (std::string_view(keys[i]) == "literal") {
114+
const std::string_view value(values[i]);
115+
if (value == "true" || value == "1") {
116+
literal = true;
117+
} else if (value == "false" || value == "0") {
118+
literal = false;
119+
} else {
120+
throw std::invalid_argument("literal must be true, false, 1, or 0.");
121+
}
122+
} else {
123+
extension_keys.push_back(keys[i]);
124+
extension_values.push_back(values[i]);
125+
}
126+
}
127+
128+
if (!extension_keys.empty() || num_options == 0)
129+
CheckResult(OrtxUpdateTokenizerOptions(tokenizer_, extension_keys.data(), extension_values.data(), extension_keys.size()));
130+
literal_ = literal;
104131
}
105132

106133
std::vector<int32_t> Tokenizer::Encode(const char* text) const {
107134
OrtxPtr<OrtxTokenId2DArray> ids;
108-
CheckResult(OrtxTokenize(tokenizer_, &text, 1, ids.Address()));
109-
135+
const auto tokenize = literal_ ? OrtxTokenizeLiteral : OrtxTokenize;
136+
CheckResult(tokenize(tokenizer_, &text, 1, ids.Address()));
110137
const extTokenId_t* tokens;
111138
size_t count;
112139
CheckResult(OrtxTokenId2DArrayGetItem(ids, 0, &tokens, &count));
140+
if (count == 0) return {};
113141
return {tokens, tokens + count};
114142
}
115143

‎src/models/preprocessing/genai_tokenizer.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ struct Tokenizer : std::enable_shared_from_this<Tokenizer>, LeakChecked<Tokenize
6363
OrtxPtr<OrtxTokenizer> tokenizer_;
6464

6565
private:
66+
bool literal_ = false;
6667
int32_t bos_token_id_;
6768
std::vector<int32_t> eos_token_id_;
6869
int32_t pad_token_id_;

‎src/ort_genai.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -421,6 +421,8 @@ struct OgaTokenizer : OgaAbstract {
421421
return token_id;
422422
}
423423

424+
// Encode follows the C API's encoding-only concurrency rules:
425+
// distinct output sequences, no concurrent option updates or other tokenizer operations.
424426
void Encode(const char* str, OgaSequences& sequences) const {
425427
OgaCheckResult(OgaTokenizerEncode(this, str, &sequences));
426428
}

‎src/ort_genai_c.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1119,6 +1119,8 @@ OgaResult* OGA_API_CALL OgaTokenizerGetEorTokenId(const OgaTokenizer* tokenizer,
11191119

11201120
OgaResult* OGA_API_CALL OgaTokenizerEncode(const OgaTokenizer* tokenizer, const char* str, OgaSequences* sequences) {
11211121
OGA_TRY
1122+
if (!tokenizer || !str || !sequences)
1123+
throw std::invalid_argument("tokenizer, text, and sequences must not be null.");
11221124
sequences->emplace_back(tokenizer->Encode(str));
11231125
return nullptr;
11241126
OGA_CATCH

‎src/ort_genai_c.h‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ extern "C" {
3535

3636
/** \addtogroup Global
3737
* ONNX Runtime Generative AI C API
38-
* This API is not thread safe.
38+
* This API is not thread safe unless a function explicitly documents otherwise.
3939
* @{
4040
*/
4141

@@ -863,6 +863,16 @@ OGA_EXPORT void OGA_API_CALL OgaDestroyMultiModalProcessor(OgaMultiModalProcesso
863863
*
864864
* Supported options:
865865
*
866+
* - `literal`
867+
* - Purpose: Encodes text without interpreting registered added/special token
868+
* strings or inserting automatic control tokens. Applies to single and batch
869+
* encoding, including processor calls using this tokenizer.
870+
* - Values: `"true"` / `"false"` or `"1"` / `"0"`.
871+
* - Default: `"false"`, preserving existing encoding behavior.
872+
* - When enabled, overrides automatic insertion from `add_special_tokens`
873+
* without changing that option. Returns an error for invalid UTF-8 or text
874+
* requiring added/special IDs or unknown-token fallbacks.
875+
*
866876
* - `add_special_tokens`
867877
* - Purpose: Controls whether to add special tokens (e.g., BOS/EOS) during tokenization.
868878
* - Values: `"true"` / `"false"` or `"1"` / `"0"`.
@@ -880,6 +890,8 @@ OGA_EXPORT void OGA_API_CALL OgaDestroyMultiModalProcessor(OgaMultiModalProcesso
880890
*
881891
* Future tokenizer options may be added without changing this API signature.
882892
* Passing unknown keys will result in an error.
893+
* Option updates must not overlap with encoding or other tokenizer operations.
894+
* Use separate tokenizer instances when concurrent callers need different modes.
883895
*/
884896
OGA_EXPORT OgaResult* OGA_API_CALL OgaUpdateTokenizerOptions(
885897
OgaTokenizer* tokenizer,
@@ -947,11 +959,23 @@ OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerGetEorTokenId(const OgaTokenizer*
947959
/**
948960
* Encodes a single string and adds the encoded sequence of tokens to the OgaSequences. The OgaSequences must be freed with OgaDestroySequences
949961
* when it is no longer needed.
962+
*
963+
* Uses the tokenizer's `literal` option. Successful calls append one sequence,
964+
* including an empty sequence for empty literal input. Errors leave the output
965+
* sequences unchanged. Input strings must be null-terminated UTF-8.
966+
*
967+
* Thread safety: OgaTokenizerEncode calls may run concurrently on the same
968+
* tokenizer with fixed options. Each concurrent call must use a distinct
969+
* OgaSequences output, which must not be accessed or destroyed until that call
970+
* completes. Keep the tokenizer alive until all encoding calls finish.
971+
* Do not update tokenizer options or run other operations on that tokenizer
972+
* while encoding calls are in progress.
950973
*/
951974
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerEncode(const OgaTokenizer*, const char* str, OgaSequences* sequences);
952975

953976
/**
954977
* Batch encode an array of strings and return a single tensor output
978+
* using the tokenizer's `literal` option. Padding retains the model's pad token ID.
955979
*/
956980
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerEncodeBatch(const OgaTokenizer*, const char** strings, size_t count, OgaTensor** out);
957981

0 commit comments

Comments
 (0)