Skip to content

Commit 17ab8aa

Browse files
committed
[llm][2/4] Echo-gated special-token filtering and EOS metadata merge
Part 2 of the chat-template support stack split out of #16987. What this PR adds ----------------- * extension/llm/runner/text_llm_runner.cpp: Add 'is_special_token()' with a small kKnownSpecialTokens set covering Llama 3.x, Gemma, and generic <s>/</s>/<pad>/<unk> tokens, plus a regex-style match for Llama-format <|...|> tokens. wrapped_callback now suppresses these from the printed stream when GenerationConfig.echo == false. When echo == true, raw model output (including chat-template tokens) is emitted unchanged - this preserves backward compatibility for users who explicitly want to see raw tokens. * extension/llm/runner/llm_runner_helper.cpp: get_eos_ids() now MERGES the tokenizer's primary eos_tok() with any additional EOS IDs the model metadata exports under kEosIds, instead of clearing the set when metadata is present. This is correct for HF-tokenizer models (e.g. Llama 3.x) where eos_tok() = <|end_of_text|> but the model also wants <|eot_id|> as a stop token. Also logs the primary tok and only logs metadata IDs that are newly inserted. Why this is split out --------------------- These are runner-behavior changes that affect ALL TextLLMRunner users, not just the new chat-template path. They deserve focused review for backward-compat impact (echo gating) and EOS-set semantics (merge vs clear). Depends on: PR-A (extension/llm/chat_template/* + JinjaChatFormatter library) — only for stack ordering; this PR has no include or symbol dependency on that library. Original PR (full stack): #16987
1 parent 73d0b14 commit 17ab8aa

2 files changed

Lines changed: 58 additions & 6 deletions

File tree

‎extension/llm/runner/llm_runner_helper.cpp‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,9 @@ std::unordered_set<uint64_t> get_eos_ids(
158158
tokenizers::Tokenizer* tokenizer,
159159
Module* module) {
160160
std::unordered_set<uint64_t> eos_ids = {tokenizer->eos_tok()};
161-
// Get EOS IDs if available
161+
ET_LOG(Info, "Primary eos_tok = %" PRIu64, tokenizer->eos_tok());
162+
163+
// Get EOS IDs from model metadata if available
162164
auto method_names_result = module->method_names();
163165
if (method_names_result.error() != Error::Ok) {
164166
ET_LOG(Error, "Failed reading method names");
@@ -167,16 +169,17 @@ std::unordered_set<uint64_t> get_eos_ids(
167169
const auto& method_names = method_names_result.get();
168170

169171
if (method_names.count(llm::kEosIds)) {
170-
eos_ids.clear();
171172
auto execute_result = module->execute(llm::kEosIds);
172173
if (execute_result.error() != Error::Ok) {
173174
ET_LOG(Error, "Failed to execute %s", llm::kEosIds);
174175
return eos_ids;
175176
}
176177
for (const auto& eos_id : execute_result.get()) {
177178
auto value = eos_id.toScalar().to<int64_t>();
178-
eos_ids.emplace(value);
179-
ET_LOG(Info, "eos_id = %" PRId64, value);
179+
auto [_, inserted] = eos_ids.emplace(value);
180+
if (inserted) {
181+
ET_LOG(Info, "Added eos_id from model metadata: %" PRId64, value);
182+
}
180183
}
181184
}
182185
return eos_ids;

‎extension/llm/runner/text_llm_runner.cpp‎

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,51 @@
2020
#include <pytorch/tokenizers/sentencepiece.h>
2121
#include <pytorch/tokenizers/tiktoken.h>
2222

23+
#include <unordered_set>
24+
2325
namespace executorch::extension::llm {
2426

27+
namespace {
28+
// Known special tokens used by LLM chat templates.
29+
// When echo=false, these are filtered out of the streamed output so users
30+
// see clean assistant text. When echo=true, the user explicitly asked for
31+
// raw model output, so we emit them unchanged.
32+
const std::unordered_set<std::string> kKnownSpecialTokens = {
33+
// Llama 3.x tokens
34+
"<|begin_of_text|>",
35+
"<|end_of_text|>",
36+
"<|start_header_id|>",
37+
"<|end_header_id|>",
38+
"<|eot_id|>",
39+
// Gemma tokens
40+
"<bos>",
41+
"<eos>",
42+
"<start_of_turn>",
43+
"<end_of_turn>",
44+
// Common tokens
45+
"</s>",
46+
"<s>",
47+
"<pad>",
48+
"<unk>",
49+
};
50+
51+
bool is_special_token(const std::string& text) {
52+
if (text.empty()) {
53+
return false;
54+
}
55+
// Check against known special tokens
56+
if (kKnownSpecialTokens.count(text) > 0) {
57+
return true;
58+
}
59+
// Match Llama-style tokens: <|...|>
60+
if (text.size() >= 4 && text.front() == '<' && text[1] == '|' &&
61+
text.back() == '>' && text[text.size() - 2] == '|') {
62+
return true;
63+
}
64+
return false;
65+
}
66+
} // namespace
67+
2568
using ::executorch::extension::Module;
2669
using ::executorch::runtime::Error;
2770
using ::executorch::runtime::Result;
@@ -96,8 +139,14 @@ Error TextLLMRunner::generate(
96139
std::function<void(const std::string&)> wrapped_callback =
97140
[token_callback, config](const std::string& piece) {
98141
if (!config.warming) {
99-
llm::safe_printf(piece.c_str());
100-
fflush(stdout);
142+
// When echo=false, filter out special tokens (e.g. <|eot_id|>) so
143+
// users get clean assistant output. When echo=true, the user asked
144+
// for raw model output including any chat-template tokens, so emit
145+
// everything unchanged.
146+
if (config.echo || !is_special_token(piece)) {
147+
llm::safe_printf(piece.c_str());
148+
fflush(stdout);
149+
}
101150
}
102151
if (token_callback) {
103152
token_callback(piece);

0 commit comments

Comments
 (0)