Repository navigation
Commit 17ab8aa
committed
[llm][2/4] Echo-gated special-token filtering and EOS metadata merge
Part 2 of the chat-template support stack split out of #16987.
What this PR adds
-----------------
* extension/llm/runner/text_llm_runner.cpp: Add 'is_special_token()'
with a small kKnownSpecialTokens set covering Llama 3.x, Gemma, and
generic <s>/</s>/<pad>/<unk> tokens, plus a regex-style match for
Llama-format <|...|> tokens. wrapped_callback now suppresses these
from the printed stream when GenerationConfig.echo == false. When
echo == true, raw model output (including chat-template tokens) is
emitted unchanged - this preserves backward compatibility for users
who explicitly want to see raw tokens.
* extension/llm/runner/llm_runner_helper.cpp: get_eos_ids() now MERGES
the tokenizer's primary eos_tok() with any additional EOS IDs the
model metadata exports under kEosIds, instead of clearing the set
when metadata is present. This is correct for HF-tokenizer models
(e.g. Llama 3.x) where eos_tok() = <|end_of_text|> but the model
also wants <|eot_id|> as a stop token. Also logs the primary tok
and only logs metadata IDs that are newly inserted.
Why this is split out
---------------------
These are runner-behavior changes that affect ALL TextLLMRunner users,
not just the new chat-template path. They deserve focused review for
backward-compat impact (echo gating) and EOS-set semantics (merge vs
clear).
Depends on: PR-A (extension/llm/chat_template/* + JinjaChatFormatter
library) — only for stack ordering; this PR has no
include or symbol dependency on that library.
Original PR (full stack): #169871 parent 73d0b14 commit 17ab8aa
2 files changed
Lines changed: 58 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
161 | | - | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
162 | 164 | | |
163 | 165 | | |
164 | 166 | | |
| |||
167 | 169 | | |
168 | 170 | | |
169 | 171 | | |
170 | | - | |
171 | 172 | | |
172 | 173 | | |
173 | 174 | | |
174 | 175 | | |
175 | 176 | | |
176 | 177 | | |
177 | 178 | | |
178 | | - | |
179 | | - | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
180 | 183 | | |
181 | 184 | | |
182 | 185 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
| 24 | + | |
23 | 25 | | |
24 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
25 | 68 | | |
26 | 69 | | |
27 | 70 | | |
| |||
96 | 139 | | |
97 | 140 | | |
98 | 141 | | |
99 | | - | |
100 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
101 | 150 | | |
102 | 151 | | |
103 | 152 | | |
| |||
0 commit comments