Skip to content

Commit 2d09315

Browse files
seyeong-hanmeta-codesync[bot]
authored andcommitted
Add chat format support for Llama 3 Instruct models (#16987)
Summary: Without chat formatting, Instruct models behave like base models and never generate end-of-turn tokens. So, I added chat template support to the `llama_main` runner for Llama 3 Instruct models. ## Problem When running Llama 3 Instruct models without proper chat formatting: ```bash cmake-out/examples/models/llama/llama_main \ --model_path="llama/Llama-3.2-1B-Instruct/model.pte" \ --tokenizer_path="llama/original/tokenizer.model" \ --prompt="What's the capital of France?" Paris is incorrect answer so was stated so some major tech companies may choose the other city, and for instance the CEO of Apple is Tim Cook and he is from California, and the CEO of Google is Sundar Pichai and he is from India. ``` ==> Only max_tokens limit ### Solution New CLI flags enable proper Instruct model behavior: | Flag | Description | Default | |------|-------------|---------| | `--chat_format` | Chat template (llama3, none) | `none` | | `--system_prompt` | System prompt for assistant behavior | (empty) | | `--echo` | Echo input prompt in output | `true` | ### Examples #### Basic Usage ```bash cmake-out/examples/models/llama/llama_main \ --model_path="llama/Llama-3.2-1B-Instruct/model.pte" \ --tokenizer_path="llama/original/tokenizer.model" \ --chat_format="llama3" \ --prompt="What's the capital of France?" ``` **Output:** ``` <|begin_of_text|><|start_header_id|>user<|end_header_id|> What's the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|> The capital of France is Paris. ``` #### Clean Output (Recommended for Apps) ```bash cmake-out/examples/models/llama/llama_main \ --model_path="llama/Llama-3.2-1B-Instruct/model.pte" \ --tokenizer_path="llama/original/tokenizer.model" \ --chat_format="llama3" \ --echo=false \ --prompt="What's the capital of France?" ``` **Output:** ``` The capital of France is Paris. ``` #### With System Prompt ```bash cmake-out/examples/models/llama/llama_main \ --model_path="llama/Llama-3.2-1B-Instruct/model.pte" \ --tokenizer_path="llama/original/tokenizer.model" \ --chat_format="llama3" \ --system_prompt="You are a helpful assistant. Be concise and respond in one sentence." \ --echo=false \ --prompt="Explain quantum computing" ``` **Output:** ``` Quantum computing uses quantum bits (qubits) that can exist in multiple states simultaneously to solve complex problems faster than classical computers. ``` #### Backward Compatible (No Chat Format) ```bash # For base models or pre-formatted prompts cmake-out/examples/models/llama/llama_main \ --model_path="llama/Llama-3.2-1B/model.pte" \ --tokenizer_path="llama/original/tokenizer.model" \ --chat_format="none" \ --prompt="Once upon a time" ``` ### Files Changed | File | Description | |------|-------------| | `examples/models/llama/runner/chat_formatter.h` | **NEW** - ChatFormatter for Llama 3 | | `examples/models/llama/main.cpp` | Added CLI flags | | `examples/models/llama/README.md` | Documentation with examples | | `extension/llm/runner/text_llm_runner.cpp` | Special token filtering | | `extension/llm/runner/llm_runner_helper.cpp` | EOS token merge logic | ### Before/After Comparison | Metric | Before (no chat format) | After (--chat_format=llama3 --echo=false) | |--------|------------------------|------------------------------------------| | Tokens generated | 120 (max limit) | 7 (stops at EOS) | | Output | Rambling continuation | Clean answer | | Special tokens | Visible in output | Filtered out | Pull Request resolved: #16987 Test Plan: - [x] Build with `make llama-cpu` - [x] Test `--chat_format=llama3` with Llama-3.2-1B-Instruct - [x] Verify generation stops at `<|eot_id|>` token - [x] Test `--echo=false` produces clean output without special tokens - [x] Test `--system_prompt` affects model behavior - [x] Backward compatible with `--chat_format=none` (default) Differential Revision: D92221320 Pulled By: seyeong-han
1 parent 4930e7c commit 2d09315

32 files changed

Lines changed: 1563 additions & 34 deletions

‎CMakeLists.txt‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ include(${PROJECT_SOURCE_DIR}/tools/cmake/Codegen.cmake)
9494
include(${PROJECT_SOURCE_DIR}/tools/cmake/Utils.cmake)
9595
include(CMakeDependentOption)
9696
include(ExternalProject)
97+
include(FetchContent)
9798
include(GNUInstallDirs)
9899

99100
if(NOT CMAKE_CXX_STANDARD)
@@ -380,6 +381,14 @@ set(_common_include_directories
380381
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}/executorch/runtime/core/portable_type/c10>
381382
)
382383

384+
if(TARGET jinja2cpp)
385+
install(
386+
TARGETS jinja2cpp
387+
EXPORT ExecuTorchTargets
388+
DESTINATION ${CMAKE_INSTALL_LIBDIR}
389+
)
390+
endif()
391+
383392
#
384393
# The `_<target>_srcs` lists are defined by executorch_load_build_variables.
385394
#
@@ -744,7 +753,7 @@ endif()
744753

745754
if(EXECUTORCH_BUILD_EXTENSION_LLM)
746755
if(EXECUTORCH_BUILD_EXTENSION_LLM_RUNNER)
747-
set(SUPPORT_REGEX_LOOKAHEAD ON)
756+
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extension/llm/chat_template)
748757
# llama/runner/CMakeLists.txt builds a shared library libllama_runner.so
749758
# that transitively depends on tokenizers. Need to build tokenizers with
750759
# -fPIC.

‎examples/models/llama/CMakeLists.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,31 @@ set(TORCH_ROOT ${EXECUTORCH_ROOT}/third-party/pytorch)
4848

4949
include(${EXECUTORCH_ROOT}/tools/cmake/Utils.cmake)
5050

51+
include(FetchContent)
52+
cmake_policy(SET CMP0077 NEW)
53+
FetchContent_Declare(
54+
jinja2cpp
55+
GIT_REPOSITORY https://github.com/jinja2cpp/Jinja2Cpp.git
56+
GIT_TAG 1.3.2
57+
GIT_SUBMODULES_RECURSE TRUE
58+
)
59+
set(JINJA2CPP_BUILD_TESTS
60+
OFF
61+
CACHE BOOL ""
62+
FORCE
63+
)
64+
set(JINJA2CPP_BUILD_SHARED
65+
OFF
66+
CACHE BOOL ""
67+
FORCE
68+
)
69+
set(JINJA2CPP_INSTALL
70+
OFF
71+
CACHE BOOL ""
72+
FORCE
73+
)
74+
FetchContent_MakeAvailable(jinja2cpp)
75+
5176
if(NOT PYTHON_EXECUTABLE)
5277
resolve_python_executable()
5378
endif()

‎examples/models/llama/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,10 @@ If you're interested in deploying on non-CPU backends, [please refer the non-cpu
238238
```
239239
cmake --workflow llm-release
240240
```
241+
If you build with `make llama-cpu` and hit a RapidJSON CMake error, run it as:
242+
```
243+
CMAKE_POLICY_VERSION_MINIMUM=3.5 make llama-cpu
244+
```
241245
Note for Mac users: There's a known linking issue with Xcode 15.1. Refer to the section of Common Issues and Mitigations below for solutions.
242246

243247
2. Build llama runner.
@@ -252,6 +256,65 @@ popd
252256
cmake-out/examples/models/llama/llama_main --model_path=<model pte file> --tokenizer_path=<tokenizer.model> --prompt=<prompt>
253257
```
254258

259+
### Chat Format for Instruct Models
260+
261+
For **Instruct models** (e.g., Llama-3.2-1B-Instruct), use the `--chat_format` flag to automatically wrap your prompt in the appropriate chat template. Without this, Instruct models may not generate end-of-turn tokens and will run until max tokens.
262+
263+
```bash
264+
# Basic usage with chat format
265+
cmake-out/examples/models/llama/llama_main \
266+
--model_path=<model.pte> \
267+
--tokenizer_path=<tokenizer.model> \
268+
--chat_format=llama3 \
269+
--prompt="What is the capital of France?"
270+
```
271+
272+
**Template/model compatibility:**
273+
- Use Llama templates (`llama3` or the Llama vLLM template) with Llama models.
274+
- Using a Gemma template with a Llama model will cause the model to echo Gemma tokens.
275+
276+
**Supported chat formats:**
277+
| Format | Models | Template Style |
278+
|--------|--------|----------------|
279+
| `llama3` | Llama 3.x Instruct | `<\|begin_of_text\|><\|start_header_id\|>user...` |
280+
| `gemma3` | Gemma 3 Instruct | `<bos><start_of_turn>user...` |
281+
| `jinja` | Custom template | Jinja2 chat template from file |
282+
| `none` | Base models (default) | No formatting |
283+
284+
**Additional options:**
285+
| Flag | Description | Default |
286+
|------|-------------|---------|
287+
| `--chat_format` | Chat template format (llama3, gemma3, jinja, none) | `none` |
288+
| `--chat_template_file` | Path to custom Jinja2 template (overrides `--chat_format`) | (empty) |
289+
| `--system_prompt` | System prompt to set assistant behavior | (empty) |
290+
| `--echo` | Echo input prompt in output (set to false for clean output) | `true` |
291+
292+
**Example with system prompt and clean output:**
293+
```bash
294+
cmake-out/examples/models/llama/llama_main \
295+
--model_path=<model.pte> \
296+
--tokenizer_path=<tokenizer.model> \
297+
--chat_format=llama3 \
298+
--system_prompt="You are a helpful assistant. Be concise." \
299+
--echo=false \
300+
--prompt="What is the capital of France?"
301+
302+
# Output: The capital of France is Paris.
303+
```
304+
305+
**Example with a custom template file:**
306+
```bash
307+
cmake-out/examples/models/llama/llama_main \
308+
--model_path=<model.pte> \
309+
--tokenizer_path=<tokenizer.model> \
310+
--chat_template_file=./my_template.jinja \
311+
--prompt="Hello!"
312+
```
313+
314+
**Build note:** If you see a CMake error about RapidJSON requiring
315+
`CMAKE_POLICY_VERSION_MINIMUM=3.5`, add `CMAKE_POLICY_VERSION_MINIMUM=3.5`
316+
to your build environment when running `make llama-cpu`.
317+
255318
To build for CoreML backend and validate on Mac, replace `-DEXECUTORCH_BUILD_XNNPACK=ON` with `-DEXECUTORCH_BUILD_COREML=ON`
256319

257320
If you an error about "RE2 failed to compile pattern with lookahead:...SUPPORT_REGEX_LOOKAHEAD=ON", add "-DSUPPORT_REGEX_LOOKAHEAD=ON" when building the runner.

‎examples/models/llama/main.cpp‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
* @lint-ignore-every CLANGTIDY facebook-hte-Deprecated
88
*/
99

10+
#include <executorch/examples/models/llama/runner/chat_formatter.h>
1011
#include <executorch/examples/models/llama/runner/runner.h>
1112
#include <gflags/gflags.h>
1213
#include <sstream>
@@ -77,6 +78,27 @@ DEFINE_string(
7778
"etdump.in",
7879
"If an etdump path is provided, generate an ETDump file at the specified path for profiling purposes.");
7980

81+
DEFINE_string(
82+
chat_format,
83+
"none",
84+
"Chat template format for Instruct models. Supported formats: llama3, gemma3, jinja, none (default: none). "
85+
"When set, the prompt will be wrapped in the appropriate chat template.");
86+
87+
DEFINE_string(
88+
chat_template_file,
89+
"",
90+
"Path to a custom Jinja2 chat template file. Overrides --chat_format.");
91+
92+
DEFINE_string(
93+
system_prompt,
94+
"",
95+
"System prompt for chat format (optional). Sets the behavior/personality of the assistant.");
96+
97+
DEFINE_bool(
98+
echo,
99+
true,
100+
"Echo the input prompt in the output. Set to false to only show generated text.");
101+
80102
// Helper function to parse comma-separated string lists
81103
std::vector<std::string> parseStringList(const std::string& input) {
82104
std::vector<std::string> result;
@@ -109,7 +131,41 @@ int32_t main(int32_t argc, char** argv) {
109131

110132
const char* tokenizer_path = FLAGS_tokenizer_path.c_str();
111133

112-
const char* prompt = FLAGS_prompt.c_str();
134+
// Parse chat format and create formatter
135+
auto chat_format = example::parse_chat_format(FLAGS_chat_format);
136+
auto chat_formatter =
137+
example::create_chat_formatter(chat_format, FLAGS_chat_template_file);
138+
const bool using_chat_template =
139+
chat_format != example::ChatFormat::None ||
140+
!FLAGS_chat_template_file.empty();
141+
142+
// Apply chat formatting to prompt
143+
std::string formatted_prompt =
144+
chat_formatter->format(FLAGS_prompt, FLAGS_system_prompt);
145+
const char* prompt = formatted_prompt.c_str();
146+
147+
if (using_chat_template) {
148+
if (!FLAGS_chat_template_file.empty()) {
149+
ET_LOG(
150+
Info,
151+
"Using chat template file: %s",
152+
FLAGS_chat_template_file.c_str());
153+
} else {
154+
ET_LOG(
155+
Info,
156+
"Using chat format: %s",
157+
FLAGS_chat_format.c_str());
158+
}
159+
160+
// Warn if num_bos is set since chat templates already include BOS
161+
if (FLAGS_num_bos > 0 && chat_formatter->includes_bos()) {
162+
ET_LOG(
163+
Info,
164+
"Note: Chat format '%s' already includes BOS token. "
165+
"Consider setting --num_bos=0 to avoid duplicate BOS tokens.",
166+
FLAGS_chat_format.c_str());
167+
}
168+
}
113169

114170
float temperature = FLAGS_temperature;
115171

@@ -168,6 +224,7 @@ int32_t main(int32_t argc, char** argv) {
168224
}
169225
// generate
170226
executorch::extension::llm::GenerationConfig config{
227+
.echo = FLAGS_echo,
171228
.temperature = temperature};
172229

173230
config.ignore_eos = FLAGS_ignore_eos;

‎examples/models/llama/runner/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ set(llama_runner_deps
5353

5454
target_link_libraries(llama_runner PUBLIC ${llama_runner_deps})
5555
target_link_libraries(llama_runner PUBLIC tokenizers::tokenizers)
56+
target_link_libraries(llama_runner PRIVATE jinja2cpp)
5657

5758
target_include_directories(
5859
llama_runner PUBLIC ${EXECUTORCH_ROOT}/extension/llm/tokenizers/include
Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#pragma once
10+
11+
#include <memory>
12+
#include <string>
13+
#include <unordered_map>
14+
#include <utility>
15+
16+
#include <executorch/extension/llm/runner/jinja_chat_formatter.h>
17+
18+
namespace example {
19+
20+
/**
21+
* Supported chat template formats for different model families.
22+
*/
23+
enum class ChatFormat {
24+
None, // No formatting (pass-through)
25+
Llama3, // Llama 3.x Instruct models
26+
Gemma3, // Gemma 3 Instruct models
27+
Jinja, // Custom Jinja template from file
28+
};
29+
30+
/**
31+
* Abstract base class for chat formatters.
32+
* Implementations format user prompts into model-specific chat templates.
33+
*/
34+
class ChatFormatter {
35+
public:
36+
virtual ~ChatFormatter() = default;
37+
38+
/**
39+
* Format a user prompt into the model's expected chat template.
40+
*
41+
* @param prompt The user's input message
42+
* @param system_prompt Optional system prompt to set model behavior
43+
* @return Formatted string ready for tokenization
44+
*/
45+
virtual std::string format(
46+
const std::string& prompt,
47+
const std::string& system_prompt = "") const = 0;
48+
49+
/**
50+
* Whether this formatter includes BOS token in the template.
51+
* If true, the runner should not prepend additional BOS tokens.
52+
*/
53+
virtual bool includes_bos() const = 0;
54+
};
55+
56+
/**
57+
* No formatting (pass-through).
58+
* Use when the prompt is already formatted or for base models.
59+
*/
60+
class NoChatFormatter : public ChatFormatter {
61+
public:
62+
std::string format(
63+
const std::string& prompt,
64+
const std::string& system_prompt = "") const override {
65+
(void)system_prompt; // Unused in pass-through mode
66+
return prompt;
67+
}
68+
69+
bool includes_bos() const override {
70+
return false; // User controls BOS token
71+
}
72+
};
73+
74+
class JinjaChatFormatterAdapter : public ChatFormatter {
75+
public:
76+
explicit JinjaChatFormatterAdapter(
77+
std::unique_ptr<executorch::extension::llm::JinjaChatFormatter> formatter)
78+
: formatter_(std::move(formatter)) {}
79+
80+
std::string format(
81+
const std::string& prompt,
82+
const std::string& system_prompt = "") const override {
83+
return formatter_->format(prompt, system_prompt);
84+
}
85+
86+
bool includes_bos() const override {
87+
return formatter_->includesBos();
88+
}
89+
90+
private:
91+
std::unique_ptr<executorch::extension::llm::JinjaChatFormatter> formatter_;
92+
};
93+
94+
/**
95+
* Parse a chat format string into the corresponding enum value.
96+
*
97+
* @param format_str String identifier (e.g., "llama3", "none")
98+
* @return ChatFormat enum value, defaults to None for unknown formats
99+
*/
100+
inline ChatFormat parse_chat_format(const std::string& format_str) {
101+
static const std::unordered_map<std::string, ChatFormat> format_map = {
102+
{"none", ChatFormat::None},
103+
{"llama3", ChatFormat::Llama3},
104+
{"llama3.2", ChatFormat::Llama3},
105+
{"llama32", ChatFormat::Llama3},
106+
{"llama3_2", ChatFormat::Llama3},
107+
{"gemma3", ChatFormat::Gemma3},
108+
{"jinja", ChatFormat::Jinja},
109+
};
110+
auto it = format_map.find(format_str);
111+
if (it != format_map.end()) {
112+
return it->second;
113+
}
114+
return ChatFormat::None;
115+
}
116+
117+
/**
118+
* Get a human-readable list of supported chat formats.
119+
*/
120+
inline std::string get_supported_formats() {
121+
return "llama3, gemma3, jinja, none";
122+
}
123+
124+
/**
125+
* Factory function to create the appropriate ChatFormatter instance.
126+
*
127+
* @param format The chat format to use
128+
* @return Unique pointer to a ChatFormatter instance
129+
*/
130+
inline std::unique_ptr<ChatFormatter> create_chat_formatter(
131+
ChatFormat format,
132+
const std::string& template_file = "") {
133+
using executorch::extension::llm::ChatTemplateType;
134+
using executorch::extension::llm::JinjaChatFormatter;
135+
136+
if (!template_file.empty()) {
137+
return std::make_unique<JinjaChatFormatterAdapter>(
138+
JinjaChatFormatter::fromFile(template_file));
139+
}
140+
141+
switch (format) {
142+
case ChatFormat::Llama3:
143+
return std::make_unique<JinjaChatFormatterAdapter>(
144+
JinjaChatFormatter::fromTemplate(ChatTemplateType::Llama3));
145+
case ChatFormat::Gemma3:
146+
return std::make_unique<JinjaChatFormatterAdapter>(
147+
JinjaChatFormatter::fromTemplate(ChatTemplateType::Gemma3));
148+
case ChatFormat::Jinja:
149+
return std::make_unique<NoChatFormatter>();
150+
case ChatFormat::None:
151+
default:
152+
return std::make_unique<NoChatFormatter>();
153+
}
154+
}
155+
156+
} // namespace example

0 commit comments

Comments
 (0)