Skip to content

Commit 7141b95

Browse files
seyeong-hanfacebook-github-bot
authored andcommitted
Add Jinja2Cpp-based chat template formatter library (#19533)
Summary: Foundation PR for the chat-template support stack split out of #16987 per reviewer request from kirklandsign. This PR adds the Jinja2Cpp-based `JinjaChatFormatter`, supporting chat-types, embedded Llama3 / Llama3.2 / Gemma3 templates, build glue (CMake/Buck), and a focused C++ unit-test suite. This PR is **reviewable in isolation** — it has no behavior change for any existing runner; downstream PRs (B/C/D) plug it in. ## Stack overview | PR | Subject | |---|---| | **1/4 (this PR)** | Library + tests | | 2/4 | TextLLMRunner echo-gated special-token filter + EOS merge | | 3/4 | Python bindings + Python LlamaRunner integration | | 4/4 | llama_main CLI flags + chat_formatter wrapper + universal Jinja docs | ## What this PR adds - **`extension/llm/chat_template/{chat_templates.h, BUCK, CMakeLists.txt, targets.bzl}`** — embedded Llama3 / Llama3.2 / Gemma3 templates and the `ChatTemplateType` enum + `ModelTokens`. The CMake file `FetchContent`s Jinja2Cpp 1.3.2, with `SUPPORT_REGEX_LOOKAHEAD` set **before** `FetchContent_MakeAvailable` so it propagates correctly, plus header staging for `nonstd` headers that some Jinja2Cpp installations omit. Installs `chat_templates.h` so SDK consumers can include it. - **`extension/llm/runner/{chat_types.h, jinja_chat_formatter.{h,cpp}}`** — the **Universal Jinja chat formatter** that supports any HuggingFace / vLLM chat template, not just the embedded ones. Loadable via `fromTemplate` (built-in), `fromString` (any string), or `fromFile` (any `.jinja` file). `formatConversation` injects vLLM/HuggingFace-standard params (`tools=[]`, `tool_choice=None`, `date_string`, `chat_template_kwargs`) so any template that references those variables renders correctly. - `normalizeTemplate` handles vLLM/HF template quirks for Jinja2Cpp: notably, `not tools is none` maps to `tools` (truthy check), preserving the intent of `tools is not none` for empty-list defaults. - **`extension/llm/runner/{CMakeLists.txt, targets.bzl}`** — link `extension_llm_runner` against `jinja2cpp` (PRIVATE) and define `EXECUTORCH_USE_JINJA2CPP`. - **`extension/llm/runner/test/test_jinja_chat_formatter.cpp`** + test build files — unit tests covering Llama3 / Llama3.2 / Gemma3 embedded templates, `parseChatTemplateType` (case-insensitive), and **three universal-Jinja regression tests**: - generic HuggingFace-style template (proves it's not Llama-specific) - tools-aware template (validates the `tools=[]` default) - `not tools is none` normalization regression test - **`CMakeLists.txt`** — adds `add_subdirectory(extension/llm/chat_template)` guarded by `EXECUTORCH_BUILD_EXTENSION_LLM_RUNNER`. - **`shim_et/xplat/executorch/build/build_variables.bzl`** — adds `jinja_chat_formatter.cpp` to the runner sources. ## Universal Jinja support Any HuggingFace / vLLM-style Jinja template works: ```cpp // From a template file (HuggingFace tokenizer_config.json, vLLM examples) auto formatter = JinjaChatFormatter::fromFile("path/to/template.jinja"); // From a string auto formatter = JinjaChatFormatter::fromString(template_str); // Built-in shortcuts auto formatter = JinjaChatFormatter::fromTemplate(ChatTemplateType::Llama3); ``` ## Notes - **No behavior change** for existing TextLLMRunner / MultimodalRunner users: the formatter is opt-in, only invoked when downstream code calls it. - **Sample vLLM templates are NOT checked in** (per reviewer feedback metascroy). Documentation in the follow-up CLI PR (4/4) points users to vLLM's `examples/` directory and HuggingFace `tokenizer_config.json` files. ## `messages[1:]` slicing Jinja2Cpp cannot parse Python-style slice syntax, so templates that do `{% set messages = messages[1:] %}` are normalized to a `_executorch_messages_tail(...)` call registered through Jinja2Cpp's `UserCallable`. Because Jinja2Cpp represents strings as non-owning `string_view`s, `formatConversation` materializes the sliced messages into a render-lifetime owning store, so the values stay valid across repeated `messages[1:]` reassignments. Test Plan: - [x] Build with `cmake --workflow llm-release` - [x] Build with `make llama-cpu` - [x] Run unit tests: `extension/llm/runner/test/test_jinja_chat_formatter` - [x] Verify embedded templates render correctly for Llama3 / Llama3.2 / Gemma3 - [x] Verify universal Jinja support with HuggingFace tokenizer_config.json templates ## Original PR Splitting #16987 into 4 reviewable PRs. cc kirklandsign larryliu0820 metascroy lucylq mergennachin ## Internal build & test (Buck / ASAN) - [x] `buck2 build fbcode//executorch/extension/llm/runner:runner_lib` — builds clean - [x] `buck2 test fbcode//executorch/extension/llm/runner/test:test_jinja_chat_formatter` — 27/27 pass under ASAN (embedded Llama3 / Llama3.2 / Gemma3 templates, the universal-Jinja regression tests, and the repeated `messages[1:]` slice lifetime test) Differential Revision: D106817172 Pulled By: seyeong-han
1 parent 0e6b67e commit 7141b95

15 files changed

Lines changed: 1161 additions & 20 deletions

File tree

‎CMakeLists.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -830,7 +830,7 @@ endif()
830830

831831
if(EXECUTORCH_BUILD_EXTENSION_LLM)
832832
if(EXECUTORCH_BUILD_EXTENSION_LLM_RUNNER)
833-
set(SUPPORT_REGEX_LOOKAHEAD ON)
833+
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extension/llm/chat_template)
834834
# llama/runner/CMakeLists.txt builds a shared library libllama_runner.so
835835
# that transitively depends on tokenizers. Need to build tokenizers with
836836
# -fPIC.

‎extension/llm/chat_template/BUCK‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
load("@fbcode_macros//build_defs:build_file_migration.bzl", "fbcode_target", "non_fbcode_target")
2+
oncall("executorch")
3+
4+
# Any targets that should be shared between fbcode and xplat must be defined in
5+
# targets.bzl. This file can contain fbcode-only targets.
6+
7+
load(":targets.bzl", "define_common_targets")
8+
9+
non_fbcode_target(_kind = define_common_targets,)
10+
11+
# !!!! fbcode/executorch/extension/llm/chat_template/TARGETS was merged into this file, see https://fburl.com/workplace/xl8l9yuo for more info !!!!
12+
13+
# Any targets that should be shared between fbcode and xplat must be defined in
14+
# targets.bzl. This file can contain fbcode-only targets.
15+
16+
load(":targets.bzl", "define_common_targets")
17+
18+
fbcode_target(_kind = define_common_targets,)
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
if(NOT EXECUTORCH_BUILD_EXTENSION_LLM_RUNNER)
2+
return()
3+
endif()
4+
5+
include(FetchContent)
6+
cmake_policy(SET CMP0077 NEW)
7+
8+
FetchContent_Declare(
9+
jinja2cpp
10+
GIT_REPOSITORY https://github.com/jinja2cpp/Jinja2Cpp.git
11+
GIT_TAG 1.3.2
12+
GIT_SHALLOW TRUE
13+
GIT_SUBMODULES_RECURSE TRUE
14+
)
15+
16+
set(JINJA2CPP_BUILD_TESTS
17+
OFF
18+
CACHE BOOL "" FORCE
19+
)
20+
set(JINJA2CPP_BUILD_SHARED
21+
OFF
22+
CACHE BOOL "" FORCE
23+
)
24+
set(JINJA2CPP_INSTALL
25+
OFF
26+
CACHE BOOL "" FORCE
27+
)
28+
# Enable PCRE2-based regex lookahead support in Jinja2Cpp. This must be set
29+
# BEFORE FetchContent_MakeAvailable(jinja2cpp) so it propagates to the Jinja2Cpp
30+
# configure step.
31+
set(SUPPORT_REGEX_LOOKAHEAD
32+
ON
33+
CACHE BOOL "" FORCE
34+
)
35+
36+
FetchContent_MakeAvailable(jinja2cpp)
37+
if(NOT TARGET jinja2cpp)
38+
message(FATAL_ERROR "Jinja2Cpp target not found after FetchContent.")
39+
endif()
40+
if(APPLE)
41+
target_compile_options(jinja2cpp PRIVATE -Wno-shorten-64-to-32)
42+
endif()
43+
44+
if(DEFINED jinja2cpp_SOURCE_DIR)
45+
function(executorch_copy_nonstd_header dep_name target header_name dest_root)
46+
set(_copied FALSE)
47+
if(TARGET ${target})
48+
get_target_property(_aliased ${target} ALIASED_TARGET)
49+
if(_aliased)
50+
set(_resolved_target ${_aliased})
51+
else()
52+
set(_resolved_target ${target})
53+
endif()
54+
get_target_property(
55+
_include_dirs ${_resolved_target} INTERFACE_INCLUDE_DIRECTORIES
56+
)
57+
foreach(_dir IN LISTS _include_dirs)
58+
if(EXISTS "${_dir}/nonstd/${header_name}")
59+
file(MAKE_DIRECTORY "${dest_root}/nonstd")
60+
file(COPY "${_dir}/nonstd/${header_name}"
61+
DESTINATION "${dest_root}/nonstd"
62+
)
63+
set(_copied TRUE)
64+
break()
65+
endif()
66+
endforeach()
67+
endif()
68+
if(NOT _copied)
69+
set(_fallback_path
70+
"${CMAKE_BINARY_DIR}/_deps/${dep_name}-src/include/nonstd/${header_name}"
71+
)
72+
if(EXISTS "${_fallback_path}")
73+
file(MAKE_DIRECTORY "${dest_root}/nonstd")
74+
file(COPY "${_fallback_path}" DESTINATION "${dest_root}/nonstd")
75+
endif()
76+
endif()
77+
endfunction()
78+
79+
set(_jinja2cpp_nonstd_root "${jinja2cpp_SOURCE_DIR}/thirdparty/nonstd")
80+
executorch_copy_nonstd_header(
81+
expected-lite nonstd::expected-lite expected.hpp
82+
"${_jinja2cpp_nonstd_root}/expected-lite/include"
83+
)
84+
executorch_copy_nonstd_header(
85+
variant-lite nonstd::variant-lite variant.hpp
86+
"${_jinja2cpp_nonstd_root}/variant-lite/include"
87+
)
88+
executorch_copy_nonstd_header(
89+
optional-lite nonstd::optional-lite optional.hpp
90+
"${_jinja2cpp_nonstd_root}/optional-lite/include"
91+
)
92+
executorch_copy_nonstd_header(
93+
string-view-lite nonstd::string-view-lite string_view.hpp
94+
"${_jinja2cpp_nonstd_root}/string-view-lite/include"
95+
)
96+
endif()
97+
98+
# Install the chat_templates.h header so that downstream consumers of the
99+
# installed ExecuTorch SDK can include
100+
# <executorch/extension/llm/chat_template/chat_templates.h>.
101+
install(FILES chat_templates.h
102+
DESTINATION include/executorch/extension/llm/chat_template
103+
)
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#pragma once
10+
11+
#include <string>
12+
#include <string_view>
13+
#include <unordered_map>
14+
#include <vector>
15+
16+
namespace executorch::extension::llm {
17+
18+
enum class ChatTemplateType {
19+
None,
20+
Llama3,
21+
Llama32,
22+
Gemma3,
23+
Custom,
24+
};
25+
26+
constexpr std::string_view kLlama3Template =
27+
R"({{ bos_token }}{%- for message in messages -%}<|start_header_id|>{{ message.role }}<|end_header_id|>
28+
29+
{{ message.content }}<|eot_id|>{%- endfor -%}{%- if add_generation_prompt -%}<|start_header_id|>assistant<|end_header_id|>
30+
31+
{%- endif -%})";
32+
33+
constexpr std::string_view kGemma3Template =
34+
R"({{ bos_token }}{%- for message in messages -%}{%- if message.role == 'assistant' -%}<start_of_turn>model
35+
{%- else -%}<start_of_turn>{{ message.role }}
36+
{%- endif -%}{{ message.content }}<end_of_turn>{%- endfor -%}{%- if add_generation_prompt -%}<start_of_turn>model
37+
{%- endif -%})";
38+
39+
inline const std::unordered_map<ChatTemplateType, std::string_view>&
40+
getEmbeddedTemplates() {
41+
static const std::unordered_map<ChatTemplateType, std::string_view>
42+
embedded_templates = {
43+
{ChatTemplateType::Llama3, kLlama3Template},
44+
{ChatTemplateType::Llama32, kLlama3Template},
45+
{ChatTemplateType::Gemma3, kGemma3Template},
46+
};
47+
return embedded_templates;
48+
}
49+
50+
struct ModelTokens {
51+
std::string bos_token;
52+
std::string eos_token;
53+
std::vector<std::string> stop_tokens;
54+
};
55+
56+
inline const std::unordered_map<ChatTemplateType, ModelTokens>&
57+
getModelTokens() {
58+
static const std::unordered_map<ChatTemplateType, ModelTokens> model_tokens =
59+
{
60+
{ChatTemplateType::Llama3,
61+
{"<|begin_of_text|>",
62+
"<|eot_id|>",
63+
{"<|eot_id|>", "<|end_of_text|>"}}},
64+
{ChatTemplateType::Llama32,
65+
{"<|begin_of_text|>",
66+
"<|eot_id|>",
67+
{"<|eot_id|>", "<|end_of_text|>"}}},
68+
{ChatTemplateType::Gemma3,
69+
{"<bos>", "<end_of_turn>", {"<end_of_turn>", "<eos>"}}},
70+
};
71+
return model_tokens;
72+
}
73+
74+
} // namespace executorch::extension::llm
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
load("@fbsource//xplat/executorch/build:runtime_wrapper.bzl", "runtime")
8+
9+
def define_common_targets():
10+
runtime.cxx_library(
11+
name = "chat_templates",
12+
exported_headers = [
13+
"chat_templates.h",
14+
],
15+
visibility = ["PUBLIC"],
16+
)

‎extension/llm/runner/CMakeLists.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,11 @@ endif()
5454
list(APPEND runner_deps kernels_util_all_deps)
5555

5656
target_link_libraries(extension_llm_runner PUBLIC ${runner_deps})
57+
target_link_libraries(extension_llm_runner PRIVATE $<LINK_ONLY:jinja2cpp>)
58+
target_include_directories(
59+
extension_llm_runner
60+
PRIVATE $<TARGET_PROPERTY:jinja2cpp,INTERFACE_INCLUDE_DIRECTORIES>
61+
)
5762
set_target_properties(
5863
extension_llm_runner PROPERTIES POSITION_INDEPENDENT_CODE ON
5964
)

‎extension/llm/runner/chat_types.h‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#pragma once
10+
11+
#include <string>
12+
#include <vector>
13+
14+
namespace executorch::extension::llm {
15+
16+
struct ChatMessage {
17+
std::string role;
18+
std::string content;
19+
};
20+
21+
struct ChatConversation {
22+
std::vector<ChatMessage> messages;
23+
std::string bos_token;
24+
std::string eos_token;
25+
bool add_generation_prompt = true;
26+
// Injected as the `date_string` template variable. Defaults to the fixed
27+
// date used by HuggingFace/vLLM template fixtures so output stays
28+
// deterministic; callers may override it.
29+
std::string date_string = "26 Jul 2024";
30+
};
31+
32+
} // namespace executorch::extension::llm

0 commit comments

Comments
 (0)