Repository navigation
Commit 7141b95
Add Jinja2Cpp-based chat template formatter library (#19533)
Summary:
Foundation PR for the chat-template support stack split out of #16987 per reviewer request from kirklandsign. This PR adds the Jinja2Cpp-based `JinjaChatFormatter`, supporting chat-types, embedded Llama3 / Llama3.2 / Gemma3 templates, build glue (CMake/Buck), and a focused C++ unit-test suite.
This PR is **reviewable in isolation** — it has no behavior change for any existing runner; downstream PRs (B/C/D) plug it in.
## Stack overview
| PR | Subject |
|---|---|
| **1/4 (this PR)** | Library + tests |
| 2/4 | TextLLMRunner echo-gated special-token filter + EOS merge |
| 3/4 | Python bindings + Python LlamaRunner integration |
| 4/4 | llama_main CLI flags + chat_formatter wrapper + universal Jinja docs |
## What this PR adds
- **`extension/llm/chat_template/{chat_templates.h, BUCK, CMakeLists.txt, targets.bzl}`** — embedded Llama3 / Llama3.2 / Gemma3 templates and the `ChatTemplateType` enum + `ModelTokens`. The CMake file `FetchContent`s Jinja2Cpp 1.3.2, with `SUPPORT_REGEX_LOOKAHEAD` set **before** `FetchContent_MakeAvailable` so it propagates correctly, plus header staging for `nonstd` headers that some Jinja2Cpp installations omit. Installs `chat_templates.h` so SDK consumers can include it.
- **`extension/llm/runner/{chat_types.h, jinja_chat_formatter.{h,cpp}}`** — the **Universal Jinja chat formatter** that supports any HuggingFace / vLLM chat template, not just the embedded ones. Loadable via `fromTemplate` (built-in), `fromString` (any string), or `fromFile` (any `.jinja` file). `formatConversation` injects vLLM/HuggingFace-standard params (`tools=[]`, `tool_choice=None`, `date_string`, `chat_template_kwargs`) so any template that references those variables renders correctly.
- `normalizeTemplate` handles vLLM/HF template quirks for Jinja2Cpp: notably, `not tools is none` maps to `tools` (truthy check), preserving the intent of `tools is not none` for empty-list defaults.
- **`extension/llm/runner/{CMakeLists.txt, targets.bzl}`** — link `extension_llm_runner` against `jinja2cpp` (PRIVATE) and define `EXECUTORCH_USE_JINJA2CPP`.
- **`extension/llm/runner/test/test_jinja_chat_formatter.cpp`** + test build files — unit tests covering Llama3 / Llama3.2 / Gemma3 embedded templates, `parseChatTemplateType` (case-insensitive), and **three universal-Jinja regression tests**:
- generic HuggingFace-style template (proves it's not Llama-specific)
- tools-aware template (validates the `tools=[]` default)
- `not tools is none` normalization regression test
- **`CMakeLists.txt`** — adds `add_subdirectory(extension/llm/chat_template)` guarded by `EXECUTORCH_BUILD_EXTENSION_LLM_RUNNER`.
- **`shim_et/xplat/executorch/build/build_variables.bzl`** — adds `jinja_chat_formatter.cpp` to the runner sources.
## Universal Jinja support
Any HuggingFace / vLLM-style Jinja template works:
```cpp
// From a template file (HuggingFace tokenizer_config.json, vLLM examples)
auto formatter = JinjaChatFormatter::fromFile("path/to/template.jinja");
// From a string
auto formatter = JinjaChatFormatter::fromString(template_str);
// Built-in shortcuts
auto formatter = JinjaChatFormatter::fromTemplate(ChatTemplateType::Llama3);
```
## Notes
- **No behavior change** for existing TextLLMRunner / MultimodalRunner users: the formatter is opt-in, only invoked when downstream code calls it.
- **Sample vLLM templates are NOT checked in** (per reviewer feedback metascroy). Documentation in the follow-up CLI PR (4/4) points users to vLLM's `examples/` directory and HuggingFace `tokenizer_config.json` files.
## `messages[1:]` slicing
Jinja2Cpp cannot parse Python-style slice syntax, so templates that do `{% set messages = messages[1:] %}` are normalized to a `_executorch_messages_tail(...)` call registered through Jinja2Cpp's `UserCallable`. Because Jinja2Cpp represents strings as non-owning `string_view`s, `formatConversation` materializes the sliced messages into a render-lifetime owning store, so the values stay valid across repeated `messages[1:]` reassignments.
Test Plan:
- [x] Build with `cmake --workflow llm-release`
- [x] Build with `make llama-cpu`
- [x] Run unit tests: `extension/llm/runner/test/test_jinja_chat_formatter`
- [x] Verify embedded templates render correctly for Llama3 / Llama3.2 / Gemma3
- [x] Verify universal Jinja support with HuggingFace tokenizer_config.json templates
## Original PR
Splitting #16987 into 4 reviewable PRs.
cc kirklandsign larryliu0820 metascroy lucylq mergennachin
## Internal build & test (Buck / ASAN)
- [x] `buck2 build fbcode//executorch/extension/llm/runner:runner_lib` — builds clean
- [x] `buck2 test fbcode//executorch/extension/llm/runner/test:test_jinja_chat_formatter` — 27/27 pass under ASAN (embedded Llama3 / Llama3.2 / Gemma3 templates, the universal-Jinja regression tests, and the repeated `messages[1:]` slice lifetime test)
Differential Revision: D106817172
Pulled By: seyeong-han1 parent 0e6b67e commit 7141b95
15 files changed
Lines changed: 1161 additions & 20 deletions
File tree
- extension/llm
- chat_template
- runner
- test
- shim_et/xplat/executorch/build
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
830 | 830 | | |
831 | 831 | | |
832 | 832 | | |
833 | | - | |
| 833 | + | |
834 | 834 | | |
835 | 835 | | |
836 | 836 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
57 | 62 | | |
58 | 63 | | |
59 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
0 commit comments