Repository navigation
Open
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationroadmapPart of a roadmap projectPart of a roadmap project
Description
Overview
This is a list of changes to the public interface of the llama library. Collaborators are encouraged to edit this post in order to reflect important changes to the API that end up merged into the master branch.
If you are building a 3rd party project that relies on libllama, it is recommended to follow this issue and check it before upgrading to new versions.
See also:
Recent API changes (most recent at the top)
| version | PR | desc |
|---|---|---|
| TBD | #32306 | Deprecate old llama_batch APIs (migrate to llama_batch_ext) |
| v1.0.0 | #30565 | Remove various deprecated APIs |
| b11223 | #28876 | Add llama_get_causal_attn |
| b11019 | #28993 | Add llama_adapter_lora_init_from_file_ptr |
| b10875 | #28334 | Remove CLI flags --mmap, --mlock, --direct-io in favor for --load-mode flag |
| b10273 | #26524 | Remove n_ctx_train from llama_sampler_init_dry, -1 no longer means full-context for llama_sampler_init_penalties and llama_sampler_init_dry |
| b10258 | #26520 | Move n_vocab from llama_sampler_data to llama_sampler_init_penalties |
| b10242 | #25262 | Added n_vocab to llama_sampler_data |
| b10206 | #26296 | Update llama_model_params - new load_mtp flag |
| b10145 | #26135 | Update load mode API - add LLAMA_LOAD_MODE_MMAP_MLOCK |
| b10105 | #20834 | Deprecate --mmap/mlock/dio flags for --load-mode flag |
| b9460 | #23861 | Update llama_context_params (add n_outputs_max) |
| b8608 | #20346 | Update llama_model_quantize_params |
| b8049 | #19280 | Update llama_*_adapter_lora() API |
| b7672 | #18390 | Memory margin per device for llama_params_fit |
| b7668 | #18166 | Update llama_model_params - new use_direct_io flag |
| b7639 | #18607 | Add llama_model_n_embd_out |
| b7628 | #17004 | Add backend sampling API |
| b7551 | #18374 | Return an enum instead of a boolean to indicate teh result of llama_params_fit |
| b7407 | #16653 | Add flag no_alloc to llama_model_param, no change with llama_model_default_params |
| b6976 | #16928 | Add llama_model_n_embd_inp |
| TBD | #15665 | Remove llama_sampler_init_softmax() + dist sampler no longer implicitly sorts |
| b6239 | #15472 | Remove llama_kv_self_... API |
| b6157 | #15293 | Add llama_state_seq_..._ext API |
| b5913 | #14363 | Update llama_context_params - add bool kv_unified |
| b5740 | #13037 | Update llama_model_quantize_params |
| b5870 | #14631 | Remove enum llama_vocab_pre_type |
| b5435 | #13653 | Remove llama_kv_cache_view_* API |
| b5429 | #13194 | Update llama_context_params - add bool swa_full |
| b5311 | #13284 | Update llama_context_params - remove logits_all + rearrange flags |
| b5125 | #12511 | Update llama_model_quantize_params |
| b5028 | #11397 | Update llama_model_params |
| b4882 | #12181 | Change llama_kv_cache_... -> llama_kv_self_... |
| b4599 | #9639 | Add llama_sampler_init_grammar_lazy to support lazy grammars w/ trigger words & tokens |
| b4524 | #11016 | Add name parameter to llama_model_chat_template (uses default template if NULL) |
| b4501 | #11262 | Remove rpc_servers from llama_model and llama_model_params |
| b4464 | #11110 | Add llama_vocab and rename various structs and calls |
| b4424 | #11063 | Update llama_model API naming |
| b4357 | #10784 | Remove llama_model_get_tensor() |
| b4337 | #10803 | Change llama_sampler_init_penalties() |
| b4282 | #10446 | Remove support for Q4_0_N_M model files in favor of automatic repacking of Q4_0 |
| b4167 | #10497 | Add devices to llama_model_params |
| b3948 | #9897 | Deprecate softmax sampler and update dist sampler` |
| b3988 | #10071 | Remove Tail-Free sampling |
| b3943 | #9745 | Remove all_pos_0, all_pos_1, all_seq_id from llama_batch |
| b3908 | #9798 | Update FIM-related API |
| b3841 | #9510 | Add LLAMA_POOLING_TYPE_RANK |
| b3774 | #9512 | Add llama_n_head() |
| b3750 | #9355 | Add llama_perf API + param to disable internal profiling |
| b3749 | #9445 | Add llama_sampler_chain_remove() |
| b3681 | #9294 | Major changes to the sampling API (see PR for more info) |
| b3651 | #8980 | Add LLAMA_VOCAB_TYPE_RWKV enum value |
| b3644 | #8672 | Add llama_threadpool API + change uint32_t -> int32_t |
| b3614 | #8526 | Add llama_model_is_recurrent |
For older changes, use:
git log --oneline -p b3614 -- include/llama.h(For collaborators) To link between PR number vs Build number:
git log --oneline | tail -r | nlUpcoming API changes
- TBD
Reacted by Clint Herron, Ettore Di Giacinto, lin72h, Djip007, rainy liu, carlsonlin, Erik Scholz, Art39printnew and hydai
Reacted by Ettore Di Giacinto, Archit Gupta, lin72h, Britt and hydai
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationroadmapPart of a roadmap projectPart of a roadmap project
Activity
ggerganov commentedon Sep 13, 2024
#9355 restores the functionality for getting performance measurements from within
libllama(which was removed in #9294) via a newllama_perfAPI. Thellama_context_paramsis extended with a newbool no_perfparameter that can be used to disable the internal timings duringlibllamacompute.ggml : add support for dynamic loading of backends (#10469)
ddh0 commentedon Jan 3, 2025
Looks like
llama_model_get_tensorwas removed from the API but that change was not documented hereggerganov commentedon Jan 4, 2025
I didn't expect that this function is being used by anyone, so I skipped updating the changelog. It's updated now.
Btw, what do you use this call for?
ddh0 commentedon Jan 5, 2025
I don't use it personally but the function was included in my Python code, I started to get
ctypes"symbol not found" errors and I had to do some digging to figure out why. No worries!ggerganov commentedon Jan 12, 2025
Significant updates to the public API - added
struct llama_vocaband applied multiple naming changes for better consistency: #11110. Update of user code should be relatively easy - mostly changing functions to use the new names and callingllama_model_get_vocab(model)where the old API requiredllama_modeland the new API requiresllama_vocab.ngxson commentedon Apr 7, 2025
Not actually related to the core library, but we're turn on
LLAMA_CURLby default, link to the PR here. If you're building llama.cpp in your project just to get thelibllama, make sure to disable eitherLLAMA_CURLorLLAMA_BUILD_COMMONEAddario commentedon Apr 8, 2025
PR #12511 changed the llama_model_quantize_params API by introducing an additional void * tensor_types parameter
ddh0 commentedon Apr 9, 2025
PR #11397 added
llama_model_tensor_buft_overridetollama_model_params, causing a segmentation fault when trying to load a llama model due to this line in src/llama-model.cpp:Because I did not change the way I was loading the model after that PR, I got a segfault because I didn't include the new
llama_model_tensor_buft_overridein myllama_model_params.ggerganov commentedon Apr 9, 2025
@ddh0 We have missed to document this change. I added an entry to the table above. Thanks.
Art39print commentedon Jul 14, 2025
ddh0 commentedon Oct 3, 2025
PR #16382 updated the
libllamainterface to addLLAMA_API bool llama_model_is_hybrid(...).Gadflyii commentedon Oct 6, 2025
PR #16310 added new param "--no-host" to disable host buffer to allow extra buffers (Repack + AMX) to enable AMX acceleration on CPU layers when GPU(s) is present.
fatinm1 commentedon Sep 5, 2026
Noticed two public API changes since the last table entry (b10273 / #26524) that aren't reflected yet:
bappisinsane commentedon Oct 10, 2026
Proposed changelog update
The following entries should be added to the table to reflect recent API changes since the last update:
llama_batch_extbatch API withllama_embd,llama_process(),llama_process_type; newllama_prefetch_rows()for MADVISE-based prefetching of PLE tensors; session formats bumped toLLAMA_SESSION_VERSION11 andLLAMA_STATE_SEQ_VERSION4Key changes in this release:
include/llama.h: newllama_batch_extbatch API withllama_embd,llama_process()andllama_process_type(llama: add llama_batch_ext #24669), newllama_get_causal_attn()(server : allow RANK pooling batch splitting for causal LLM rerankers (ie. Qwen3 and Qwen3-VL) #28876), session formats bumped toLLAMA_SESSION_VERSION11 andLLAMA_STATE_SEQ_VERSION4include/llama-cpp.h: addedllama_batch_ext_ptrand deleter for the new extended batch API (llama: add llama_batch_ext #24669)tools/mtmd/mtmd.h:mtmd_get_memory_usage()now returns anmtmd_memory_usagestruct withimage_max_tokensanduse_non_causal(mtmd: cap max_image to ubatch for non_causal models #29773)llama_prefetch_rows()using MADVISE-based prefetching of PLE tensors in Qwen4Exp and Gemma4 (llama: llama_prefetch_rows #29599)llama_batch_ext(llama: support both embd + raw tokens in batch #29622)llama_batch_extAPI (batch: migrate speculative, mtmd and server to batch_ext #29385, batch: migrate the rest of examples to llama_batch_ext #29601)Please apply at your convenience. Thanks!
tmwally80-sys commentedon Nov 1, 2026