Skip to content

changelog : libllama API #9289

Open
Open

Description

@ggerganov

Overview

This is a list of changes to the public interface of the llama library. Collaborators are encouraged to edit this post in order to reflect important changes to the API that end up merged into the master branch.

If you are building a 3rd party project that relies on libllama, it is recommended to follow this issue and check it before upgrading to new versions.

See also:

Recent API changes (most recent at the top)

version PR desc
TBD #32306 Deprecate old llama_batch APIs (migrate to llama_batch_ext)
v1.0.0 #30565 Remove various deprecated APIs
b11223 #28876 Add llama_get_causal_attn
b11019 #28993 Add llama_adapter_lora_init_from_file_ptr
b10875 #28334 Remove CLI flags --mmap, --mlock, --direct-io in favor for --load-mode flag
b10273 #26524 Remove n_ctx_train from llama_sampler_init_dry, -1 no longer means full-context for llama_sampler_init_penalties and llama_sampler_init_dry
b10258 #26520 Move n_vocab from llama_sampler_data to llama_sampler_init_penalties
b10242 #25262 Added n_vocab to llama_sampler_data
b10206 #26296 Update llama_model_params - new load_mtp flag
b10145 #26135 Update load mode API - add LLAMA_LOAD_MODE_MMAP_MLOCK
b10105 #20834 Deprecate --mmap/mlock/dio flags for --load-mode flag
b9460 #23861 Update llama_context_params (add n_outputs_max)
b8608 #20346 Update llama_model_quantize_params
b8049 #19280 Update llama_*_adapter_lora() API
b7672 #18390 Memory margin per device for llama_params_fit
b7668 #18166 Update llama_model_params - new use_direct_io flag
b7639 #18607 Add llama_model_n_embd_out
b7628 #17004 Add backend sampling API
b7551 #18374 Return an enum instead of a boolean to indicate teh result of llama_params_fit
b7407 #16653 Add flag no_alloc to llama_model_param, no change with llama_model_default_params
b6976 #16928 Add llama_model_n_embd_inp
TBD #15665 Remove llama_sampler_init_softmax() + dist sampler no longer implicitly sorts
b6239 #15472 Remove llama_kv_self_... API
b6157 #15293 Add llama_state_seq_..._ext API
b5913 #14363 Update llama_context_params - add bool kv_unified
b5740 #13037 Update llama_model_quantize_params
b5870 #14631 Remove enum llama_vocab_pre_type
b5435 #13653 Remove llama_kv_cache_view_* API
b5429 #13194 Update llama_context_params - add bool swa_full
b5311 #13284 Update llama_context_params - remove logits_all + rearrange flags
b5125 #12511 Update llama_model_quantize_params
b5028 #11397 Update llama_model_params
b4882 #12181 Change llama_kv_cache_... -> llama_kv_self_...
b4599 #9639 Add llama_sampler_init_grammar_lazy to support lazy grammars w/ trigger words & tokens
b4524 #11016 Add name parameter to llama_model_chat_template (uses default template if NULL)
b4501 #11262 Remove rpc_servers from llama_model and llama_model_params
b4464 #11110 Add llama_vocab and rename various structs and calls
b4424 #11063 Update llama_model API naming
b4357 #10784 Remove llama_model_get_tensor()
b4337 #10803 Change llama_sampler_init_penalties()
b4282 #10446 Remove support for Q4_0_N_M model files in favor of automatic repacking of Q4_0
b4167 #10497 Add devices to llama_model_params
b3948 #9897 Deprecate softmax sampler and update dist sampler`
b3988 #10071 Remove Tail-Free sampling
b3943 #9745 Remove all_pos_0, all_pos_1, all_seq_id from llama_batch
b3908 #9798 Update FIM-related API
b3841 #9510 Add LLAMA_POOLING_TYPE_RANK
b3774 #9512 Add llama_n_head()
b3750 #9355 Add llama_perf API + param to disable internal profiling
b3749 #9445 Add llama_sampler_chain_remove()
b3681 #9294 Major changes to the sampling API (see PR for more info)
b3651 #8980 Add LLAMA_VOCAB_TYPE_RWKV enum value
b3644 #8672 Add llama_threadpool API + change uint32_t -> int32_t
b3614 #8526 Add llama_model_is_recurrent

For older changes, use:

git log --oneline -p b3614 -- include/llama.h

(For collaborators) To link between PR number vs Build number:

git log --oneline | tail -r | nl

Upcoming API changes

  • TBD

Activity

  1. pinned this issue on Sep 3, 2024
  2. ggerganov commented on Sep 13, 2024

    @ggerganov
    MemberAuthor

    #9355 restores the functionality for getting performance measurements from within libllama (which was removed in #9294) via a new llama_perf API. The llama_context_params is extended with a new bool no_perf parameter that can be used to disable the internal timings during libllama compute.

  3. ddh0 commented on Jan 3, 2025

    @ddh0
    Contributor

    Looks like llama_model_get_tensor was removed from the API but that change was not documented here

  4. ggerganov commented on Jan 4, 2025

    @ggerganov
    MemberAuthor

    Looks like llama_model_get_tensor was removed from the API but that change was not documented here

    I didn't expect that this function is being used by anyone, so I skipped updating the changelog. It's updated now.

    Btw, what do you use this call for?

  5. ddh0 commented on Jan 5, 2025

    @ddh0
    Contributor

    I didn't expect that this function is being used by anyone, so I skipped updating the changelog. It's updated now.

    Btw, what do you use this call for?

    I don't use it personally but the function was included in my Python code, I started to get ctypes "symbol not found" errors and I had to do some digging to figure out why. No worries!

  6. ggerganov commented on Jan 12, 2025

    @ggerganov
    MemberAuthor

    Significant updates to the public API - added struct llama_vocab and applied multiple naming changes for better consistency: #11110. Update of user code should be relatively easy - mostly changing functions to use the new names and calling llama_model_get_vocab(model) where the old API required llama_model and the new API requires llama_vocab.

  7. ngxson commented on Apr 7, 2025

    @ngxson
    Collaborator

    Not actually related to the core library, but we're turn on LLAMA_CURL by default, link to the PR here. If you're building llama.cpp in your project just to get the libllama, make sure to disable either LLAMA_CURL or LLAMA_BUILD_COMMON

  8. EAddario commented on Apr 8, 2025

    @EAddario
    Contributor

    PR #12511 changed the llama_model_quantize_params API by introducing an additional void * tensor_types parameter

  9. ddh0 commented on Apr 9, 2025

    @ddh0
    Contributor

    PR #11397 added llama_model_tensor_buft_override to llama_model_params, causing a segmentation fault when trying to load a llama model due to this line in src/llama-model.cpp:

        pimpl->has_tensor_overrides = params.tensor_buft_overrides && params.tensor_buft_overrides[0].pattern;
    

    Because I did not change the way I was loading the model after that PR, I got a segfault because I didn't include the new llama_model_tensor_buft_override in my llama_model_params.

  10. ggerganov commented on Apr 9, 2025

    @ggerganov
    MemberAuthor

    @ddh0 We have missed to document this change. I added an entry to the table above. Thanks.

  11. Art39print commented on Jul 14, 2025

    @Art39print
  12. ddh0 commented on Oct 3, 2025

    @ddh0
    Contributor

    PR #16382 updated the libllama interface to add LLAMA_API bool llama_model_is_hybrid(...).

  13. Gadflyii commented on Oct 6, 2025

    @Gadflyii
    Contributor

    PR #16310 added new param "--no-host" to disable host buffer to allow extra buffers (Repack + AMX) to enable AMX acceleration on CPU layers when GPU(s) is present.

  14. fatinm1 commented on Sep 5, 2026

    @fatinm1

    Noticed two public API changes since the last table entry (b10273 / #26524) that aren't reflected yet:

    - #26081 added LLAMA_LOAD_MODE_AUTO = -1 to enum llama_load_mode, making auto-detection (mmap unless a non-Metal iGPU is present) the new default load mode.
    
    - #27794 added a new public enum, enum llama_tensor_read_lazy (OFF / AUTO / ON), plus a corresponding tensor_read_lazy field on llama_model_params, enabling on-demand reading of large tensors marked by the model architecture.
    
  15. bappisinsane commented on Oct 10, 2026

    @bappisinsane

    Proposed changelog update

    The following entries should be added to the table to reflect recent API changes since the last update:

    version PR desc
    v0.6.0 #29997 New llama_batch_ext batch API with llama_embd, llama_process(), llama_process_type; new llama_prefetch_rows() for MADVISE-based prefetching of PLE tensors; session formats bumped to LLAMA_SESSION_VERSION 11 and LLAMA_STATE_SEQ_VERSION 4

    Key changes in this release:

    Please apply at your convenience. Thanks!

  16. tmwally80-sys commented on Nov 1, 2026

    @tmwally80-sys

    Overview

    This is a list of changes to the public interface of the llama library. Collaborators are encouraged to edit this post in order to reflect important changes to the API that end up merged into the master branch.

    If you are building a 3rd party project that relies on libllama, it is recommended to follow this issue and check it before upgrading to new versions.

    See also:

    Recent API changes (most recent at the top)

    version PR desc
    b11223 #28876 Add llama_get_causal_attn
    b11019 #28993 Add llama_adapter_lora_init_from_file_ptr
    b10875 #28334 Remove CLI flags --mmap, --mlock, --direct-io in favor for --load-mode flag
    b10273 #26524 Remove n_ctx_train from llama_sampler_init_dry, -1 no longer means full-context for llama_sampler_init_penalties and llama_sampler_init_dry
    b10258 #26520 Move n_vocab from llama_sampler_data to llama_sampler_init_penalties
    b10242 #25262 Added n_vocab to llama_sampler_data
    b10206 #26296 Update llama_model_params - new load_mtp flag
    b10145 #26135 Update load mode API - add LLAMA_LOAD_MODE_MMAP_MLOCK
    b10105 #20834 Deprecate --mmap/mlock/dio flags for --load-mode flag
    b9460 #23861 Update llama_context_params (add n_outputs_max)
    b8608 #20346 Update llama_model_quantize_params
    b8049 #19280 Update llama_*_adapter_lora() API
    b7672 #18390 Memory margin per device for llama_params_fit
    b7668 #18166 Update llama_model_params - new use_direct_io flag
    b7639 #18607 Add llama_model_n_embd_out
    b7628 #17004 Add backend sampling API
    b7551 #18374 Return an enum instead of a boolean to indicate teh result of llama_params_fit
    b7407 #16653 Add flag no_alloc to llama_model_param, no change with llama_model_default_params
    b6976 #16928 Add llama_model_n_embd_inp
    TBD #15665 Remove llama_sampler_init_softmax() + dist sampler no longer implicitly sorts
    b6239 #15472 Remove llama_kv_self_... API
    b6157 #15293 Add llama_state_seq_..._ext API
    b5913 #14363 Update llama_context_params - add bool kv_unified
    b5740 #13037 Update llama_model_quantize_params
    b5870 #14631 Remove enum llama_vocab_pre_type
    b5435 #13653 Remove llama_kv_cache_view_* API
    b5429 #13194 Update llama_context_params - add bool swa_full
    b5311 #13284 Update llama_context_params - remove logits_all + rearrange flags
    b5125 #12511 Update llama_model_quantize_params
    b5028 #11397 Update llama_model_params
    b4882 #12181 Change llama_kv_cache_... -> llama_kv_self_...
    b4599 #9639 Add llama_sampler_init_grammar_lazy to support lazy grammars w/ trigger words & tokens
    b4524 #11016 Add name parameter to llama_model_chat_template (uses default template if NULL)
    b4501 #11262 Remove rpc_servers from llama_model and llama_model_params
    b4464 #11110 Add llama_vocab and rename various structs and calls
    b4424 #11063 Update llama_model API naming
    b4357 #10784 Remove llama_model_get_tensor()
    b4337 #10803 Change llama_sampler_init_penalties()
    b4282 #10446 Remove support for Q4_0_N_M model files in favor of automatic repacking of Q4_0
    b4167 #10497 Add devices to llama_model_params
    b3948 #9897 Deprecate softmax sampler and update dist sampler`
    b3988 #10071 Remove Tail-Free sampling
    b3943 #9745 Remove all_pos_0, all_pos_1, all_seq_id from llama_batch
    b3908 #9798 Update FIM-related API
    b3841 #9510 Add LLAMA_POOLING_TYPE_RANK
    b3774 #9512 Add llama_n_head()
    b3750 #9355 Add llama_perf API + param to disable internal profiling
    b3749 #9445 Add llama_sampler_chain_remove()
    b3681 #9294 Major changes to the sampling API (see PR for more info)
    b3651 #8980 Add LLAMA_VOCAB_TYPE_RWKV enum value
    b3644 #8672 Add llama_threadpool API + change uint32_t -> int32_t
    b3614 #8526 Add llama_model_is_recurrent

    For older changes, use:

    git log --oneline -p b3614 -- include/llama.h

    (For collaborators) To link between PR number vs Build number:

    git log --oneline | tail -r | nl

    Upcoming API changes

    • TBD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationroadmapPart of a roadmap project

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions