Skip to content

changelog : llama-server REST API #9291

Open

Description

@ggerganov

Overview

This is a list of changes to the public HTTP interface of the llama-server example. Collaborators are encouraged to edit this post in order to reflect important changes to the API that end up merged into the master branch.

If you are building a 3rd party project that relies on llama-server, it is recommended to follow this issue and check it carefully before upgrading to new versions.

See also:

Recent API changes (most recent at the top)

version PR desc
b9358 #23778 The env names of some CLI args have been updated
b7231 #17668 Default model name removed; "model" value from request no longer be reflected back
b7218 #17470 Add model load / unload endpoints
b7170 #17524 Enable --jinja by default
b6937 #16943 Add "model_alias" to /props endpoint
b6890 #16818 /metrics: Rename llamacpp:n_past_max -> llamacpp:n_tokens_max
b6523 #16109 In stream mode, error events are now OAI-compatible
b6508 #16052 include usage statistics only when stream_options.include_usage is specified
b6399 #15827 added return_progress and timings.cache_n
b6243 #15108 Add multimodal support to completions and embeddings endpoints
b6205 #15416 Disable context shift by default
b5441 #13660 Remove /metrics fields related to KV cache tokens and cells`
b5441 #13660 Remove /metrics fields related to KV cache tokens and cells`
b5223 #13174 For chat competion, if last message is assistant, it will be a prefilled message
b4599 #9639 /v1/chat/completions now supports tools & tool_choice
TBD. #10974 /v1/completions is now OAI-compat
TBD. #10783 logprobs is now OAI-compat, default to pre-sampling probs
TBD. #10861 /embeddings supports pooling type none
TBD. #10853 Add optional "tokens" output to /completions endpoint
b4337 #10803 Remove penalize_nl
b4265 #10626 CPU docker images working directory changed to /app
b4285 #10691 (Again) Change /slots and /props responses
b4283 #10704 Change /slots and /props responses
b4027 #10162 /slots endpoint: remove slot[i].state, add slot[i].is_processing
b3912 #9865 Add option to time limit the generation phase
b3911 #9860 Remove self-extend support
b3910 #9857 Remove legacy system prompt support
b3897 #9776 Change default security settings, /slots is now disabled by default
Endpoints now check for API key if it's set
b3887 #9510 Add /rerank endpoint
b3754 #9459 Add [DONE]\n\n in OAI stream response to match spec
b3721 #9398 Add seed_cur to completion response
b3683 #9308 Environment variable updated
b3599 #9056 Change /health and /slots

For older changes, use:

git log --oneline -p b3599 -- examples/server/README.md

Upcoming API changes

  • TBD

Activity

  1. pinned this issue on Sep 3, 2024
  2. ngxson commented on Sep 7, 2024

    @ngxson
    Collaborator

    Not a REST API breaking change, but is server-related: some environment variables are changed in #9308

  3. slaren commented on Sep 13, 2024

    @slaren
    Member

    After #9398, in the completion response seed contains the seed requested by the user, while seed_cur contains the seed used to generate the completion. The values can be different if seed is LLAMA_DEFAULT_SEED (or -1), in which case a random seed is generated and returned in seed_cur.

  4. ngxson commented on Oct 8, 2024

    @ngxson
    Collaborator

    Breaking change #9776 : better security control for public deployments

    • /slots endpoint is now disabled by default, start server with --slots to enable it
    • If an API key is set, all endpoints (including /slots and /props) requires a correct API key to access.
      Note: Only /health and /models are always publicly accessible
    • Setting "system_prompt" is removed from /completions endpoint. It is now moved to POST /props (see documentation)

    Please note that GET /props is always enabled to avoid breaking the web UI.

  5. ngxson commented on Nov 4, 2024

    @ngxson
    Collaborator

    Breaking change for /slots endpoint #10162

    slot[i].state is removed and replaced by slot[i].is_processing

    slot[i].is_processing === false means the slot is idle

  6. isaac-mcfadyen commented on Nov 4, 2024

    @isaac-mcfadyen
    Contributor

    Breaking change for /slots endpoint #10162

    slot[i].state is removed and replaced by slot[i].is_processing

    slot[i].is_processing === false means the slot is idle

    Was the slots endpoint also disabled by default? (or maybe just a documentation change?)
    https://github.com/ggerganov/llama.cpp/pull/10162/files#diff-42ce5869652f266b01a5b5bc95f4d945db304ce54545e2d0c017886a7f1cee1aR698

  7. ngxson commented on Nov 5, 2024

    @ngxson
    Collaborator

    For security reasons, "/slots" was disabled by default since #9776 , and was mentioned in the breaking changes table. I just forgot to update the docs.

  8. ngxson commented on Nov 7, 2024

    @ngxson
    Collaborator

    Not an API change, but maybe good to know that the default web UI for llama-server changed in #10175

    If you want to use the old completion UI, please follow instruction in the PR.

  9. ggerganov commented on Nov 25, 2024

    @ggerganov
    MemberAuthor

    cache_prompt: true is now used by default (#10501)

  10. ngxson commented on Dec 7, 2024

    @ngxson
    Collaborator

    /propsand /slots endpoints has changed in #10691 and #10704 , see server/README.md for more

  11. ngxson commented on Dec 18, 2024

    @ngxson
    Collaborator

    /embeddings will NOT be OAI-compat after #10861

    For clarification, we will maintain OAI-compat for all API under /v1 prefix, including:

    • /v1/embeddings
    • /v1/chat/completions

    NOTE: OAI support for /v1/completions will come in the near future

  12. ngxson commented on Dec 19, 2024

    @ngxson
    Collaborator

    Behavior of n_probs has changed in #10783 , we're now providing OAI-compatible logprobs option

  13. ngxson commented on Dec 31, 2024

    @ngxson
    Collaborator

    Added OAI-compat support for /v1/completions here: #10974

    If you want to use it with downstream library, be sure to add /v1 prefix. For example, using python library:

    from openai import OpenAI
    
    client = OpenAI(api_key="dummy", base_url=f"http://localhost:8080/v1")
    res = client.completions.create(
        model="davinci-002",
        prompt="I believe the meaning of life is",
        max_tokens=8,
    )

    If you want to use the old non-OAI style, remove the /v1 from endpoint path.

  14. 1 remaining item

  15. github-actions commented on Mar 17, 2025

    @github-actions
    Contributor

    This issue was closed because it has been inactive for 14 days since being marked as stale.

  16. isaac-mcfadyen commented on Mar 17, 2025

    @isaac-mcfadyen
    Contributor

    Guessing we want to keep this open.

  17. added
    roadmapPart of a roadmap project
    and removed on Mar 17, 2025
  18. aviallon commented on May 11, 2025

    @aviallon
    Contributor

    We need to add the new vision support

  19. Ifiht commented on Jun 28, 2025

    @Ifiht

    Along with the vision support, are there currently plans to add the MCP tool-calling function? Or maybe it's already here & I just don't know where to look?
    OpenAI MCP Ref

  20. 65a commented on Aug 28, 2025

    @65a
    Contributor

    Wasn't sure how to add to the API changelog...

    #15108 is a forward compatible API change to support multimodal in the embeddings and completion endpoints. It also adds the capability "multimodal" to the models endpoint when llama-server was started with a multimodal-capable model and mmproj.

  21. ngxson commented on Sep 6, 2025

    @ngxson
    Collaborator

    #15827 : added return_progress and timings.cache_n, useful for tracking prompt processing progress and context length usage

  22. Cyrille37 commented on May 1, 2026

    @Cyrille37

    Hi. I hope I'm in the right place for talking about #9639 :-)

    In server/server-chat.cpp# the tool_choice is forced to String as "auto" or "required". That generates a log "Wrong type supplied for parameter 'tool_choice'. Expected 'string', using default value: [json.exception.type_error.302] type must be string, but is object".

    • Is there a reason for this forced type casting ? Perhaps a standby while the full OpenAI specification will be implemented ?
    • Would the log be replaced from "Wrong" to "Notice" or "Info" ?

    Regards & cheers.

  23. fatinm1 commented on Sep 6, 2026

    @fatinm1

    A few things I noticed that aren't reflected in the table or the "Upcoming API changes" section yet:

    -    Upcoming: #26508 added a runtime warning that the default server port will change from :8080 to :9931 in a future release —     seems worth listing under "Upcoming API changes" rather than leaving it as TBD.
    -    #26347 made the /models and /props endpoints require the API key when authentication is enabled (previously accessible regardless).
    -    #26045 added support for "reasoning_effort": "none" in the OAI-compatible chat API.
    
  24. bappisinsane commented on Oct 10, 2026

    @bappisinsane

    Proposed changelog update

    The following entries should be added to the table to reflect recent API changes since the last update:

    version PR desc
    v0.6.0 #29997 New /v1/systemone endpoint for decision models; /v1/embeddings accepts typed vision/audio/video content; GET /models reports input/output modalities; server default port changed to 9931

    Key changes in this release:

    Please apply at your convenience. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationroadmapPart of a roadmap project

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions