Repository navigation
Open
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationroadmapPart of a roadmap projectPart of a roadmap project
Description
Overview
This is a list of changes to the public HTTP interface of the llama-server example. Collaborators are encouraged to edit this post in order to reflect important changes to the API that end up merged into the master branch.
If you are building a 3rd party project that relies on llama-server, it is recommended to follow this issue and check it carefully before upgrading to new versions.
See also:
Recent API changes (most recent at the top)
| version | PR | desc |
|---|---|---|
| b9358 | #23778 | The env names of some CLI args have been updated |
| b7231 | #17668 | Default model name removed; "model" value from request no longer be reflected back |
| b7218 | #17470 | Add model load / unload endpoints |
| b7170 | #17524 | Enable --jinja by default |
| b6937 | #16943 | Add "model_alias" to /props endpoint |
| b6890 | #16818 | /metrics: Rename llamacpp:n_past_max -> llamacpp:n_tokens_max |
| b6523 | #16109 | In stream mode, error events are now OAI-compatible |
| b6508 | #16052 | include usage statistics only when stream_options.include_usage is specified |
| b6399 | #15827 | added return_progress and timings.cache_n |
| b6243 | #15108 | Add multimodal support to completions and embeddings endpoints |
| b6205 | #15416 | Disable context shift by default |
| b5441 | #13660 | Remove /metrics fields related to KV cache tokens and cells` |
| b5441 | #13660 | Remove /metrics fields related to KV cache tokens and cells` |
| b5223 | #13174 | For chat competion, if last message is assistant, it will be a prefilled message |
| b4599 | #9639 | /v1/chat/completions now supports tools & tool_choice |
| TBD. | #10974 | /v1/completions is now OAI-compat |
| TBD. | #10783 | logprobs is now OAI-compat, default to pre-sampling probs |
| TBD. | #10861 | /embeddings supports pooling type none |
| TBD. | #10853 | Add optional "tokens" output to /completions endpoint |
| b4337 | #10803 | Remove penalize_nl |
| b4265 | #10626 | CPU docker images working directory changed to /app |
| b4285 | #10691 | (Again) Change /slots and /props responses |
| b4283 | #10704 | Change /slots and /props responses |
| b4027 | #10162 | /slots endpoint: remove slot[i].state, add slot[i].is_processing |
| b3912 | #9865 | Add option to time limit the generation phase |
| b3911 | #9860 | Remove self-extend support |
| b3910 | #9857 | Remove legacy system prompt support |
| b3897 | #9776 | Change default security settings, /slots is now disabled by defaultEndpoints now check for API key if it's set |
| b3887 | #9510 | Add /rerank endpoint |
| b3754 | #9459 | Add [DONE]\n\n in OAI stream response to match spec |
| b3721 | #9398 | Add seed_cur to completion response |
| b3683 | #9308 | Environment variable updated |
| b3599 | #9056 | Change /health and /slots |
For older changes, use:
git log --oneline -p b3599 -- examples/server/README.mdUpcoming API changes
- TBD
Reacted by vb, Xuan-Son Nguyen, Vladimir Sotnikov, Akarshan Biswas, Erik Scholz, Isaac McFadyen, lin72h, PierreCarceller, chen, Andrii Podanenko and 10 more
Reacted by Gaël James
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationroadmapPart of a roadmap projectPart of a roadmap project
Activity
ngxson commentedon Sep 7, 2024
Not a REST API breaking change, but is server-related: some environment variables are changed in #9308
slaren commentedon Sep 13, 2024
After #9398, in the completion response
seedcontains the seed requested by the user, whileseed_curcontains the seed used to generate the completion. The values can be different ifseedisLLAMA_DEFAULT_SEED(or -1), in which case a random seed is generated and returned inseed_cur.ngxson commentedon Oct 8, 2024
Breaking change #9776 : better security control for public deployments
/slotsendpoint is now disabled by default, start server with--slotsto enable it/slotsand/props) requires a correct API key to access.Note: Only
/healthand/modelsare always publicly accessible"system_prompt"is removed from/completionsendpoint. It is now moved to POST/props(see documentation)Please note that GET
/propsis always enabled to avoid breaking the web UI.ngxson commentedon Nov 4, 2024
Breaking change for
/slotsendpoint #10162slot[i].stateis removed and replaced byslot[i].is_processingslot[i].is_processing === falsemeans the slot is idleisaac-mcfadyen commentedon Nov 4, 2024
Was the
slotsendpoint also disabled by default? (or maybe just a documentation change?)https://github.com/ggerganov/llama.cpp/pull/10162/files#diff-42ce5869652f266b01a5b5bc95f4d945db304ce54545e2d0c017886a7f1cee1aR698
ngxson commentedon Nov 5, 2024
For security reasons, "/slots" was disabled by default since #9776 , and was mentioned in the breaking changes table. I just forgot to update the docs.
ngxson commentedon Nov 7, 2024
Not an API change, but maybe good to know that the default web UI for
llama-serverchanged in #10175If you want to use the old completion UI, please follow instruction in the PR.
ggerganov commentedon Nov 25, 2024
cache_prompt: trueis now used by default (#10501)ngxson commentedon Dec 7, 2024
/propsand/slotsendpoints has changed in #10691 and #10704 , see server/README.md for morengxson commentedon Dec 18, 2024
/embeddingswill NOT be OAI-compat after #10861For clarification, we will maintain OAI-compat for all API under
/v1prefix, including:/v1/embeddings/v1/chat/completionsNOTE: OAI support for
/v1/completionswill come in the near futurengxson commentedon Dec 19, 2024
Behavior of
n_probshas changed in #10783 , we're now providing OAI-compatiblelogprobsoptionngxson commentedon Dec 31, 2024
Added OAI-compat support for
/v1/completionshere: #10974If you want to use it with downstream library, be sure to add
/v1prefix. For example, using python library:If you want to use the old non-OAI style, remove the
/v1from endpoint path.1 remaining item
github-actions commentedon Mar 17, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale.
isaac-mcfadyen commentedon Mar 17, 2025
Guessing we want to keep this open.
aviallon commentedon May 11, 2025
We need to add the new vision support
Ifiht commentedon Jun 28, 2025
Along with the vision support, are there currently plans to add the MCP tool-calling function? Or maybe it's already here & I just don't know where to look?
OpenAI MCP Ref
65a commentedon Aug 28, 2025
Wasn't sure how to add to the API changelog...
#15108 is a forward compatible API change to support multimodal in the embeddings and completion endpoints. It also adds the capability "multimodal" to the models endpoint when llama-server was started with a multimodal-capable model and mmproj.
ngxson commentedon Sep 6, 2025
#15827 : added
return_progressandtimings.cache_n, useful for tracking prompt processing progress and context length usageCyrille37 commentedon May 1, 2026
Hi. I hope I'm in the right place for talking about #9639 :-)
In server/server-chat.cpp# the
tool_choiceis forced to String as "auto" or "required". That generates a log "Wrong type supplied for parameter 'tool_choice'. Expected 'string', using default value: [json.exception.type_error.302] type must be string, but is object".Regards & cheers.
docs(todo): record llama.cpp drift findings; mark CLI surface tasks s…
fatinm1 commentedon Sep 6, 2026
A few things I noticed that aren't reflected in the table or the "Upcoming API changes" section yet:
bappisinsane commentedon Oct 10, 2026
Proposed changelog update
The following entries should be added to the table to reflect recent API changes since the last update:
/v1/systemoneendpoint for decision models;/v1/embeddingsaccepts typed vision/audio/video content;GET /modelsreports input/output modalities; server default port changed to 9931Key changes in this release:
/v1/systemoneendpoint supporting five decision models: laya, julia-1, lev, openjev (+vision), kev (llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) #29818)/v1/embeddingsnow accepts typed vision/audio/video content via OpenAI-style content arrays (server : support typed content (vision/audio/video) input for /v1/embeddings endpoint #29556)GET /modelsreports model input/output modalities (common, server : report model input/output modalities in GET /models #29987)Please apply at your convenience. Thanks!