Skip to content

llm: skip quantized kv cache for models that don't support it - #16955

Open
tenderdeve wants to merge 1 commit into
ollama:mainfrom
tenderdeve:fix/16946-gptoss-kv-quant
Open

tenderdeve wants to merge 1 commit into
ollama:mainfrom
tenderdeve:fix/16946-gptoss-kv-quant

Conversation

@tenderdeve

Copy link
Copy Markdown

Fixes #16946

Serving gpt-oss with OLLAMA_KV_CACHE_TYPE=q8_0 core dumps llama-server. gpt-oss uses attention with sinks, which does not support a quantized KV cache.

A guard for this used to live in GGML.SupportsKVCacheType, but that function lost all of its callers and the gpt-oss branch was removed in 19e6796 (assuming a GGML update made it safe). The requested cache type now flows straight to llama-server as --cache-type-k/--cache-type-v unchecked, so it crashes. This is a regression — the reporter confirms it works on 0.24.0 and fails on 0.30.x.

This restores the gpt-oss guard, moves SupportsKVCacheType onto KV, and consults it when building the llama-server command: if the model doesn't support the requested cache type, fall back to the default instead of passing an unsupported one.

Added a unit test covering gpt-oss (quantized rejected, f16/unset allowed) and other architectures (q8_0/q4_0 allowed).

gpt-oss uses attention with sinks, which crashes llama-server when served
with a quantized KV cache (OLLAMA_KV_CACHE_TYPE=q8_0). A guard for this
lived in GGML.SupportsKVCacheType, but that function lost all of its
callers and the gpt-oss branch was later removed in 19e6796, so the
requested cache type now reaches llama-server unchecked and core dumps.

Restore the gpt-oss guard, move SupportsKVCacheType onto KV, and consult
it when building the llama-server command: if the model does not support
the requested cache type, fall back to the default instead of passing an
unsupported one.

Fixes ollama#16946

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

llama-server core dumps when serving GPT-OSS with OLLAMA_KV_CACHE_TYPE=q8_0

1 participant