server: allow assistant prefill when the prefill itself closes the <think> block #21889

Summary

When --reasoning-budget is non-zero, /v1/chat/completions unconditionally rejects any request where the last message has role: "assistant" (i.e. assistant prefill), returning:

HTTP 400: {"error": {"code": 400, "message": "Assistant response prefill is incompatible with enable_thinking."}}

This restriction is too broad. When the prefill content is itself a fully closed <think>...</think> block, there is no incompatibility: the user is explicitly telling the model that reasoning is complete and it should proceed directly to output. The server has no basis to refuse this.

We ran a 10-pair experiment on Qwen3.5-27B (distilled, i1-Q6_K, llama.cpp b8763) using the native /completion endpoint (which accepts the prefill) to validate the technique works correctly. Mean wall-time reduction: -8.64s per request. Mean tokens saved: 456 per request. Quality regressions: 0/10. The blocker is the /v1/chat/completions guard that prevents using this technique through the standard OpenAI-compatible path.

Current behaviour

Sending an assistant prefill via /v1/chat/completions while --reasoning-budget -1 is set:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-27b",
    "messages": [
      {"role": "user", "content": "Write a Python one-liner that reverses a list."},
      {"role": "assistant", "content": "<think>\nDirect answer follows.\n</think>\n\n"}
    ],
    "chat_template_kwargs": {"add_generation_prompt": false}
  }'

Response:

{"error": {"code": 400, "message": "Assistant response prefill is incompatible with enable_thinking."}}

The check is unconditional. It fires in tools/server/server-common.cpp at the point where prefill_assistant_message is true and inputs.enable_thinking is true, with no inspection of the prefill string:

// tools/server/server-common.cpp:1062-1063
if ( inputs.enable_thinking ) {
    throw std::invalid_argument("Assistant response prefill is incompatible with enable_thinking.");
}

Workaround that currently works

Using the native /completion endpoint with a manually serialized chat-template prompt:

curl -s http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "<|im_start|>user\nWrite a Python one-liner that reverses a list.<|im_end|>\n<|im_start|>assistant\n<think>\nDirect answer follows.\n</think>\n\n",
    "n_predict": 2000,
    "temperature": 0.6
  }'

This works and the model correctly outputs only the answer with no reasoning tokens. The problem is that /completion is not OpenAI-compatible: it returns no usage object, no finish_reason codes, and no reasoning_content field. Any production client built against the OpenAI API cannot use this path without a compatibility shim.

Proposed change

Skip the rejection when the prefill content ends with </think> (or, more robustly, when it contains a fully closed <think>...</think> block). The user is explicitly signaling that reasoning is pre-empted from their side.

Minimal version of the condition change in server-common.cpp:

if ( inputs.enable_thinking ) {
    // Allow prefill if it already closes the think block —
    // the user is explicitly signaling reasoning is done.
    const auto & prefill_content = last_message.content;
    const bool prefill_closes_think =
        prefill_content.find("</think>") != std::string::npos;
    if (!prefill_closes_think) {
        throw std::invalid_argument(
            "Assistant response prefill is incompatible with enable_thinking. "
            "Exception: a prefill that itself closes a <think> block is allowed.");
    }
    // When prefill closes the think block, proceed — model will output
    // answer tokens directly without entering the reasoning phase.
}

We are open to a stricter variant (e.g. require the prefill to match <think> ... </think> exactly with no trailing answer content) if that avoids edge cases the maintainers are aware of.

Measured impact

From a 10-pair experiment on Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled.i1-Q6_K, b8763, RTX 5090, --reasoning-budget -1, 10 simple Codex-style prompts:

Full 10-pair results in the project at experiments/prefix_priming_2026-04-14/results.json.

• Mean wall delta: -8.64s (primed always faster)
• Median wall delta: -7.96s
• Mean tokens saved: 456 / request
• Quality regressions: 0 / 10 (answers coherent, no <think> leaks, all stop-finished)

The standout case (prompt 4, docstring): baseline produced a 1,460-character "Architectural Decisions and Reasoning" essay in 10.46s / 541 tokens. Primed produced "Add two numbers and return their sum." in 0.29s / 9 tokens.

Question for maintainers

Is there a principled reason the restriction is unconditional — i.e., is there a scenario where a prefill that fully closes the <think> block still causes problems downstream (template rendering, KV cache, sampler state)? If so, we would like to understand it. If not, we think the narrowed condition above is safe and worth the OpenAI-compatibility benefit.

Environment

• llama.cpp build: b8763-ff5ef8278
• Model: Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled.i1-Q6_K (mradermacher i1 quant, Apache 2.0)
• Server flags relevant: --reasoning-budget -1 --jinja --temp 0.6
• Hardware: RTX 5090 32 GB, Ubuntu 24.04 WSL2
• Source location of the rejection: tools/server/server-common.cpp:1062–1063