Repository navigation
[ENHANCEMENT] Avoid needless prompt-cache invalidation so local models (llama.cpp) can reuse their KV cache #1891
Description
Problem (one or two sentences)
Several situations change the start of the request (system prompt, tools, or an earlier message) although nothing meaningful changed, so local servers such as llama-server throw away their KV cache and re-evaluate the whole context, which can take minutes on a 100k-token task.
Context (who is affected and when)
Users running local models (llama.cpp, Ollama, LM Studio) on long tasks, where re-evaluating the context is slow. Hosted providers with prefix caching pay for the same misses in cost.
These servers reuse their cache only for the prefix identical to the previous request. Tools and the system prompt come first, and Zoo Code rebuilds them on every request (src/core/task/Task.ts:4648). That is fine when they only change with a real edit (settings, MCP, rules, model, compaction). The cases below change the prefix although nothing meaningful did. They come from reading the code at feb3386, not from measurements, sorted by expected impact.
| # | Trigger | Where | Re-evaluated |
|---|---|---|---|
| 1 | Condensing uses its own system prompt (SUMMARY_PROMPT) and turns tool blocks into text, so the summary request shares no prefix with the task |
src/core/condense/index.ts:314, :321 |
Full context, near the context limit for automatic condensing |
| 2 | A mode switch (switch_mode, or the user picking a mode mid-task) changes the role, mode rules, skills and tools list |
src/core/prompts/system.ts:75, :99, :123; src/core/task/build-tools.ts:127 |
Full |
| 3 | The last assistant message is sent back differently from what was generated: reasoning is stripped (OpenAI Compatible cannot enable preserveReasoning, #1096 / #1118), and tool arguments are rebuilt with a fixed key order and coerced types |
src/core/task/Task.ts:5234; src/core/assistant-message/NativeToolCallParser.ts:724, :785; src/api/transform/openai-format.ts:552 |
From the last assistant message, every turn (depending on the chat template and on the model's argument order) |
| 4 | codebase_search is offered only while the index services exist, which is not the case during startup, while they are recreated after a settings change, or when embedder validation fails |
src/core/prompts/tools/effective-tool-policy.ts:270; src/services/code-index/manager.ts:87, :366, :420 |
Full |
| 5 | Restarting an MCP server (button, watchPaths, build/index.js change) re-appends it at the end of the server list, which reorders the MCP tools; if the restart exceeds the 10 s wait, its tools are missing for one request |
src/services/mcp/McpHub.ts:1752, :982, :464; src/core/task/Task.ts:4301 |
Full (twice if tools are missing) |
| 6 | Ollama fetches model metadata inside createMessage, so a task's first request uses default metadata (supportsImages: true) and the next ones the real metadata. This changes the read_file description and, through maybeRemoveImageBlocks (live getModel() read), old image blocks. Router providers do the same when nothing is cached yet and the 5 s fetch times out |
src/api/providers/native-ollama.ts:402, :549; src/core/task/build-tools.ts:113; src/api/transform/image-cleaning.ts:8 |
Full, on the second request |
Desired behavior (conceptual, not technical)
Within a task, the system prompt, the tools and earlier messages stay exactly the same from one
request to the next unless the user changes something that really affects them. Temporary states
(a server restarting, the index starting up, metadata still loading) never change them.
Constraints / preferences (optional)
Real edits (rules, MCP configuration, skills, settings) should still take effect during the task;
freezing everything for the whole task is not the goal.
Request checklist
- I've searched existing Issues and Discussions for duplicates
- This describes a specific problem with clear context and impact
Zoo Code Task Links (optional)
No response
Acceptance criteria (optional)
Given a task whose settings, rules and MCP configuration do not change
When the model calls tools over several turns
Then the system prompt and tools array are identical across requests
And the condense summary request starts with the same prefix as the task's previous request
But restarting an MCP server with unchanged tools, the code index starting up, or an Ollama task's second request does not change the system prompt or tools array
Proposed approach (optional)
- Condense with the task's own system prompt, tools and history, adding the instruction as a final user message.
- Keep the tools identical across modes and enforce restrictions at execution time (as already done for Gemini,
build-tools.ts:165); send mode instructions as a user message. Larger change, could be a follow-up. - Let OpenAI Compatible users enable
preserveReasoning, and send back the raw arguments string from the stream. - Offer
codebase_searchbased on the user's settings; return a tool error if the index is not ready. - Order MCP tools deterministically (see [Story] Rolling-search window across first N turns (union-only growth) #580) and keep a restarting server's last known tools.
- Fetch Ollama metadata before the first request (e.g.
ensureModelFetched), keep the last known metadata on failure, and pass the request's snapshot tomaybeRemoveImageBlocks.
Related: #1096, #1118 (reasoning), #1342 (condensing with llama.cpp), #580 (stable tool order), #1591 (cache reads not counted). Zoo Code 3.84.0.
Trade-offs / risks (optional)
- Point 2 changes how modes restrict tools; the model would see tools it cannot call in the current mode.
- Keeping last known MCP tools means a call can target a server that is still restarting; it must fail with a clear tool error.
Activity
bitflippersanonymous commentedon Oct 22, 2026
Qwen3.8 Flash Next, llama.cpp ... reprocesses the entire context every turn. This makes Zoo Code painful to use with this popular combination.