Keep a reusable state at the end of a generated reply, so follow-up turns don't re-prefill it (hybrid GDN models) #732

Setup: Splash 1.6.0 (Homebrew), MacBook Pro Mac16,7 (M4 Pro, 48 GB), macOS 27.2. Model: a Qwen3.8-27B fine-tune (same architecture), Q4_K_M GGUF, draft incoai/Qwen3.8-27B-DFlash2. Client: a coding-agent harness over /v1/chat/completions that sends the previous assistant turn back with its reasoning_content and tool calls (chat_template_kwargs.preserve_thinking: true).

What happens

In a multi-turn agent session, every follow-up resumes from the previous request's replay point and prefills the whole previous reply again. The docs describe this:

A request keeps its reusable model state at the last whole 32-token page before its generation prompt […]: the next turn may render it differently, so a follow-up resumes from there. (docs/internals.md)

Decode writes the rings alone, so states are published only in prefill. (docs/internals.md)

In the 1.6.0 source, Engine::publishReachedStateBoundaries is called only on the prefill path (runtime/engine/Engine.cpp, around line 1851). The decode path publishes KV pages (publishCommittedBlocks) but no GDN state.

Example from the server log

07:34:53 Done    · input 21,243 · cached 17,408 · deduplicated 3,584 · output 15,233 · TTFT 19.0s
07:34:53 Started · input 36,505 · cached 21,216
07:36:45 Done    · input 36,505 · cached 21,216 · deduplicated 13,248 · output 644 · TTFT 85.0s

The follow-up prefilled 15,289 tokens: the 15,233-token reply plus 56 new tokens. It waited 85 s. Of those tokens, 13,248 were deduplicated, so the re-rendered reply matched the generated KV pages token for token. With no state saved at that point, it was still a full prefill. Over 18 consecutive requests in the same session, re-read replies made up 24,393 of 37,689 uncached tokens (65%).

Request

Keep the current replay point, and also keep one state at the end of a finished request's generated tokens. One way would be to copy the lane's state when the request finishes. A follow-up would use it only when its prompt matches up to that point. Otherwise it is discarded, or it is the first thing reclaimed. The replay point stays the fallback, so a reply that renders differently loses nothing.

docs/internals.md already mentions a disposable checkpoint at "the point its generated history reaches" for resumed lanes, so part of this may already exist.

Prior art (as far as I can tell)

• llama.cpp server keeps the slot's live state, including generated tokens, and rolls back only when the new prompt diverges.
• SGLang inserts a finished request's final Mamba state into its radix cache, keyed by prompt plus output ([Qwen3-next] support mamba radix cache for overlap scheduler sgl-project/sglang#14792).
• mlx-lm server stores the cache after generation (cache_type="assistant" in mlx_lm/server.py).

What I can't tell

Only 3 of the 34 requests in this log show any deduplication. On the others, I can't tell whether the re-rendered reply differed or its pages had already been reclaimed. So I don't know how often the match is exact. The 85 s turn above is one where it clearly was.

I'm happy to test a build or send more logs.